
What I think and know is this, and I will say it very simply for my sake and others.:
1. The fidelious charm was cast on Godric's Hollow, noone can find it apart from the Potters and the secret keeper (pettigrew).
2. Pettigrew betrays the Potters and reveals the location to LV who comes and murders harry's parents and so forth and so on.
3. Hagrid then comes to Godric's Hollow and gets Harry. Which obviously means that when the secret was revealed the charm became void, and that is why hagrid managed to get there. Not because of some over-thought up, round about reason but simply because it is void.
Sorry if thats already been said, but it is extremely hard to follow this!