Targets for goal-directed action can be encoded in allocentric coordinates (relative to another visual landmark), but it is not known how these are converted into egocentric commands for action. Here, we investigated this using a slow event-related fMRI paradigm, based on our previous behavioral finding that the Allocentric to Egocentric (Allo-Ego) conversion for reach is done at the first possible opportunity. Participants were asked to remember (and eventually reach toward) the location of a briefly presented target relative to another visual landmark. After a 1st memory delay, participants were forewarned by a verbal instruction if the landmark would reappear at the same location, (potentially allowing them to plan a reach following the auditory cue before the 2nd delay), or at a different location where they had to wait for the final landmark to be presented before response, and then reach toward the remembered target location. As predicted, participants showed landmark-centered directional selectivity in occipital-temporal cortex during the first memory delay, only developed egocentric directional selectivity in occipital-parietal cortex during the second delay for the “Same cue” task, and during response for the “Different cue” task. We then compared cortical activation between these two tasks at the times when the Allo-Ego conversion occurred, and found common activation in right precuneus, right pre-supplementary area and bilateral dorsal premotor cortex. These results confirm that the brain converts allocentric codes to egocentric plans at the first possible opportunity, and identify the four most likely candidate sites specific to the Allo-Ego transformation for reaches.