Since I can see people getting confused, here's a really horrible map of Port Royal.
It's self-explanatory, but really off with it's size-comparison to the real thing. Should be helpful. The prison is in the fort, for those of you lost from there on.
Red boxes are bridges, oranges boxes are buildings.