It's on Port Royal. If you stand on one of the docks and face the island, look to your left and all the way up the hill to the north is a stone castle (just like the one in the movie, where Elizabeth fell off). You have to go up the hill and along the way there are a lot of guards (low levels) and you can usually just run right past them. Once you get to the top and see the castle (i'll say castle because it looks more like a stone castle than a fort), keep to the left, and you'll see a stair case. Again, try and avoid fighting with guards. Just run past them and get into the fort (trust me you need your energy).
If you're trying to get the stuff to make your voodoo doll, you are going to have to run around all levels and rooms of the fort and search the barrels and containers. When you search, it takes a while, and there are guards EVERYWHERE who will easily sneak up while you are searching and pull a fast one. A lot of the guards are high levels, so try and bring a crew or friends with you to watch your back. It's going to take a while to get all the stuff you need. I went up and there were already a bunch of people up there and we all talked and had each other's backs while we took turns searching. This was the hardest quest for me so far (aside from the theives den).
Good luck!
