Leaving Tia Dalma for an ocean swim to PR?

[Wendot]

This has happened to me twice and it's obnoxious:

When exiting the swampy area with Tia Dalma on Cuba, I reappear on the back side of an island in the ocean. My map shows that I'm on Cuba, but when I start swimming around the entire island, I end up in Port Royal as soon as I hit land I can walk on.

Anyone else have this happen? Is there something obvious that I'm missing which makes this sound logical to exit a Cuba swamp to end up in Port Royal waters??