On Port Royal, head to the Governor's Gardens - past the Governor's Mansion and through that tunnel on the left - then keep going down on the left to the bottom where you'll find another tunnel. Go through there to Murky Hollow - carefully wind your way in deeper and you'll find Brigands there.
I'd advise you to find some friends to help you with the Brigands as they are fairly high levels - 14-18 usually. It's easiest to have a few friends and all work on the same one, then heal up and take out another one.
Good luck.
