Difference of opinion here, and you might get many lol.
I have both broadsides maxed, I think it's very helpful if you sail solo a lot. If you're most with a crew, it's not such a big dea.
Like Shhlatkins said, max out treasure sense. You will see a difference if the plunder you get.
I also like to spend one point on ramming speed. But you have quite a bit on full sail already, so you would only need ramming speed if your full sail isn't ready to use yet.
I don't think taskmaster really makes that much difference right now. I've found many of the passive skills aren't really much of an impact. I did some tests on this, and other passive skills, and with taskmaster, my friends broadside was just slightly faster. Not enough to make me want to put skill points on it.
I put points in tacking, actually I have 4 in tacking. When I did a test with my friend who had none in tacking, we turned at the exact same speed. Really sad lol.
Now I keep thinking that maybe in the future, these skills we find aren't that great might be tweaked in the future. So things we find useless now, could be just what we initially expected. But who knows....
So bottom line, you have 7 skill points to use.
My advice:
2 - treasure sense
4 - broadsides
1 - ramming speed
Good luck and congrats on getting your war frigate soon
ps - it would be interesting to see how much damage a maxed broadside makes versus a 1 pt broadside.