I have seen this - Usually happens to enemy characters on upper decks hanging off the back of the ship - or sometimes the sides.
They dis-engage because they cannot "jump" the railing to get on the ship - and you cant get them (well you can - ill get to that) because they keep walking away from your character.
You CAN get these guys "most" of the time with range weapons - Throwing Daggers, Voodoo Doll, Gernades and the best is The Voodoo Staff. The other trick is that as a "ghost" enemy - you have to "wake them up" with a connected hit - which does not always happen.
Dagger and Gernaders barely work... unless you have heavy damage throwing daggers you can not kill them fast enough - and Gernades usually miss (but if you can get one to hit - it usually does decent enough damage that maybe daggers takes them out)...
Voodoo Doll is good, IF ya can get them tuned to strike - if they will walk up to you close enough and your timing is right you may be able to tune them then attack at will - once again, the Voodoo doll must be able to kill the targets before the timer runs out...
Ah, the Voodoo Staff
With Vanquish (or is it Vanish? LOL)
The Staff does two things: it usually "wakes up a ghost acting" enemy to come try to attack you and b) Vanquish has a wide range past the boat railing that it still hits the target at a quick kill rate. Since this I have never had an issue with "off deck floating enemies" who you can not get with hand-to-hand weapons (sword and dagger)...
And thats all I know