I can see your concern if running WOT, but for 20 MPH & under I wouldn't have any concern about the swivel chairs.
If your boat is new enough, you can 'lock' them to prevent the swivel if not yours could easily be retrofitted with pedestals/bases that will.