I have had the ball band slip off before with pool activity, luckily that day I was in swim trunks and not board shorts
I prefer either strapless inner wear with no ball band for swim trunks/board shorts. It’s not as much of a headturner that way because any ball support is going to lift more, but you will still get noticed
Another option I like is wearing either Stealth with or without ball bands in a swim brief/speedo and board shorts or swim trunks on top. I think this might be the visually appropriate option you are looking for and everything will be secure in place with a more modest bulge
