Parts of fluxbox that can have round corners are the toolbar, the windows and the menus (this for sure works for classic styles as for pixmap stlyes).
Just open the style file of the particular style you are using and add the needed lines to it:
menu.roundCorners: TopRight TopLeft BottomRight BottomLeft window.roundCorners: TopRight TopLeft BottomRight BottomLeft toolbar.shaped: true|false
You can set all corners round or just one or two or three and you can combine them as you like. (e.g. only 'TopLeft BottomRight')
Toolbar.shaped can be either true or false, not both at the same time. It means whether or not toolbar should have rounded corners.
Another thing to know is that e.g. borderless applications with "window.roundCorners" set, stay round, if you ever asked yourself how to get this round terms.
The toolbar can only have round corners on the upper side, so toolbar.shaped (true OR false) is all you have to set.