Besides the appearance settings in the panel preferences, there are also a couple of classes you can style through the gtkrc file. Below we'll summarize the global classes. Some plugins also have gtkrc style properties for specific settings, you can find those in the manual page of the specific plugin.
XfcePanelWindow (normal panel window), XfcePanelWindowHidden (autohide window), XfcePanelWindowExternal (4.6 external plugin), XfcePanelWindowWrapper (4.8 external plugins).