I hear it often, and can usually derive a sentence's meaning with or without it. What does it really mean? When would one use it?
It means 'whether or not this is of any use/value'.
For what it's worth, I'm very sorry I broke the window.
means, for example, that it may not make any difference to the physical state of the window, but that hopefully the apology helps placate you.
It's almost a kind of self-deprecation; it's saying 'No words of mine will be adequate, but...'
for what it's worth (informal)
said when you are giving someone a piece of information and you are not certain if that information is useful or important.
for all or for what it's worth; for whatever it's worth:
Even though it may not be important or valuable.
For example, Here's my opinion, for what it's worth, or For whatever it's worth I've decided to take the train. [Late 1800s]