Mplungjan's comments are quite correct.
I think in direct answer to your question: grammaticality is not the issue with the usage.
It is always grammatical to say, I'm sorry to [vb.].
The bigger issue is verb agreement with context.
I'm sorry to disagree with you. -- Appropriate
I'm sorry to hot air balloon. -- grammatical, but meaningless.
So, the grammaticality of the phrase is not dependent on the verb choice. It's meaning in context, however, is quite dependent on verb choice.
This discussion is furthered by mplungjan's excellent discussion of which of these phrases can stand alone as a fragment. In those cases, it is dependent on the context of the verb. Some verbs need an adverb, object or objective phrase to make their point intelligible.
I am sorry to be. -- grammatical, but even as a statement of regret of your own existence, it is odd.
I am sorry to be here. -- much better.