Contracting the main verb in a sentence is perfectly fine. It sounds awkward only when the pattern of prosodic stress falls on that word:
I have to do it.
* I've to do it.
I have yet to do it.
I've yet to do it.
But this is probably just because you can't use a contracted form in a grammatically stressed position, such as:
I don't know what it is.
* I don't know what it's.
But even then, there are sentences in which the stress falls on the contracted verbword and it doesn't sound awkward, showing that it's not ungrammatical except perhaps by the standards of a mad prescriptivist somewhere.