The have to meaning, especially when got is not preceded by have, is typically used in spoken speech in very informal contexts (if it appears in writing, it is normally just a transcription of something spoken). In such spoken contexts, this got to is typically pronounced as gotta, and in writing it is often transcribed as such (see e.g. here). Thus, in spoken language, the two senses of got to are usually pronounced differently and so there is normally no confusion.
If the pronunciation is atypical, or if the transcription does not use the 'pronunciation spelling' gotta, then there are indeed sentences like I got to spend time with my wife which, when considered in isolation, are ambiguous.
But again, in practice, the context normally makes it quite clear which meaning is meant.
One clear difference is that in the have to meaning, got to can be used in the present tense, whereas in the other meaning, it is understood to be in the past tense.
CGEL says that gotta is a morphological compound (p. 1617), whereby
the initial to of an infinitival catenative complement may, in informal speech, be morphologically incorporated into the preceding head word.
(There are six other compounds like that: going + to → gonna, have + to → hafta, ought + to → oughta, supposed + to → supposta, used + to → usta, want + to → wanna.)
CGEL further says this about these seven compounds:
This phenomenon is to be distinguished from the regular phonological reduction of to (infinitival marker or preposition) to the weak form /tə/, as in:
[10] a. I hope to see her. /hoʊp tə/ b. They drove to Paris. /droʊv tə/
The most significant difference is that the forms [gotta etc.] can be stranded, whereas the reduction to a weak form illustrated in [10] does not take place in this kind of context (cf. [5i] above). Compare, then:
[11] i a. %He doesn't want me to tell her but I'm gonna ___.
b. %I asked them to help but they don't wanna ___.
ii a. I'm not sure I'll see her, but I hope to ___. [/hoʊp tu:/, /not */hoʊp tə/]
b. That's not the place they drove to ___. [/droʊv tu:/, not */droʊv tə/]
In this respect the case is similar to that of negative forms like can't or isn't (§5.5), and we again regard it therefore as a matter of morphology, not mere phonological reduction. But it is much less systematic than the negative case, applying to just seven words which do not in other respects belong together as a class; it thus falls within the sphere of lexical morphology, not inflection.
This is to say that the forms [gonnna etc.] are morphological compounds. And because the infinitival marker has been incorporated into the compound the catenative complement is a bare infinitival, not a to-infinitival. For the same reason they can only enter into the simple catenative construction, not the complex one. The ordinary verb want can enter into either: They want to get a new car (simple) or They want me to get a new car (complex). There is naturally no compounded counterpart of the latter example because want and to are not adjacent. But even when the object NP is fronted so that the to does immediately follow want, the compound is still excluded. Compare, then:
[12] i a. %Who do you want to invite ___? b. %Who do you wanna invite ___?
ii a. Who do you want ___ to win? b. *Who do you wanna ___ win?
In [ia] who is object of invite, whereas in [iia] it is object of want. Example [ia] thus belongs to the simple catenative construction (like I want to invite Kim) and hence allows incorporation of to, as in [ib]; [iia] belongs to the complex construction (like I want Kim to win) and hence has no counterpart with wanna, for the compound verb licenses only a single complement, a subjectless bare infinitival.
