The sentence you gave does not consist of two subordinate clauses. It contains one independent clause, and one subordinate clause. The internal structure of the sentence goes like this:
[He only gave me [what he owed me.]]
The outer pair of brackets encloses the entire sentence, which is the independent clause. The inner pair of brackets indicates the inner clause. Clauses which are contained within other clauses are known as dependent clauses, and this particular one is a nominal relative clause. It is a relative clause because it begins with the relative pronoun what, and it is a nominal clause (or noun clause) because it functions as a noun within the sentence.
Your intuition is mostly correct, but you've misunderstood where to put the clause boundaries. You seem to have been misled by the false assumption that a clause must be a complete sentence, and the idea that a clause cannot contain another clause. In this case the dependent clause what he owed me is incomplete because relative clauses have to be embedded in a larger context to have meaning, which is why they're called "dependent". And the fragment He only gave me is not even a full clause, because it lacks the direct object that's required by the verb gave. It only becomes a clause when you include the noun clause that acts as its object.
EDIT:
There seems to be some confusion about whether a dependent clause goes inside or outside of the independent clause. Let's look at this deductively, beginning with a simple sentence.
(Abbreviations: [] = clause boundaries, {} = phrase boundaries, IO = indirect object, DO = direct object)
[He gave IO{me} DO{ten dollars}].
In this case, I hope that there is no doubt that the indirect object and the direct object go inside the clause that contains them. The independent clause is not just the subject and the verb, but the subject, the verb, and all of the objects of the verb.
The important thing to remember about noun clauses (and other kinds of subordinate clauses) is that the structure of the independent clause does not change when you insert a noun clause. So in the original example we have something like this:
[He gave IO{me} DO[what he owed me]].
The noun clause what he owed me is the direct object of the verb gave, and it replaces the noun phrase ten dollars. But this has no effect at all on the structure of the independent clause. You can do the same thing with the indirect object:
[He gave IO[whoever he had borrowed from] DO[what he owed them]].
You could go even further with this, adding more nested dependent clauses inside dependent clauses, doing this forever in theory. (In practice it becomes extremely hard to understand after you've nested your clauses more than two or three layers.) But no matter how deep your nesting goes or how complicated the dependent clause becomes, it's still a single component in the structure of the higher-level clause. Dependent clauses do not magically move outside the structure of their parent clauses, nor do they change the grammatical analysis of the clauses that contain them.
