According to Online Etymology Dictionary the Old English origin of "have" was "habban" and that it originated from Proto-Germanic:
Old English habban "to own, possess; be subject to,
experience," from Proto-Germanic *habejanan (source also of Old Norse
hafa, Old Saxon hebbjan, Old Frisian habba, German haben, Gothic haban
"to have"), from PIE root *kap- "to grasp." Not related to Latin
habere, despite similarity in form and sense; the Latin cognate is
Note the sentence that starts "Not related to Latin habere", and its continuation.
So English "have" came from Old English "habban", which in turn came from the Proto-Germanic *habejanan, which came from Proto-Indo-European root *kap- "to grab". The PIE root *kap- being the origin of Old English "habban" is also attested in Wikipedia's
Indo-European vocabulary article in the chart.
So English "have" doesn't at all come from a Romance language from what I've seen, as you rightly suspected. The Italian "avere" and Spanish "haber" do.
Latin "habere" from the two sources I checked originated from a different PIE root *gʰabʰ- or *ghabh-, though how it came to be "habere", I have no idea.
The following explains the shifts of sounds from a PIE to Proto-Germanic some time in the first millennium BC, so you can see how it got from *kap to *habejanan.
Proto-Germanic Wikipedia article
So to answer your question, "have" is closer to German "haben" because there is no relation with "habere" or "avoir"; "habere" has a different PIE origin. Proto-Germanic *habejanan is the source of both German "haben" and English "have".