I suspect that the person who said "We're going off tangent here" may have had in mind not "off track" but "off topic." Going "off track" may indicate either jumping the track (as a derailed train might) or leaving the marked trail (and being in danger of getting lost). Both of those possibilities are fairly strong negative descriptions of going wrong.
In contrast, "going off topic" or "getting off topic" simply means being diverted into an area that isn't the focus of present business. In this respect, it has much the same meaning as "going off on a tangent."
In geometry a "tangent" is a straight line that touches a circle or sphere at exactly one point and then runs infinitely away from that point in both directions. Going off topic is likewise moving steadily away from the point of the meeting or discussion. The problem in both cases isn't that the conversation is utterly derailed or lost in the wilderness; it's that starting from a point of contact with the relevant topic, the conversation is moving farther away from it.
A Google Books search indicates that the phrase "go off tangent" goes back more than 90 years and that it is growing in popularity rather than declining. The earliest instance I found in the search results is from Woman's Home Companion volume 51 (1924), where the usage is clearly geometric:
The head waiter went away and returned with the manager. The manager pushed directly across the floor toward the couple. Away they slid. Doggedly he stalked them. As though unaware of his presence, they drew around him a perfect circle, then went off tangent like a comet. He kept after them, bumping couples; soon he felt as absurd as a mastiff chasing birds.
The next match is unmistakably figurative, though somewhat ambiguous—in a snippet from a novel by Edward Dahlberg, Those Who Perish (1934) [combined snippets]:
"Is that a motion?" asked the Secretary in a tallish voice which went off tangent.
"I make it a motion, Mr. Chairman," said the manufacturer of broadcloth.
"Now I think we're getting somewhere," said the man with the adenoidal boom.
The oldest clear-cut instance I found is from Hearings of the U.S. Senate Committee on Small Business, volume 3 (1954), which involves testimony from a man who evidently was held in one of the infamous Japanese-American internment camps of the 1940s:
Mr. Shibata. And in the camp I became active in different work. Am I going off tangent?
Mr. Purcell. They will stop you if you go off tangent.
Mr. Jonas. Quite all right.
From the same year, in Edmund Bergler, The Writer and Psychoanalysis, we have this specimen:
The analyst, misunderstanding what is going on, interprets the oedipal repetition, with the result that facts and interpretation are at odds and the whole analysis goes off tangent.
The next oldest is also from political discourse, in this case the Nigeria House of Representatives, Parliamentary Debates, volume 18, part 2 (1965):
[Unidentified speaker]: The Member for Gaya North (Alhaji Yunusa) made some very important points though he was going off tangent, but there are arithmetical and geometrical tangents which are admissible, even in trade. This his off tangent is admissible even in trade. He said that we should—
Mr. A. T. Mbegbu (Bende West): I think that when we refer to tangents we talk of things falling off. Is the Minister of Trade now trying to include the points "off tangent" or is he trying to exclude them?
Representative of the numerous examples that a Google Books search found from recent years is this one from Dan F., Don't Drink and Go to Meetings: My Journey to Recovery (2011):
Jay also had a tendency to ramble on during meetings while he was sharing—going off tangent about the beach and the girls at the beach and so on and so forth.
In none of these examples does "going off tangent" express the intended idea as clearly as would "going off on a tangent," "going off topic," "going off track," or "going off target." But to deny that the wording is in idiomatic use is to deny reality.