The opposite of witty could be witless but in the following example
I tried to be funny but I ended up witless
It simply doesn't work.
Moreover, witless suggests that the person lacks basic intelligence, someone who doesn't have a clue, and is dependent on another. Overall an ambiguous and very negative adjective which I would generally not recommend. On the other hand...
I tried to be witty but I ended up witless
is actually quite witty IMHO, and works because of the preceding adjective. But as a stand alone, and for the reasons I explained above, I think witless is best avoided.
Normally this type of situation—a line that comes off as being unfunny (or unoriginal)—is said to fall flat.
I tried to be witty but I ended up falling flat on my face.
I tried to be witty, but it fell flat
A punchline falls flat when it fails to receive the expected reaction from the listener i.e. laughter.
Depending on the lack of positive response from an audience you could say a comedian was a complete fiasco and his act a disaster.
Most of her jokes fell flat and her act was a disaster
Of course, the most obvious solution is unwitty. It fits perfectly.
I tried to be funny but I was unwitty
I tried to be witty but I was unwitty
Wiktionary says: unwitty (comparative more unwitty, superlative most unwitty)
(chiefly archaic) not witty; without wit; silly