In your restaurant context this one would be hard to use (but see below), but if the invitation had instead been for a delicious home-cooked meal, you could’ve said:
“OK, be like that, it’s fine ‘cause there’ll be that much more {delicious home-cooked food} for me!”
(explanation of *"be like that" from 'WordReference[dot]com and example of usage of "there'll be that much more for me" from 'Heal Me: The Me Series - Book 4' by D.R. Grady, via 'Google Books")
Even for a declined restaurant invitation, however, you could perhaps use it to make a last-ditch, albeit lengthy, bluff to get him/her to reconsider and, even if unsuccessful, it would at least rub the quality of the food in his/her face one last time:
Ok, yes, thank you [very much], I think I will go alone, but I’ll
still order a plate [of their world-class specialty] for you just in
case you change your mind but if you don’t, that’s fine, too, ‘cause
there’ll be that much more [of it] for me!
*Please note that you could preface most of the good answers given so far with either "Fine, be like that!" or "Fine, be that way!" to emphasize your frustration and disappointment even further.
(warning, use the following with caution: If this person has made similar excuses in the past [and you don’t really care if you ever hear from them again!], you could also add:
“… and don’t forget to wash your hair while you’re at it”
to any of the good answers given so far).