In writing, mainly seen in novel writing, how should we put a word that doesn't exist? for example, "We're screwed beyond levels of screwedness." Please don't tell me that's too unprofessional and I need to change the sentence with a substitute word that exists. I've seen made up words in some novels, but I can't remember how they're written if they're italic or have a hyphen to separate the part that makes the word nonexistent (screwed-ness).
This is a question of style, not grammar. Grammar doesn't require you do anything to demarcate the word. Look at all the words Lewis Carroll invented, for example:
'Twas brillig, and the slithy toves did gyre and gimble in the wabe; all mimsy were the borogoves, and the mome raths outgrabe.
You'll note Carroll didn't do anything special to introduce his nonce words but just let them stand in standard text like they were any other normal word.
That said, you could italicize it, like you said. That's something people do sometimes. You might also use scare quotes, because that's one of the things they are for – to signal you're using what is in quotes "in a non-standard, ironic, or otherwise special sense."