Both are grammatically correct, and in the context you describe, it is acceptable to use them interchangeably. “How to not X” has a slightly more direct and active connotation than “how not to X” in this case.
However, it's better to say “how to keep going” or “how to persevere” rather than “how to not give up.”
“How to not X” is used for things like, “how to not snore” or “how to not drink” because, well, there’s no opposite of “snoring” except for “not snoring,” so that’s the only way to say it.
“How not to X” is sometimes used the same way, but it’s also used in cases where X is something you would do or might do, but you’re describing how to avoid doing it improperly. For example, “how not to dress” or “how not to speak to a police officer when you are pulled over.”
Unlike “snoring,” “giving up” does have an opposite- words and phrases like “persevering” or to “keep going.” So, you generally should use those instead.
It’s true, of course, that it’s quite common to say, “Never give up!” or “Don’t give up!” That’s because when you’re giving a command, forbidding someone to do something can be just as clear and direct as telling them to do something. However, if you’re giving them details on how to do something, rather than what to do, you’ll generally want to express that in terms of “how to do X” instead of “how to not do X.”
In general, the only time when the phrase “how to not X” is used is when it’s difficult or impossible to clearly express the exact same idea as “how to X.” For example, “how to not be jealous,” “how to not get pregnant,” “how to not snore,” and so forth.