In this sentence, what does probably modify? "I am probably not going to the party." Also, do adverbs modify verb phrases or just the main verb? For example, in, "I am walking quickly." Does quickly modify "am walking" or "walking?"
There is a consensus among linguists that "probably" is a sentence adverb. In your example, it modifies the sentence "I am not going to the party." The best reference I know is the chapter on adverbs in McCawley's book The Syntactic Phenomena of English.
Here is an interesting paper from 1970 by George Lakoff on the logic of adverbs: Adverbs and modal operators.
This is an example of how the grammar that people have been taught makes them look at a very simple phenomenon in an unnecessarily complicated way. Your question comes from a way of thinking about grammar that is akin to assuming that you have to study knives before you can remove the skin from an apple, and that you also have to be fully aware of all the different types of apple that there are before you can be sure that you are actually removing the skin.
Look at the sentence in terms of "pieces of meaning." Sentences are made up of small pieces of meaning that are combined together into a whole.
What is a "piece of meaning?" Common sense will tell you that "to" on its own has no meaning. The word "not" carries a little more meaning, and "am" none without "I." But "party" on its own does have more meaning or "lexical content."
Individual words combine together into meaningful groups which themselves form parts of the larger, meaningful "whole" that we call a sentence.
There are many different ways we could divide this particular sentence into meaningful word groups. We could say that the following, just for example, are all meaningful: party / the party / to the party,
But for the purpose of answering your question we do not need to define all the possible meaningful units in this sentence. This is because by simply removing the single word "probably" you are left with a single meaningful "whole" that is composed of all the other words: "I am not going to the party tonight." It is definitely this piece of meaning that "probably" is modifying.
Can you see what "definitely" is modifying in the previous statement?
In the case of your second sentence, I am walking quickly, I think the answer is quite clear: quickly modifies walking. The reason is that in your sentence, walking quickly is a syntactic constituent, so that am walking is not one. And it makes little sense to talk about modifying something that is not a constituent.
The arguments that walking quickly is a constituent are as follows:
It can be fronted:
I say I am walking quickly, and walking quickly I am.
It can be coordinated with like word sequences:
I am walking quickly and singing loudly.
It can be extracted by right node raising:
John knows that I am, but doubts that Sue is, walking quickly.
It can stand alone as a short answer:
'What are you doing?' 'Walking quickly.'
It can be replaced by the pro-form doing so:
I am walking quickly and Sue is doing so, too.
More generally, CGEL has a long defense of 'an analysis of the modal, tense, aspectual and voice auxiliaries as catenative verbs taking non-finite complementation' (p. 1214). For example, according to that analysis,
 He was writing a letter.
consists of a VP was writing a letter, which itself has another VP as a constituent, writing a letter (CGEL, p. 1218). This 'inner' VP, moreover, is a non-finite clause, of the 'gerund-participial' type (p. 1173).
Traditional analysis, which CGEL calls 'the dependent-auxiliary analysis' (pp. 1210-1214), would say that in , was writing is a constituent. As I said, CGEL dedicates a lot of space to explaining why they favor 'the catenative-auxiliary analysis' instead (pp. 1214-1220). And one of the arguments for the latter analysis is that writing a letter in  passes a lot of constituenthood tests. Thus it probably indeed is a constituent. But if it is, then was writing is not a constituent, contrary to what the dependent-auxiliary analysis would say. Note that in , either was writing is a constituent, or else writing a letter is a constituent, but they can't possibly both be constituents---constituents are either disjoint or else one wholly contains the other; they cannot only partially overlap.
I am not yet sure what to say about your first example, but I hope to have an exchange with Greg Lee (in the comments following his answer to your question) that may clarify things for me. (This is why I couldn't answer your question over on Grammarly Answers.)
An adverb modifies a verb, an adjective or another adverb. In the sentence "I am probably not going to the party" there is nothing for the adverb "probably" to modify other than the verb phrase "am not going".
In the sentence "I am walking quickly", the adverb quickly modifies the verb phrase "am walking". If 'am' is not included and only "walking" is modified, the sentence will become ungrammatical and meaningless like I am a/the (walking quickly).