This is an interesting question because by dictionary definition, "salty" and "savory" are listed as synonyms, but I wouldn't necessarily consider them to be as I have learned it natively in the US.
In my experience, it depends somehwhat on context. "Savory" tends to mean flavorful in a non-sweet way, but it doesn't have to be salty. This is used especially often with rich dishes that might feature meat, vegetable, butter/oil and/or herb tastes. A salty item could also be savory, but something savory does not have to be salty. Something with an "umami" taste quality would definitely be savory to me.
If I were talking about a dinner dish, I would definitely find "sweet" and "savory" to be the opposites I would talk about. If I called a dinner dish "salty" I would probably be saying it was too salty, or that somehow the saltiness overwhelmed the general flavor. But if I called a dish savory, it could be flavorful from herbs or richness.
Snack foods are a different matter, though. A salty snack is purposefully salty, and satisfies a craving for salt in a way I would not expect a savory meal to do. "Savory" isn't a word I would use as much for, for instance, a cracker, unless it had some particular flavoring apart from salt that set it apart. But if I am thinking about crackers or chips, or French Fries, etc. I would say something was "salty" in a positive, crave-able way.
While the dictionaries don't make this distinction, here is a resource that considers savory to be the opposite of salty:
In the world of cuisine, savory is also often used to mean the opposite of sweet, or salty.