In the context, and for difference where it makes sense:
Adversity is: things that happen [to someone or something] (generally bad things).
Deterrent is: things that happen that block some event from occurring.
Paraphrased, the statement is saying:
Numerous bad things that have happened to us have prevented us from going to the lakeside again this year.
To replace adversities with deterrents, the statement is saying:
Numerous blocks or detours (possibly relating to us, but also possibly relating to the status of the lakeside destination) have prevented us from going to the lakeside again this year.
The adversities could, indeed, be solely related to the lakeside destination (flooding, drought, fire, road blockage) but the use of prevented us versus prevented us and others, as well as the writing in the first person, is a clue to a reader that the adversities are related to the author and not the destination. On the other hand, if the author so chooses, she may indicate the adversities themselves and clarify that point.
Further adding to the mix is the possibility to say the following:
Numerous adversities have deterred us from going to the lakeside again this year.
Can deterrent and adversity be interchangeable? In this case, sure, but the use of the word adversity implies something that was not good and not neutral. A road block isn't bad but it's a deterrent. A car engine failure on the way is an adversity.