"To whom we must have dined"
Who we must have invited for dinner or entertained, before misfortune struck us.
He is further saying that, misfortune is a blessing in disguise as it is saving him money, time and wine, as because of the 'excuse' of misfortune he is no longer obligated to entertain these people.