"To whom we must have dined"

Who we must have invited for dinner or entertained,before misfortune struck us.

He is saying further that, misfortune is a blessing in disguise as it is saving him money, time and wine, as because of the 'excuse' of misfortune he is no longer obligated to entertain these people.