"To whom we must have dined"

Who we must have invited for dinner or entertained, before misfortune struck us.

He is further saying  that, misfortune is a blessing in disguise as it is saving him money, time and wine, as because of the 'excuse' of misfortune he is no longer obligated to entertain these people.