Both ways can potentially be equally grammatical. You just have to work with it.
To see this, break the phrase up with commas where you would naturally pause. We'll start with the version including of:
- Sunday, the 24th of May, is...
Doing this to the version without of renders something a little different:
- Sunday the 24th, May, is...
Now that we have these, let's look at their meanings.
Number one's appositive, the 24th of May, correctly renames the preceding subject, Sunday; Sunday is the 24th of May.
Number two's appositive also correctly renames the preceding subject, but here we have a mismatch of sorts—a day of the week is being paired with a day of the month. This is slightly less logical than number one which pairs a day of the month with a month.
Therefore including of is ideal.