I am supposed to clock in at 8 am every day to work. I was late today, just like every day, but today I was only late by 5 minutes, less than every other time. Is saying "earliest I've been to work" correct? Even though I was late?
Since you're saying, "[This is the] earliest I have been to work," I would say yes, this is correct. You're specifically referring to yourself and the past.
Whether you're on time or not is irrelevant to the actual statement since the subject of the sentence is you getting to work, not the time your work begins.
There is a logical and a linguistic answer to your question.
If you wanted to make your statement completely precise, you might say something like this:-
The time at which I arrived today was the earliest of all the times at which I have ever arrived.
But that precision is bought at the expense of making it stilted and laboured.
Your statement is, strictly, a sentence fragment with no main verb. So you would not be well-advised to such a casually unrepentant justification to your enraged boss.
Nevertheless, it is clear. It is a sort of ellipsis, which takes for granted something like
[My arrival today was the] earliest [time at which] I’ve ever been to work.
This is not to say that the works in angle brackets passed through the speaker’s (or listener’s) mind unspoken!
At least in US English, "been" wouldn't be narrow enough for what you're trying to convey. So here are some slight variants of what you proposed, that would be clearer:
This is the earliest I've ever gotten to work or ever started work or ever arrived at work
These seem fine. However, I would say: This is the closest to being on time I've ever been!