There are 2 factors:
4-digit integers can be written without a comma
In (American) English prose, 4-digit integers may or may not have a comma. The choice is a matter of style; both "4000 widgets" and "4,000 widgets" are commonly used. So it's unexceptional that years are written without commas.
That said, it is simply a convention in written English that ordinary 4-digit Gregorian years don't take a comma. Basically, "4,000" is always a number (e.g. "In 4,000 years, I hope to move to Mars."), whereas "4000" can be either a number or a Gregorian year (e.g. "In the year 4000, I hope to move to Mars.").
Aside: Years outside the basic standard calendar system, such as 5-digit years, sometimes do take commas. For example, Wikipedia's style guide recommends "10,400 BC".
(A possible rationale is that American-style full dates ("January 2, 2000") there is already a comma, so if spacing gets messed up it would be harder to read with two commas. But that doesn't explain why dates are still written without a digit separator in British usage ("2 January 2000") or in regions that use a dot or other mark as delimiter.)