The earlier answerers are certainly correct in saying that "A's" is properly punctuated with an apostrophe. One of the answers, however, focuses on the general question of how to treat the plural of a single letter (in this case, A); and the other suggests that the "A's" in "Oakland A's" is an abbreviation. In my view, "A's" is more properly viewed as a contraction of "Athletics"—not that the distinction alters the bottom line that "A's" should have an apostrophe.
As a historical point of interest, I note that the Philadelphia Athletics' cap logo was "A," not "A's." (See the first photo in Wikipedia's article on the Philadelphia Athletics.)
The "A" cap logo carried over through the first five years or so of the Kansas City Athletics, as these images indicate. Around 1960, the Athletics changed their cap logo to the overlaid KC that the team retained until it left for Oakland at the end of 1967.
In Oakland, the cap logo reverted to "A," as these photos from 1969 show. The still-current "A's" cap logo seems to have originated in 1970.
On a final language-related note, I've read that Charlie Finley (the owner of the Athletics during their last years in in Kansas City and early years in Oakland) was responsible for the push in the early 1970s to call the team "the A's" rather than "the Athletics"—an effort that he promoted quite effectively by ordering his TV and radio broadcasters never to say "Athletics." Supposedly, Finley objected to calling his team "the Athletics" because "There's no such thing as an athletic!"
It seems likely that Finley misunderstood the nature of his team's name. In Pamela A. Bakker, Eyes on the Sporting Scene, 1870–1930: Will and June Rankin, New York's Sportswriting Brothers (2013), the authors point out that two Philadelphia sports clubs played an early form of baseball by the year 1860:
During the boyhood of Will and June, it is unknown if they were exposed to ball games like round town ball, Philadelphia town ball, or bandy wicket, all of which had been played in Pennsylvania before the Civil War. ... The New York Knickerbockers had been formed in September 1845, and a host of New York City, Brooklyn, Long Island, and New Jersey clubs joined suit from 1854 through 1857. Philadelphia's Olympic club had switched from townball and cricket to the New York style of baseball in 1859 and Philadelphia's Athletics club switched in April 1860[.]
This passage suggests that the name "Athletics" didn't signify a plural form of "Athletic," but referred to "Athletics" in the sense of athletic activities, as pursued by an athletics club.
In any event, Finley wanted the "A" in "A's" to stand for nothing but the letter A, which would bring us full circle to Jay's answer above, if Finley had succeeded in eradicating the name "Athletics"—which he didn't.