Your interpretation is right: '2:40' does indeed refer to the amount of time it took the horse to go a mile. What's important is that these are not cart-pulling horses, these are racehorses.
Merriam-Webster tells us:
1 : a speed of a mile in two minutes and forty seconds
ran like two-forty
2 : high speed
so called from its having once been a trotting record
That trotting record appears to have been the record set by a popular horse named 'Top Gallant'. From Wallace's Monthly:
Top Gallant (foaled about 1806) was perhaps the most popular horse of his day. He does not appear to have won much nor often, yet, like some politicians, he preserved his popularity for all of that, and his time, 2:40 (although beaten the same year by the Treadwell mare), was the slang phrase for speed for twenty years later.
A slightly different explanation is offered in the 1867 book Black and White:
Two minutes and forty seconds is the least time in which a match horse is expected to do his mile, and a regular 2'40' is a slang phrase expressive of anything fast' all through the States.
Here, for instance, is the 1906 New York Times article announcing the horse 'Sweet Marie' setting a new world-record mile for a mare:
It's clear from reading this that the '2:03¾' here is the amount of time it took for her to do the mile. It appears to have been a metric for measuring horse-speed; the 1900 Clark Horse Review is full of mentions of horses going at 2:10, 2:48, 2:90, etc.
Here's another example from The Breeder's Gazette:
Honest Charley could trot his half mile in 1:20 or at a 2:40 rate, and won more than one race at county fairs.
And from 1848, in the newspaper Religious Telescope
Justin Huntly McCarthy's new book reminds one of a horse race at a county fair where the steeds are called back about a dozen times; but when he gets started he goes at a 2:20 rate with growing excitement and compelling interest until he rounds up under the wire.
But a second meaning seems to emerge in the mid-nineteenth century. Here is, from 1856, a sort of travelogue published in the New-York Dispatch:
Don't know what to do exactly, so go to sleep, and dream that the devil is dragging me along at 2:40 rate, and his imps are sticking pins in me.
And an 1862 letter, describing rapid snowfall:
Don’t talk to us about “mild southern climates,” and the “sunny south!” The snow is six inches deep this morning, and falling at a 2:40 rate still. We are to have a sleigh-ride this afternoon. The boys all have fires in their tents and are comfortable.
I assume that, as with the human mile-time, the mile-time for horses has slowly been whittled down; the 2:40 that was so impressive in Top Gallant's time was nothing to write about in the following decades. That would perhaps account for some of the discrepancies noted by Weather Vane in the comments below.