Etymonline links this to howling dogs
"howl of a dog," early 14c., earlier "howling chorus raised (by
hounds) when in contact with the hunted animal," c.1300, from O.Fr.
bayer, from PIE base *bai- echoic of howling (cf. Gk. bauzein, L.
baubari "to bark," English bow-wow; cf. also bawl).
From the hunting usage comes the transferred sense of "final
encounter," and thence, on the notion of putting up an effective
defense, at bay. As a verb, "to bark or howl (at)," from late 14c.
An interesting explanation is found on the Word Detective archives
It's that last, "howling dog," sense of "bay" that gave us "at bay."
When "bay" first appeared in this sense in English around 1300, it
meant the chorus of howling barks of a pack of hunting hounds in hot
pursuit of their prey. (This "bay" as "howling bark" also is used in
the phrase "baying at the moon," which some dogs lacking access to
television are known to do.)
In the final stage of such a grim chase, the hunted animal (fox,
raccoon, etc.) will often find itself cornered and turn to face the
pursuing pack. If the animal is sufficiently determined (and who
wouldn't be?), it may be able to at least temporarily repel the dogs,
rendering them able only to stand and howl, thus keeping them "at
As a metaphor for holding off difficulties (and making a perhaps
ultimately futile last stand), "to hold at bay" was in use applied to
humans and their problems by the 16th century.