Everyday language for medical issues is often based on old theories, and our understanding of germs is fairly recent. Consider, for example, malaria ("bad air"). Etymonline has food poisoning from 1864. Despite earlier attempts to explain disease in terms of germs, our current understanding of infectious organisms starts around a similar time. I suspect that this is no coincidence, that the term reflects a developing understanding of a food-borne cause.
In fact some food poisonings, including some of the most severe, are caused by species that don't thrive in a human host, but grow and produce toxins in/on the food before it's eaten. Botulism "is an intoxication usually caused by ingestion of potent neurotoxins, the botulinum toxins, formed in contaminated foods." (WHO fact sheet). So poisoning is more correct than infection in those cases, and person-to-person transmission is impossible. It's a case of a pathogen outside the body putting a toxin into your food that then makes you ill - poisoning is apt.
On the other hand E. coli, for example, is a bacteria species that often causes a food-borne infection (it occurs in the lower gut anyway, but causes illnesswhen ingested via the so-called faecal-oral route) (NHS). This is also described as food poisoning, and not inaccurately, as the bacteria "produce toxins (Shiga toxins) that can cause severe illness" (same NHS link). It's just that they do so in your body.