Think of ‘jersey’ as being like an ancient ‘brand-name’ for ‘sweater’, originating from an island off France, where they were made, that eventually became synonymous with ‘sweater, pullover, and jumper’ which are other words for similar garments.
A bit like how ‘hoover’ (a brand-name) became the name for ‘vacuum cleaner’, in the UK.
As Jeeped has commented, a ‘jersey’ is a kind of knitted sweater, made in the island of Jersey in the Channel Islands (UK). Jersey is also the name for various kind of stretchy fabrics, such as cotton jersey, which is similar to ‘T-shirt fabric’ that you find in ‘sports jerseys’, and wool jersey which is used for knitted dresses and other items.
Jersey came to be known as any kind of knitted jumper, or sweater, and the words jersey, sweater, pullover, and jumper are interchangeable.
Jersey, sweater, pullover, and jumper are all pulled over the head - they have no front opening.
Whereas ‘cardigan’ traditionally opens all the way down the front, with buttons fastening it, and is made of the same kinds of material as the others.
Lastly ‘jersey’ or ‘jersey fabric’ also refers to a kind of fine knitted cloth of cotton or wool that is drapeable, stretchy, smooth and flowing - of which T-shirt material is an example.
Sweaters (and jumpers, pullovers, and jerseys) may also be hand-knitted - they may therefore show up as bulky and thick, as in hefty ‘cable knits’ from Scotland or the Isle of Skye, thicker than cardboard (like 1cm thick or more - a ‘fisherman’s jumper’ - to keep out the cold) in which case they are not stretchy, not soft or flowing as in ‘jersey fabric’. And sometimes, such garments may also be softer and made with a jersey kind of fabric.
So sweater, jersey, jumper, and pullover are all synonymous and all mean ‘some kind of knitted top that you pull on over your head’.
This article states that jersey fabric was originally of wool, and then expanded to include cotton and other fibres.