The albatross around one’s neck as a symbol of endless, self-inflicted bad luck comes from “The Rime of the Ancient Marineer” by Samuel Taylor Coleridge. The poem is a fairly common assignment in high-school English classes, at least in my experience. I went to high school in New York City; we had to memorize stanzas of it as punishment if we forgot our materials for class. I had a number of friends who attended other schools, and were also required to read it. Can’t say it was a popular assignment (as I said, for my class it was literally punishment), but it did get read.
On the other hand, it wasn’t a particularly popular assignment, and I suspect its tendency to stick out in many people’s minds stems mostly from its considerable length. For anyone who either didn’t pay enough attention in the first place, or wasn’t particularly struck by the poem (and/or by its length) to remember it, or simply didn’t happen to have it assigned in the first place, the phrase is probably unknown. It is not a widely used phrase, in either conversation or media.
But as some evidence that it is widely-known enough to see use, the film Serenity includes this dialogue:
The Operative: That girl will rain destruction down on you and your ship. She is an albatross, Captain.
Capt. Malcolm Reynolds: Way I remember it, albatross was a ship’s good luck, ’til some idiot killed it.
Capt. Malcolm Reynolds: [to Inara] Yes, I’ve read a poem. Try not to faint.
and then later, to the “that girl” in question:
Capt. Malcolm Reynolds: But it ain’t all buttons and charts, little albatross.
Serenity was a feature-length major motion picture from 2005; the people who made it apparently considered their audience likely enough to know of “The Rime of the Ancient Marineer” and its albatross to include these lines (but then also basically explained the reference with the “Yes, I’ve read a poem,” line).