The comparative notion of "jail" as a temporary, local facility with "Prison" as permanent distant facility seems to be a subtle implication developing toward a definitive distinction.
Short-term or long-term?
Most of the answers indicated that regardless of the institution's location, the length of expected stay influences the usage--jail for short-term, and prison for long term, but the OED does not necessarily support that distinction
Jail
NOUN
A place for the confinement of people accused or convicted of a crime:
he spent 15 years in jail
Prison
NOUN
A building to which people are legally committed as a punishment for a
crime or while awaiting trial:
he died in prison
But there is a subtle distinction between the definitions of jail and prison: jail is defined as a place, while prison is defined as a building.
This distinction in the definition may arise from the etymology of the words.
Jail
late 13c., gayhol, from Old North French gaiole and Old French jaole,
both meaning "a cage, prison," from Medieval Latin gabiola, from Late
Latin caveola, diminutive of Latin cavea "cage, enclosure, stall,
coop" (see cave (n.)). Both forms carried into Middle English; now
pronounced "jail" however it is spelled. Persistence of Norman-derived
gaol (preferred in Britain) is "chiefly due to statutory and official
tradition" [OED].
Prison
early 12c., from Old French prisoun "captivity, imprisonment; prison;
prisoner, captive" (11c., Modern French prison), altered (by influence
of pris "taken;" see prize (n.2)) from earlier preson, from Vulgar
Latin *presionem, from Latin prensionem (nominative prensio),
shortening of prehensionem (nominative *prehensio) "a taking," noun of
action from past participle stem of prehendere "to take" (see
prehensile). "Captivity," hence by extension "a place for captives,"
the main modern sense.
If the meaning of cage and the word picture of cave prevail, jail would imply a smaller place, and by extension a less permanent place. Jail evokes the popular image of the barred cell in the sheriff's office in almost every western US town during the expansion of the 19th century. People may have spent short amounts of time in these jails for minor offenses, but if hardened criminals weren't executed on the spot, they were commonly remanded to state (and later federal) authorities for permanent punishment. The implication of this practice seems to prevail in the word jail.
"He spent the night in jail."
By contrast, the word prison is associated with fortified French prisons like the Bastille. The 20th century brought urbanization with increasing crime in the US, requiring more and stronger buildings that resembled the Bastille, and these buildings are called prisons regardless of their location. Federal and and state institutions are predominantly large fortified buildings and so prison dominates the expression at that level.
Local or Distant?
In general, prison is used about twice as frequently as jail in the US Ngram General, but a federal institution is rarely referred to as a federal jail It is called a federal prison or penitentiary, and the distinction has become more pronounced over time. In the federal context, prison is 44 times more common than jail, and jail is never an official designation. Ngram Federal
Likewise, for a state institution, state prison is 30 times more common than state jail Ngram State, and jail is rarely an official designation. Conversely a county institution is 30 times more likely to be called county jail than county prison. Ngram County Moving to local institutions, local jail is three times more common than local prison Ngram Local, with similar results for city and town institutions. The fact that the county level seems to buck this local-distant distinction, deserves some deeper investigation.
The local-distat distinction is pronounced, consistent over a century, and trending toward a tipping point for the definitions, but there is still sufficient overlap to recognize it as a matter of subtle inference or connotation. At the federal and state levels, prison has always dominated usage. At the county level jail has always dominated the usage. For local institutions (local, city, town), the usage curve of prison and jail tracked together until the turn of the 20th century, when the federal prison system was expanded significantly.
How much of this quantum shift is due to state and federal governments manipulating public perceptions? How much of this distinction is an intuitive understanding of our current justice system: distant facilities are bigger and more secure?
Professional Opinions
What do sheriffs have to say about this distinction What's the difference between jail and prison? offers an example:
Think short-term and long-term.
Jails are most often run by sheriffs and/or local governments and are
designed to hold individuals awaiting trial or a serving short
sentences (in Florida, inmates serving 364 days or less serve their
time in jail).
Prisons are operated by state governments and the Federal Bureau of
Prisons (BOP) and are designed to hold individuals convicted of
crimes.
Jails operate work release programs, boot camps, and other specialized
services. They try to address educational needs, substance abuse
needs, and vocational needs while managing inmate behavior. Inmate
idleness contributes to management problems.
State prison systems operate halfway houses, work release centers and
community restitution centers - all considered medium or minimum
custody. Inmates assigned to such facilities are usually reaching the
end of their sentences.
There are approximately 3,600 jails in the United States. The BSO jail
system is the 10th largest local jail system in the United States. It
is one of only 3% of the local jails in the United States to have
earned accreditation from the Commission on Accreditation for
Corrections.
The Broward Sheriff's Office maintains five jails: the Main Jail, the
North Broward Bureau, the Joseph V. Conte Facility, Paul Rein
Detention Facility and the Stockade Facility. The number of beds in
the entire system is nearly 4,800.
One would assume the Broward County Sheriff knows something about his legal system, and at least one lawyer confers on the Difference between Prison and Jail
Yes, there are definitely many differences between prison and jail.
They are entirely different entities. Here are some of the differences
that you’ll want to know about if it’s possible that you or a member
of your family is facing the prospect of going to jail or to prison:
Jails are locally operated places of incarceration — usually the
county runs the jail. There are about 3,600 jails in the U.S.
Prisons are operated by the state government, or by the federal
government (the federal Bureau of Prisons).
Since jails are within the county where the individual was arrested,
the jail isn’t very far away.
A state or federal prison could be very far away from a convicted
person’s home and family. There are only about 100 federal prisons,
detention centers, and correctional institutions in the U.S.
A person who is being held in custody before a trial/has not yet paid
bail/was only recently arrested will be held at a local jail, not in
prison.
Jails are also a place for people who have been convicted of
relatively minor crimes. A jail sentence rarely exceeds a year or two.
Conclusion
Although the words clearly share a semantic field, there are strong inferences and connotations that impact the perceived meaning of each word. They can be used interchangeably, but the context can easily suggest one meaning over the other.
"*He is going to *prison**," may say nothing about the length of his stay
And
"He's sitting in jail," may say nothing of the institution where he sits.
In his use of language, the common man frequently rejects professional opinion, but it seems quite clear that state and federal governments in the United States have drawn a real distinction between prison and jail. Eventually, the common man may concede to the brute force of the government's definition.