The "it" referred to in the sentence "Like all that pertains to crime, it seemed never to have known a youthful era" is a wooden jail, which obviously pertains to crime. It seems likely, therefore, that Hawthorne intends "all" to stand for "all else"—that is, "everything else."
This situation is functionally equivalent to a situation where a person stands up at city council meeting and says
Like everyone attending this meeting, I'm concerned about [topic X].
The person speaking doesn't intend to exclude himself or herself from "everyone attending this meeting," since, obviously, the speaker is attending it, too. But in both written and spoken English, many people omit the "else" and leave it to their readers or hearers to infer the word's unspoken presence. This can lead to confusion on the part of hearers or readers who interpret the words of the writer or speaker literally.
Anyway, it seems clear to me that Hawthorne means this:
Like all else [that is, "Like everything else"] that pertains to crime, the jail seemed never to have known a youthful era.
Your first alternative sentence,
Like whatever that pertains to crime, it seemed never to have known a youthful era.
sounds vaguer and flabbier to me than "Like all [else]..." or "Like everything [else]...," as though the writer or speaker weren't sure what things might pertain to crime but decided to include all of them anyway. I would interpret it as meaning "Like the things that pertain to crime..." rather than "Like everything that pertains to crime..."
It also has a syntactical problem: The that following "whatever" should not be included if the rest of the sentence is to run as written. Either the sentence should begin "Like whatever [else] pertains to crime..." or whatever should be replaced by a word or phrase that provides a firm anchor for the following that phrase—for example, "Like any other human work that pertains to crime..." Although parenthetical constructions of the form "whatever that [statement] means" are common in English, declarative constructions of the type "whatever that pertains to X" do not normally occur.
Your second alternative sentence,
Like what pertains to crime, it seemed never to have known a youthful era.
seems somewhat more awkwardly worded than your first alternative. I'm not sure why that should be the case, but idiomatically phrases of the type "Like what pertains to X" sound off-kilter and strangely unsophisticated. I wouldn't use that wording by choice.