In this case, their intent was clearly that it should be an non-restrictive clause. Here, the details about alchemy are supplementing our understanding of why it was like alchemy, rather than explaining which kind of alchemy it was.
Another way to talk about restrictive/non-restrictive that might make it clearer is "defining" and "non-defining" (respectively). A restrictive/defining relative clause should define "which one".
In this case, we can replace the sentence with a couple options, to see which one makes sense:
- The chemistry that Bacon first learned was really more like the kind of alchemy that used poetic, esoteric language.
- The chemistry that Bacon first learned was really more like alchemy because it used poetic, esoteric language.
The first option, because we're talking about "what kind", is the same as using a restrictive/defining relative clause.
The second option, because we're adding extra information (in this case a new subordinate clause) is similar to using a non-restrictive/non-defining clause.
Hope that helps!