In a mathematical context (explaining a formula just written) the following seems unobjectionable: "The set of unitary polynomials has been denoted by P". My question is whether it sounds right to skip the preposition "by", so that the sentence would read: "The set of unitary polynomials has been denoted P".
It's a standard passive voice construction, whereby
P denotes the set of unitary polynomials
the set of unitary polynomials is denoted by P
Of course, people tend to shorten things: the form “denoted P” is actually used both in oral and writing, but is the minority according to a couple of quick Google Scholar searches: “is denoted by x” vs. “is denoted x”.
Signs versus Things
I consider "is denoted x" to be a minor grammatical mistake, because usually the fix is obvious. Just to illustrate a bit, "X is denoted Y." could potentially mean one of two opposing things:
X is denoted by Y.
X denotes Y.
The first would seem far more likely, because the edit-distance is smaller, but really it just depends on X and Y: Which is more likely to be a sign, and which a thing?
Consider the arguably ungrammatical:
2+2 is denoted the canonical example.
In this case, the author presumably intends:
2+2 denotes the canonical example.
That is because the alternating reading, while at shorter edit-distance, is much harder to understand in any typical context:
2+2 is denoted by the canonical example.
This last sentence is arguably correct (depending on how one feels about the grammatical necessity of quotation/formatting). That it is difficult to follow is just because it asks for something extremely unusual: it asks for English phrases to be subject to interpretation as signs for mathematical expressions. Normally one expects mathematical expressions to stand for prose, not the other way around.
There is a particular context in which prose-standing-for-mathematics does actually make sense: in the teaching of the teaching of mathematics. Here, the author of the manual, and the would-be teacher, mutually understand "2+2" going in. The question is how to get the child up to speed.
In that context, I might very well want to say something normally considered odd like:
Let "the canonical example" mean 2+2.
I'd want to say something to that effect so that later I could write as in:
In this situation, use the canonical example.
to mean roughly the same thing as:
In this situation, use 2+2.
I'd prefer the former, because it ever-so-subtly better conveys the full responsibility of the would-be teacher: figure out the state of mind of the child, and tailor the use of the example appropriately. In an extreme case, the would-be teacher must expand "use the canonical example" out, somewhat on their own and somewhat on-the-fly, to something akin to (but hopefully better than):
Imagine a basket with apples in it. Let's say we count the apples, and find two. Now imagine another basket with two oranges in it. Then put all the fruit in one basket. How many pieces of fruit are in the one basket?
Okay, so how can we talk about, and solve, all problems even just a little bit like this one, once and for all?
Just to jump ahead a little bit: "2+2" is the nice way to write down the original question. "4" is the nice way of writing down the answer. These strange but wonderful things really are the best tools we have for counting, and they really are much better than imagining and counting out on fingers (and toes). They're called Arabic numerals, but actually the Hindu invented them a very long time ago. In fact, it was so long ago that you won't even really be able to understand how long ago it was until you learn their much better way of counting things!
I'm going to show you why they are so cool, by starting over with some much harder counting problems, and solving them the hard way: fingers and toes. Then I'd like to show you the easier way, so we never have to do it the hard way again. Pretty cool, eh? Math is for turning hard problems into easy problems!
Well, okay, maybe not "easy". But easier, for sure. Anyways, I'm getting ahead of myself...first the hard way...
[some problem involving counting, literally, up to more than 100]
All that said, even here, a would-be teacher reading such a manual is still in a context where that which are signs and that which are things remains perfectly clear. So clear, in fact, that it is likely possible to state the original directive backwards and still be understood correctly.
That is, the sentence:
Let 2+2 mean the canonical example.
technically means a kind of opposite of:
Let the canonical example mean 2+2.
but both will be read the same in any typical context. That is because, in any typical context, which expressions are signs and which expressions are things will be abundantly clear. As such, the reader will flip around the directive, if necessary, to fix any error of trying to make things be signs and signs be things. (Given a training in logic, a reader will grumble, but do it anyways; without a training in logic the reader will do it instinctively and perhaps without even the ability to recognize that the sentences do, in fact, mean very different things.)
In fact, "denotes", in all its forms, is almost universally poor style. "Denotes" is to "means" as "not light" is to "dark"; the direct forms are better.
For example, consider:
The canonical example is denoted by 2+2.
The canonical example is written 2+2.
2+2 means the canonical example.
"signifies", "evaluates to", "stands for", "represents", "means", good'ol "is", and so forth all convey the same idea as "denotes" without jumping through the hoops of (1) quoting the thing ("notate"), and then immediately (2) undoing that via "de-".
Keep in mind that "is" is first learned when one is an infant. "Means" crops up as a young child, if not sooner. "Denotes" is learned in college, or later yet. (Also, it adds a syllable.)
That kind of difference is not even a competition. Even someone that passionately loves the word "denotes" will still understand "means" faster than "denotes". They are not, of course, precise synonyms. (Indeed, they are not precise words, so they certainly cannot be precise synonyms.) But the person that cares for the difference will not, in fact, be satisfied by using "denotes"; the kind of person that cares about the many meanings of "means" is a logician (or philosopher, or both), and will only truly be satisfied if one uses the appropriate tools (notation such as T ⊧ φ) to document the precise sense in which "means" is meant.
The proper place for such precision is strictly in notation, never prose: "means" is still a better word than "denotes" even when bizarre meaning is meant.
The half-exception is that, in the midst of prose, all things not-prose should be clearly separated. "Clear", as always, is a matter of good judgment.
2+2 means the same thing as the canonical example of basic arithmetic.
Let "2+2" mean the canonical example.
The example is 2+2.
This last form, in this particular case, is likely best. Its key advantage is that it is concise; the key drawback is that it is ambiguous. The ambiguity is irrelevant in this particular case; it is easier for you to figure out what is meant than it is for you to work through the more explicit statements.
In less familiar territory, the verdict changes; more explicit mechanisms for separating notation and prose are beneficial, rather than distracting.
For a more elaborate example, consider:
Foos and bars are completely different (disjoint); in notation, with S = x | foo(x) and T = y | bar(y): S ∩ T = ∅.
Alternatively, but less clearly:
The set of foo, say S, and the set of bar, say T, share no elements, that is, no foo is a bar and vice-versa, or in notation: S ∩ T = ∅.
Note that rendering the equations through TeX doesn't really alter the point substantially; understanding the notation produced would still be substantially slower than following the prose, which is what makes the first approach superior, because it doesn't interrupt the flow of the natural language with the much slower process of understanding the formal language. That is, in this case, there was no call for the second strategy shown, because the example is too familiar to justify such a heavy-handed tack.
Reflection and Redundancy
A key exception to preferring to be implicit about the distinction between signs and their meanings is when signs and things do meaningfully overlap. When writing about thinking about thought (reflective), for example, then signs and things may very well sometimes be the same. In such reflective scenarios, it is conceivable — nay, likely — that the author will, at some point, write:
X means Y.
but in fact mean:
X is meant by Y.
Noticing, and/or fixing, such errors is much harder in reflective scenarios. That is because meaningfulness ceases to be locally asymmetric; both directions of definitions are, in the absence of global understanding, plausible. (With global understanding, one can check that the definitions are acyclic.)
One strategy for dealing with such errors is to try and guarantee their absence in some fashion. This is hard to do without giving up on the desire to write about reflective settings; such writing is inherently bound up with a subject matter where humans are simply especially prone to errors.
If the aim is to write about reflection, then we should plan for, and allow, mistakes.
The only really good advice I know for fault-tolerance in writing is to say everything important (at least) thrice, in three independent ways (for example: prose, notation, example). That way, you can get away with one otherwise fatal mistake per part: majority voting still yields a correct description.