In a mathematical context (explaining a formula just written) the following seems unobjectionable: "The set of unitary polynomials has been denoted by P". My question is whether it sounds right to skip the preposition "by", so that the sentence would read: "The set of unitary polynomials has been denoted P".
It's a standard passive voice construction, whereby
Of course, people tend to shorten things: the form “denoted P” is actually used both in oral and writing, but is the minority according to a couple of quick Google Scholar searches: “is denoted by x” vs. “is denoted x”.
I found both correct.
Signs versus Things
I consider "is denoted x" to be a minor grammatical mistake, because usually the fix is obvious. Just to illustrate a bit, "X is denoted Y." could potentially mean one of two opposing things:
The first would seem far more likely, because the edit-distance is smaller, but really it just depends on X and Y: Which is more likely to be a sign, and which a thing?
Consider the arguably ungrammatical:
In this case, the author presumably intends:
That is because the alternating reading, while at shorter edit-distance, is much harder to understand in any typical context:
This last sentence is arguably correct (depending on how one feels about the grammatical necessity of quotation/formatting). That it is difficult to follow is just because it asks for something extremely unusual: it asks for English phrases to be subject to interpretation as signs for mathematical expressions. Normally one expects mathematical expressions to stand for prose, not the other way around.
There is a particular context in which prose-standing-for-mathematics does actually make sense: in the teaching of the teaching of mathematics. Here, the author of the manual, and the would-be teacher, mutually understand "2+2" going in. The question is how to get the child up to speed.
In that context, I might very well want to say something normally considered odd like:
I'd want to say something to that effect so that later I could write as in:
to mean roughly the same thing as:
I'd prefer the former, because it ever-so-subtly better conveys the full responsibility of the would-be teacher: figure out the state of mind of the child, and tailor the use of the example appropriately. In an extreme case, the would-be teacher must expand "use the canonical example" out, somewhat on their own and somewhat on-the-fly, to something akin to (but hopefully better than):
All that said, even here, a would-be teacher reading such a manual is still in a context where that which are signs and that which are things remains perfectly clear. So clear, in fact, that it is likely possible to state the original directive backwards and still be understood correctly.
That is, the sentence:
technically means a kind of opposite of:
but both will be read the same in any typical context. That is because, in any typical context, which expressions are signs and which expressions are things will be abundantly clear. As such, the reader will flip around the directive, if necessary, to fix any error of trying to make things be signs and signs be things. (Given a training in logic, a reader will grumble, but do it anyways; without a training in logic the reader will do it instinctively and perhaps without even the ability to recognize that the sentences do, in fact, mean very different things.)
In fact, "denotes", in all its forms, is almost universally poor style. "Denotes" is to "means" as "not light" is to "dark"; the direct forms are better.
For example, consider:
"signifies", "evaluates to", "stands for", "represents", "means", good'ol "is", and so forth all convey the same idea as "denotes" without jumping through the hoops of (1) quoting the thing ("notate"), and then immediately (2) undoing that via "de-".
Keep in mind that "is" is first learned when one is an infant. "Means" crops up as a young child, if not sooner. "Denotes" is learned in college, or later yet. (Also, it adds a syllable.)
That kind of difference is not even a competition. Even someone that passionately loves the word "denotes" will still understand "means" faster than "denotes". They are not, of course, precise synonyms. (Indeed, they are not precise words, so they certainly cannot be precise synonyms.) But the person that cares for the difference will not, in fact, be satisfied by using "denotes"; the kind of person that cares about the many meanings of "means" is a logician (or philosopher, or both), and will only truly be satisfied if one uses the appropriate tools (notation such as $T \models \phi$) to document the precise sense in which "means" is meant.
The proper place for such precision is strictly in notation, never prose: "means" is still a better word than "denotes" even when bizarre meaning is meant.
The half-exception is that, in the midst of prose, all things not-prose should be clearly separated. "Clear", as always, is a matter of good judgment.
This last form, in this particular case, is likely best. Its key advantage is that it is concise; the key drawback is that it is ambiguous. The ambiguity is irrelevant in this particular case; it is easier for you to figure out what is meant than it is for you to work through the more explicit statements.
In less familiar territory, the verdict changes; more explicit mechanisms for separating notation and prose are beneficial, rather than distracting.
For a more elaborate example, consider:
Alternatively, but less clearly:
Note that rendering the equations through TeX doesn't really alter the point substantially; understanding the notation produced would still be substantially slower than following the prose, which is what makes the first approach superior, because it doesn't interrupt the flow of the natural language with the much slower process of understanding the formal language. That is, in this case, there was no call for the second strategy shown, because the example is too familiar to justify such a heavy-handed tack.
Reflection and Redundancy
A key exception to preferring to be implicit about the distinction between signs and their meanings is when signs and things do meaningfully overlap. When writing about thinking about thought (reflective), for example, then signs and things may very well sometimes be the same. In such reflective scenarios, it is conceivable --- nay, likely --- that the author will, at some point, write:
but in fact mean:
Noticing, and/or fixing, such errors is much harder in reflective scenarios. That is because meaningfulness ceases to be locally asymmetric; both directions of definitions are, in the absence of global understanding, plausible. (With global understanding, one can check that the definitions are acyclic.)
One strategy for dealing with such errors is to try and guarantee their absence in some fashion. This is hard to do without giving up on the desire to write about reflective settings; such writing is inherently bound up with a subject matter where humans are simply especially prone to errors.
If the aim is to write about reflection, then we should plan for, and allow, mistakes.
The only really good advice I know for fault-tolerance in writing is to say everything important (at least) thrice, in three independent ways (for example: prose, notation, example). That way, you can get away with one otherwise fatal mistake per part: majority voting still yields a correct description.