I've seen the word "byproduct" written several different ways and I'm wondering which is the preferred format? Is this a variation between US and British English or just a matter of a "house style?"
The OED lists two categories of words which begin "by" or bi; -see below. By-product belongs to the second category and is spelled with a hyphen.
To the first category belong such words as because (originally bycause). But the second category contains, among other things, those words already formed in Old English with by, or later words where "by" 'already has an attributive sense' -by-road, by-product, by-station etc.
A Middle English variant spelling of the prefix bi- prefix, be- prefix, under which see most of the words, as, under be- prefix, bycause, bydene, bydryve, byfall, byfore, byget, bygynne, bygile, etc.; under bi- prefix, byreusy, byweve, etc. Those words only are given under by- comb. form for which no forms with be- or bi- have been met with.
- by- (sometimes bye-): the preposition, adverb, or adjective by prep. and adv., by adj. in combination, either in words already formed in Old English with the accented form of the prefix, bí-, big-, or in words of later formation, especially those in which by has an attributive sense, and cannot be separated by any clear line from by adj., since the use of the hyphen is very uncertain. All the principal words so formed are treated as main words in their alphabetical places; the less important and more obvious combinations here follow, under the various uses and senses of the prefix. a. Compounds in which by- is a preposition. See also by-hither, by south at by prep. 9c, by-rote adj., by ordinary, by common at by prep. 7, etc., and by hand at hand n. Phrases 1b(a). b. Compounds in which by- has an adverbial force. (a) With nouns of agent or action, with senses ‘beside, past’. (i)
Dictionaries that list by-product:
Unhyphenated (single word):
Random House Kernerman Webster's College Dictionary, © 2010 (based on Random House Unabridged)(AmE)
Collins Dictionary 12th Edition(BrE)
Cambridge Advanced Learner's Dictionary & Thesaurus(BrE)
Macmillan Dictionary Online(BrE)
Oxford Living Dictionaries(BrE)
Oxford Learner's Dictionary(BrE)
As you can see, all British dictionaries list the hyphenated version except one (Cambridge Academic Content Dictionary), and some include the unhyphenated version as an alternative. Among the American dictionaries I have found one that hyphenates the word (Merriam-Webster), two that don't (American Heritage Dictionary and Random House (which often publishes dictionaries under the Webster name)), and one (Webster’s New World College Dictionary) that lists both, however lists the unhyphenated version first.
Is this a variation between US and British English or just a matter of a "house style?"
I'd say mostly a matter of style, although if I were to speculate based on these results I'd say that byproduct without the hyphen is more likely to occur in American orthography.
Also, the labeling of each dictionary as AmE and BrE as I've done may be misleading, as some dictionaries claim to be representative of both.
Also about your question in the title about spelling it as two separate words without a hyphen "by product", don't do that. I've seen this spelling nowhere, at least among what most consider to be authoritative or respected sources.
The two most widely consulted U.S. dictionary makers currently disagree on the spelling. Merriam-Webster's Eleventh Collegiate Dictionary (2003) lists only one spelling as acceptable:
And Merriam-Webster online has shown no movement on this point in the 16 years since the Eleventh Collegiate appeared.
In contrast, The American Heritage Dictionary of the English Language, fifth edition (2010) lists two acceptable spellings:
byproduct or by-product
The fact that byproduct comes first in this entry indicates that AHDEL deems it the preferable spelling.
Interestingly, both dictionaries' current preferences represent reversals of their original preferences. The First Collegiate (1898) lists only one spelling:
But the Third Collegiate (1916) introduces the hyphen that MW has stuck with ever since:
Meanwhile, the first edition (1969) of AHDEL lists only the hyphenated form:
But the third edition (1992) switches to the two-option listing (with priority going to the closed-up form):
byproduct or by-product
So it isn't enough in the United States to say that you follow Merriam-Webster or American Heritage; you also have to specify which edition of Merriam-Webster or American Heritage.
Clearly, there is no consensus (in the United States, anyway) as to the preferred spelling of the word. If your publisher generally follows (modern) Merriam-Webster, you're bound to use by-product. If it follows AHDEL, you're more likely than not to be expected to use byproduct. But in either case, the institution's own word list may make an exception to its chosen dictionary's preference.