Should the phrase "we studied it both on- and off-site" have both hyphens? Or would "we studied it both onsite and offsite" be better?
I would write the sentence without either hyphen and keeping the space between the words:
We studied it both on and off site.
There is no need to hyphenate when the phrase is used plainly and not as a modifier. Compare your sentence with the following:
We conducted both on- and off-site studies.
In the last sentence, "on-site" (implied) and "off-site" are used as modifiers of "study". If you would like another example consider the following:
I go off duty in two hours.
An off-duty cop foiled a robbery today.
In this particular context, on-site and off-site need to be hyphenated.
The word off is not a prefix as it can stand alone as a word. This is the reason, there could exist other versions depending on the context. For example: "He went off duty" but "He made an off-record confession".
That said, there may be other contexts or constructions, where a hyphen may not be required, as per the comments from the language enthusiasts here.
(As the language evolves and when more and more users (more importantly writers) begin to use what was once considered inappropriate, it becomes 'the norm' and dictionaries are updated quickly. As some of our community members pointed out here: e-mail = email, on-line = online. All one can do is to change with the changing times.)
Ngram doesn't treat hyphenated compound words as hyphenated words; instead, it treats them as two words separated by a letter space. So instances of "off-site" in a Google Books search show up in Ngram charts as examples of "off site."
Even so, this Ngram chart of off site (red line) versus offsite (blue line) indicates that, for the population of books in the Google Books database that have been published each year since 1963, instances of offsite have been more frequent than instances of off site and off-site combined:
Though I'm by no means certain that the Ngram algorithm captures all (or even most) instances of off-site, the data indicating a substantial rise in the frequency of offsite between 1940 and 2000 seems strong. And as the frequency of instances of offsite grows, the case for including offsite as a legitimate alternative to off-site and off site gets stronger. (Merriam-Webster's Eleventh Collegiate Dictionary  does not mention the spelling offsite in its entry for off-site, even as a variant.)
The Ngram results for on site (red line) and onsite (blue line) over the same period are quite different:
The two Ngram charts suggest that offsite has made considerably more headway against off-site than onsite has against on-site. At the very least, the Ngram results might justify adding a third alternative to the poster's two original options of "We studied it both on- and off-site" and "We studied it both onsite and offsite"—namely,
We studied it both on- and offsite.
Just as online began as on line, shifted over time to on-line, and eventually became fused as online, I expect that onsite and offsite will eventually become the dominant spellings of those two terms. Nevertheless, the Ngram charts above indicate that, although offsite may be well on its way to dominant status, onsite still has a serious deficit in popularity to overcome.
For what it's worth, the Chicago Manual of Style, fifteenth edition (2003), approves of hyphenating the first (truncated) term of a pair such as "on- and offsite" above:
7.89 Hyphen with word space. When the second part of a hyphenated expression is omitted, the hyphen is retained followed by a word space.
[Relevant example:] fifteen and twenty-year mortgages
Omission of the second part of a solid compound follows the same pattern.
[Relevant example:] both over- and underfed cats