The "whether" clause is nonrestrictive and should thus be surrounded by paired punctuation. That typically involves dashes, commas, or parentheses:
If your school doesn't offer AP courses, we recommend that you take the most challenging courses you can—whether they be honors, IB, or even dual enrollment courses.
If your school doesn't offer AP courses, we recommend that you take the most challenging courses you can, whether they be honors, IB, or even dual enrollment courses.
If your school doesn't offer AP courses, we recommend that you take the most challenging courses you can (whether they be honors, IB, or even dual enrollment courses).
The second dash or comma is omitted because it would be adjacent to the terminal period. (Parentheses are never omitted.)
As Edwin Ashworth notes in a comment, the offsetting comma might be confused with the following serial commas. (I don't mind it, but others might.) Dashes are often considered the strongest of the three options—i.e., representing a relatively sharp break—while commas are often considered the weakest.
My standard caveat is that people have a wide variety of opinions about punctuation—and especially commas—so this is certainly not the only possible approach.