Both will be understood easily enough, but I would say you actually used the best option (for your specific example) in your question:
Use the correct font size for the browser.
More generally, it's slightly more correct/common to use in when referring to the context of a computer program. This is because the program has an interface (for a browser, its tabs and controls) and the thing you're describing is contained by that interface.
By contrast, when referring to a particular website, on is more common, e.g. "on Facebook" or "on English.SE". Also, people who are using a computer are usually described as "on the computer" as opposed to 'in' or 'at' the computer.