This is what The Cambridge Grammar of the English Language (p. 1586, The alternation between ·s and ·es) has to say about it:
With bases ending in o, where o does not follow a consonant symbol (i.e. where it is preceded by a vowel or is part of the composite vowel symbol oo), the plural takes ·s:
bamboos, cameos, embryos, folios, kangaroos, patios, radios, studios, zoos
Where o does follow a consonant, the plural has to be specified for the lexeme concerned. There are three classses:
i. ·es only: echo ~ echoes. Also domino, embargo, hero, mango, negro, potato, tomato, torpedo, veto
ii. ·s or ·es: motto ~ mottos/mottoes. Also, archipelago, banjo, buffalo, cargo, dado, dodo, grotto, halo, innuendo, manifesto, mulatto, proviso, tornado, volcano
iii. ·s only: bistro ~ bistros. Also calypso, do, dynamo, beano; clippings such as demo, kilo, memo, photo; nouns of Italian origin: cello, concerto, contralto, libretto, maestro, piano, quarto, solo, soprano, virtuoso; and names of ethnic groups: Chicano, Eskimo, Filipino, Texano.
Cargo and volcano are marginal members of class [ii]: they usually take ·es, but the forms cargos and volcanos are sometimes found.
As an additional rule of thumb, almost of all the exceptions for the consonant + o + s = es rule seem to involve plural nouns where there is no homophonous verb, as per CGEL, p. 1580.
For class [ii], it might be useful to use Google Books Ngram Viewer to assess the ·s/·es distribution, and to go with the ending that seems to be preferred in literary sources. For instance,
Google Books Ngram Viewer — mottos vs mottoes — English
Google Books Ngram Viewer — banjos vs banjoes — English