One hint when trying to figure out issues such as these: draw brackets to figure out part-whole relationships between the phrases (and sub-phrases).
... dialog with an ok and cancel button... [incorrect]
... dialog with {[(an ok) and (a cancel)] button}... [correct]
... dialog with an ok and cancel buttons... [incorrect]
... dialog with an ok and cancel buttons... [incorrect]
... dialog with {[ok and cancel] buttons}... [correct]
What you have in 2. is a kind of ellipsis for "... dialog with an ok [button] and a cancel button..."
Think of it this way:
(A catholic X and a protestant X)[pastor] went into a bar...
Here, the text in the square brackets replace the Xs in the phrase in round brackets. Given that you have marked 1. as correct, you might think that the second "a" is unnecessary. Try with this now:
(A blind X and a deaf X)[pastor] went into a bar...
If the second determiner is omitted, the phrase needs to be analysed differently:
A ({blind and deaf} X)[pastor] went into a bar...