In the book How to Read a Person Like a Book by Gerard I. Nierenberg and Henry H. Calero, such a smile is named after its shape, oblong, which TFD defines as:
Having the shape of or resembling a rectangle or ellipse.
An excerpt from Nierenberg and Calero describes it thus:
Smiles should not always be associated strictly with happy moments. "Beware of the oblong smile," says Dr. Ewan Grant of Birmingham University. He uses this name for the smile that many of us tend to use when we have to be polite. The lips are drawn fully back from both upper and lower teeth, forming the oblong with the lips [emphasis added]. Somehow there is no depth to this smile. "This is the smile or grimace when one is pretending to enjoy a joke or off-the-cuff remark. Or when a girl gets too much attention from a drunk, or is being chased around the office by the boss."