It can go either way in different contexts.
If the court had insisted that everyone keep a straight face, the meaning of it is what you're supposing: that the court is, in the time frame referenced, insisting that portrait subjects remain serious in their portraits.
If the court really had insisted that everyone kept a straight face, the object of the instance could be an action in the past and be correct. In this sense, kept could be used to explain that the court had insisted that everyone had remained serious during [during an implicitly referenced event], possibly because someone was questioning or criticizing the court on a point to the contrary. This isn't what's intended by use of the word in the article.
It's important to remember that the tense of the verb that is being performed by your subject does not have to agree in tense with the object of that verb (if the verb is transitive, like in this case). For example, if I'm talking about a baseball game I played last week, it would sound ridiculous to say something like, "In the second inning, coach insisted that I stole a base." The reason for this is simple: the verb "insisted" indicates that the insisting was done in the past, but the object of the verb is something that happened in the present of that context. "Coach insisted that I steal a base," is the correct way to say it because the object of my coach's insistence is a mandate, and we typically use the present tense of verbs for mandates. ("Eat all your veggies," "Don't run with scissors!", "Visit your grandparents on Saturday.")
For the second context I mention, the word "kept" as the object of the verb "insisted" isn't serving as a mandate--it's serving as a regular verb. If I insist that I did something, I'm insisting (in the present) that I did something (in the past, relative to the context of the insistence happening in the present). If I insisted that I will do something, I insisted (in the past) that I will do something (in the future, relative to the context of the insistence in the past). And so on. I would only use "kept" as the object of the verb like in the question if I were emphasizing that someone kept something by insisting on it.