What you want to know is the antecedent of the relative pronoun that, which will determine whether the verb in the relative clause is was or were. This hardly sounds like rocket science, except for a grammar rule one is tempted to misapply here.
One of the roughly hundred students who were to be awarded a certificate at yesterday's ceremony was unfortunately stranded at the airport.
At some point in our schooling, we were bombarded with sentences with partitive constructions like this where the conjugated verb is so far from the word one that it's easy to forget the verb must be singular and not plural for students. So the rule we internalized tells us "one of many = singular verb."
In your example, however, there were multiple balls on the lawn and other balls elsewhere. Otherwise the clause would be non-restrictive. Of these balls that were on the lawn, John picked up one, yielding:
John picked up one of the balls that were on the lawn.
I can read this sentence any number of times and it still sounds wrong. That's because the rule I mentioned earlier involuntarily kicks in. Using was here is a hypercorrection: applying a grammar rule drilled into our heads by an army of English teachers in a situation where it doesn't apply.
The other factor is that a normal mortal would likely say:
John picked up one of the balls on the lawn.
and be done with it.