If you conceive of returns as occurring along a bell curve rising to a single maximum point and then falling from there, the high point in the curve is the point of diminishing returns—that is, the point at which the returns stop growing and (immediately thereafter) begin to shrink.
This is the supposed insight at the core of the famous "Laffer curve," where the curve in question purports to track tax revenue based on different tax rates and their effects on taxpayers' inclination to earn taxable income. In the Wikipedia article about the Laffer curve, the point at which returns cease to increase and begin to diminish is called the "maximum revenue point."
I suspect, however, that most people who hear or use the phrase "point of diminishing returns" do not immediately visualize it as the apex of a roughly parabolic curve; instead, they probably think of a point at which the value of the returns has become disappointingly small. Such a point is likely to be fairly far down the flank of the curve from the apex, and might be thought of in practical terms as the "point of abandonment." After all, the hardest part of halting any endeavor at the true "point of diminishing returns" is recognizing that point for what it is: unlike, say, a mountain peak, an economic apex isn't easy to identify until you've gone a considerable way down the other side.