As nouns, in many contexts, iterate and iteration are synonymous.
The definition given by Wiktionary for iterate (noun) encompasses not just a quantity, but a mathematical concept:
(mathematics) a function that iterates
f2(x0) is the second iterate of x0 under f.
(PlanetMath has a more detailed mathematical description; they show the nouns iterate and iteration as synonyms for this idea.)
Each iterate refers to each subsequent result of iteration (xk+1, in the paper you mentioned).
So, if an iterate is the result of iteration, what is an iteration?
Wiktionary tells us that it can mean a single repetition, which is what makes the two nearly synonymous. This could be used in math, computer science, or indeed in art or music or writing, meaning:
A variation of a design.
Iteration also means the act or use of repetition, and it has a domain-specific meaning you alluded to:
(computing) the use of repetition in a computer program, especially in the form of a loop
So a computing iteration sounds a lot like a mathematical iterate, except that iteration is much more common; I'll certainly say:
Hmm. It failed on the fourth iteration.
The iteration in my for loop isn't working!