I looked up Webster, Oxford and some other sources and fund nothing for the meaning of "count off" in the followig sentence.
Each track on the CDs that accompanies the book is presented with an intro, count off, occasional instructions and piano part. The track is immediately followed by a version without count offs, instructions, or piano.