Aside from pastime, what is a word or phrase that can be used to describe an activity that is not pleasurable and used only as a diversion while letting time pass?
The noun "pastime" actually describes activity that is pleasurable.
an activity that someone does regularly for enjoyment rather than work; a hobby: his favorite pastimes were shooting and golf.
To describe an activity that is used only as a diversion while letting time pass, you can say you are doing something to kill time.
to do something which is not very useful or interesting while you are waiting for time to pass
Or you can just say you are idling, or, lollygagging.
spend time aimlessly: he sends her to Arizona every January to lollygag in the sun.
Personally I don't think there is any word that describes an activity which is not pleasurable and used only as a diversion while letting time pass as when you want to kill time, you don't do things that are particularly not pleasurable, but things that have no purpose.
A phrase I hear all the time in the software industry is a "time suck". Especially in reference to meetings, which are also frequently referred to as "time wasters" which is essentially the same thing, time suck just sounds better.