I am not sure about 'rule', but there is a guide and it is down to perception.
When the items can be perceived as individuals, it is plural. When the items cannot be perceived as individuals, they are literally uncountable and hence singular.
When you look in a sack of seed you don't really see individuals, and they are beyond counting, so it is seed. Sand, or powder, is a more extreme example. In contrast, if you have three seeds in your hand, you see them as individuals and easily countable, so it would be three seeds in your hand.
Compare this with, say, a sack of potatoes, where the individual potatoes are easily discerned.
As for 'fish', one explanation I have seen is that when they are animals (in other words they are alive) the plural is 'fish'; when they are food (in other words they are dead) the plural is 'fishes'. This explains why we have all the fish in the sea but five loaves and two fishes.
There are alternative explanations, of course.
People is inherently plural - the singular is 'person'.