In general, the grammatical number (singular or plural) must agree. For your examples:
- If you have over-sized items, you must pay for them in advance.
The word items is plural, so to refer to the items, you say them.
- If you have [an] over-sized item, you must pay for it in advance.
The word item is singular, so to refer to the item, you say it. It's also grammatically possible to say him or her if you are applying a grammatical gender, which is normally done only for persons or animals in English, but it is sometimes done stylistically for inanimate objects as well; see Pronoun question: referring to inanimate objects as 'he' or 'she').
If the noun is a mass noun with no unit of measure specified, the grammatical number is treated as singular:
If you have over-sized luggage, you must pay for it in advance.
When we say a mass noun like luggage in that way, luggage and it refer to the whole collection of those things, even if it turns out that there are 3, 4, or 5 of them.
If the mass noun is combined with a unit of measure, then the grammatical number should agree with the grammatical number of the measure word:
If you have an over-sized piece of luggage, you must pay for it in advance.
If you have over-sized pieces of luggage, you must pay for them in advance.
My family, Microsoft, CNBC, etc.
Some nouns are known as collective nouns and could themselves be treated as grammatically singular or plural, depending on the exact situation (e.g. my family is big vs. my family are doctors) and depending on regional conventions (e.g. in the U.S., family is conventionally used as singular and names of organizations are conventionally treated as singular, while in the UK, names of organizations are conventionally treated as plural).
See also: Are collective nouns (and in particular companies) always given a plural verb form, or are certain ones treated as singular?
A number of, etc.
For the phrase "a number of" to indicate that there are several of something, I believe the dominant convention in all English-speaking countries is to treat the phrase/collection as grammatically plural, even though the measure word itself (number) appears to be grammatically singular. Thus,
There are a number of houses.
*There is a number of houses.
A number of things still need to be done.
*A number of things still needs to be done.
* = ungrammatical.
For the general case of "a X of Y", where X is a measure word, then you can use a singular if it makes sense (e.g. if you are counting, as in one X, two Xs, three Xs):
There is a school of fish.
I believe you can also treat it as plural, though, if it makes sense (see above point about collective nouns):
A school of fish are swimming.
"one or more"
Phrases such as "one or more" are considered plural, despite the fact that the word one is singular:
If you have one or more carry-on bags, please place them in the overhead container.
The notation n+ is often used in writing to mean "the quantity n or more". In speech, you can read this notation as "n or more" or as "n plus" if you wish. The grammatical number of the resulting phrase is plural, even if n is 1, and regardless of how you read it aloud:
If you have 1+ carry-on bags, please place them in the overhead container.
n + m, n * m, etc.
When reading out maths problems, at least in American English, the whole expression is conventionally considered singular, regardless of the quantities involved and regardless of the operation involved:
One plus one is two.
Five plus seven is twelve.
Six times seven is forty-two.
If you referring to more than one expression, or no expressions, then the expressions would be plural:
One plus one and two plus two are two examples of addition problems.
The problem is solved. There are no more expressions left.
See also: Why is "zero" followed by a plural noun?