I will interpret your question narrowly, that is, as really being about grammatical tense as opposed to other marked verbal1 systems such as aspect and mood. Even so, I will have quite a bit to say about aspect and mood.
1Verbal as in 'having to do with verbs'.
Answers to many questions will be more complicated or less complicated depending on whether you really have in mind the narrow technical sense of 'tense' or if you rather mean any marked verbal system (what some refer to as tense-aspect-mood). For example, in English, there are only four tenses in the strict sense (at least according to CGEL (p. 115)), but, by one count, there are as many as 88 'tenses' if the term is interpreted broadly.
Technically, grammatical tense is a verbal system
i. which is marked either inflectionally on the verb or analytically (e.g. by auxiliaries)
and where the basic or characteristic meaning of the terms
ii. is to locate the situation, or part of it, at some point or period of time
(CGEL, pp. 115-116).
As is often the case, however, in order to really explain a complicated concept, one must specify not only what it is, but also what it isn't. That is to say, one should explain what are some categories with which it might be confused. In the case of grammatical tense, there is possible confusion with other marked verbal systems, notably aspect and mood. Indeed, some people talk of tense–aspect–mood, since it is often difficult to untangle these features of a language.
Often any two of tense, aspect, and mood (or all three) may be conveyed by a single grammatical construction, but this system may not be complete in that not all possible combinations may have an available construction. In other cases there may not be clearly delineated categories of tense and mood, or aspect and mood. For instance, many Indo-European languages do not clearly distinguish tense from aspect. In some languages, such as Spanish and Modern Greek, the imperfective aspect is fused with the past tense in a form traditionally called the imperfect. Other languages with distinct past imperfectives include Latin and Persian. (source)
Contrast with aspect and mood
Aspect and mood satisfy i. but replace ii. by something else. In the case of aspect, that something else is ii. 'has to do with the internal temporal constituency of the situation' (CGEL, p. 117). Another way to describe aspect is to say that it relates to 'other temporal information, such as duration, completion, or frequency, as it relates to the time of action. Thus, tense refers to temporally when while aspect refers to temporally how' (source).
CGEL takes the English aspect to contrast the progressive vs the non-progressive (p. 117):
The term aspect applies to a system where the basic meanings have to do with the internal temporal constituency of the situation. Compare:
 PRESENT PRETERITE
non-progressive She goes to school. She went to school.
progressive She is going to school. She was going to school.
If we change present tense She goes to school to preterite She went to school we change from present to past time, but if we change it to She is going to school the time (and the tense) remains the same. The difference is a matter of how the speaker views the situation. The progressive takes an internal view, looking at it from the inside, as it were, as something ongoing, in progress. The unmarked, non-progressive, version takes an external view: there is no explicit reference to any internal phase or to any feature of the temporal flow (such as whether the situation is conceived of as instantaneous or having duration through time).
There are a considerable number of verbs which express aspectual meaning. Begin and finish, for example, are semantically similar to progressive be in that they too take an internal view, focusing on the initial or final phase of the situation. They do not belong to the grammatical class of auxiliary verbs, however, and we will refer to them therefore as lexical aspectual verbs.
About mood, CGEL says (p. 117)
It is not so easy to give a succinct account of the kind of meaning characteristically associated with mood. The unmarked mood is associated with factual assertions, whereas the marked terms involve various kinds of non-factuality or non-actuality, indicating that the situation is merely possible, is predicted or inferred rather than known, and so on.
Mood will be very relevant to the discussion of the (lack of a) future tense in English, below.
English tenses according to CGEL
CGEL says (p. 116) that English has two tense systems: the primary one, which contrasts the preterite and the present, and the secondary one, which contrasts the perfect and the non-perfect. All four combinations are possible:
non-perfect went goes
perfect had gone have gone
The general term tense applies to a system where the basic or characteristic meaning of the terms is to locate the situation, or part of it, at some point or period of time. English has two tense systems, illustrated in:
 i a. She went to school. b. She goes to school. [preterite vs present]
ii a. He may have known her. b. He may know her. [perfect vs non-perfect]
In [i], the verb-forms refer respectively to past and present time, so that this is a very obvious example of a contrast of tense. We saw in §1.7 that the same inflectional contrast can convey other meanings too. In If she went with us tomorrow she'd have plenty of time vs If she goes with us tomorrow she'll have plenty of time, for example, the time is the same (future) in both, the difference being that the first presents her going with us as a more remote possibility than the second. But this meaning difference is found only in a restricted range of constructions, such as conditionals: it is examples like [i] that illustrate the basic meanings of the forms.
In [2ii] the perfect auxiliary have (in combination with the following past participle) serves to locate the knowing in past time and given that this can again be regarded as the basic meaning we analyse have as a tense auxiliary. We shall see that the inflectional preterite and the analytic perfect have a great deal in common, and we shall use the term past tense to cover them both: for reasons explained in §6.3, we take the inflectional preterite to be the primary past tense and the perfect the secondary past tense.
To understand CGEL's reasoning in §6.3, we first need to explain several concepts.
Relation between time referred to and time of orientation
The semantic categories past, present, and future are inherently relational: one time is defined by its relation to another. Consider such examples as:
 i He died of lung cancer. [past time]
ii I promise to let you have it back tomorrow. [present time ]
iii If you see her tomorrow give her my best regards. [future time]
When we say of [i], for example, that the time of dying is past, this is understood as a time earlier than now, than the time at which I utter the sentence. We will speak of the two terms in the relation as the time referred to, symbolised Tr, and the time of orientation, symbolised To. In [i], Tr is the time of dying, To is the time of utterance, and the relation is "earlier than", or to use a more technical term "anterior to". In [ii], Tr is the time of promising, To again the time of utterance, and the relation is "simultaneous with". In [iii], Tr is the time of your seeing her, To the time of utterance, and the relation is "later than", or "posterior to". These three relations may be symbolised as shown in:
 i Past time Tr anterior to To Tr < To
ii Present time Tr simultaneous with To Tr = To
iii Future time Tr posterior to To Tr > To
In addition to Tr and To, there are two other 'times', of which we will only need one: the deictic time, Td. The word deictic comes from deixis, from the Ancient Greek word which means 'pointing, indicating, reference'. Deixis has to do with the 'reference point' from which the speaker 'points' to the things he or she is speaking about. Actually, the description of the deictic time will help with understanding of what deixis is about.
Time of orientation and deictic time
For primary tense To is normally the time of speaking or writing. We use the term deictic time to allow for the fact that in special circumstances it can be the time of decoding rather than that of encoding. Compare:
 i I am writing this letter while the boys are at school. [To is time of encoding]
ii You are now leaving West Berlin. [a written notice] [To is time of decoding]
In ordinary speech the time of encoding and the time of decoding are identical, but in writing they can be different. Where this is so, the default identification of To, as in [i], is with the time of encoding, the writer's time, but in notices like [ii] it is identified as the time of decoding, the addressee's time.29 The difference between these is not marked linguistically in any way and the term deictic time covers both cases: it is defined by the linguistic event itself.
29The same applies to recorded speech, as for example in radio broadcasts—You have been listening to 'The Goon Show'.
To, we have said, is normally identified as Td; in this case, illustrated in , we say that the tense is interpreted deictically. But it is not invariably interpreted in this way. Consider:
 i If she beats him he'll claim she cheated. [non-deictic past]
ii If you eat any more you'll say you don't want any tea. [non-deictic present]
The preterite and present tense inflections on cheat and do indicate that Tr is respectively anterior to and simultaneous with To, but here To is clearly not Td. The time of the (possible) cheating is not anterior to the time of my uttering [5i], but to the time of his (possibly) making a claim of cheating. Similarly in [5 ii] the time of your not wanting any tea is not simultaneous with my utterance but with your future utterance. This is why we need to distinguish the concepts of To and Td: they do not necessarily coincide, even though they usually do.
Finally, we are ready to explain...
6.3 Primary vs secondary past tense
We have been referring to the preterite and the perfect as respectively primary and secondary past tenses; there are three reasons for distinguishing them in this way.
(a) Relation to Td
In languages in general, tense systems prototypically locate Tr relative to To/Td, i.e. they are deictic, and we have seen that the English preterite is most often interpreted in this way, whereas the perfect is generally non-deictic. The preterite is thus a clearer instance of a tense than is the perfect. Moreover, when they combine in the preterite perfect to express double anteriority, it is the preterite that encodes the first move back from Td, while the perfect encodes a further move back beyond that: the primary/secondary contrast has been shown in our notation by the superscript numerals, as in [44-45] of §5.
(b) Degree of grammaticalisation
The primary tense system is more highly grammaticalised than the secondary one. One obvious reflection of this is that it is marked inflectionally rather than analytically. The perfect marker have is a member of the small closed class of auxiliary verbs, so that the perfect can properly be regarded as a grammatical category, but analytic marking of this kind represents a lesser degree of grammaticalisation than inflection. No less important, however, is the fact that the preterite is in contrast with another tense, the present, whereas the perfect merely contrasts with its absence. The present perfect and the preterite perfect are compound tenses (involving two Tr-To relations), whereas the present non-perfect and the preterite non-perfect are simple tenses (involving a single Tr-To relation): non-perfect is not a tense. The present tense is distinct from the absence of primary tense (She is/They are ill vs She is/They are believed to be ill), but there is no such distinction with secondary tense. A third factor commonly involved in grammaticalisation is discrepancy between form and meaning: highly grammaticalised elements tend to develop uses which depart from their core meaning. The preterite is used for modal remoteness and to indicate backshift as well as with its core meaning of locating Tr anterior to To, whereas the perfect is almost restricted to this latter use.
(c) Anteriority expressed by the perfect when the preterite not available
We have noted that when the preterite is used for modal remoteness or in backshift (to locate To), the perfect takes over the role of expressing the anteriority of Tr relative to To: see . It similarly has this role in non-finite clauses, which lack primary tense altogether: compare finite He died in 1806 with non-finite He is believed to have died in 1806.
Controversy 1: the perfect
Now the problem is that the division between tense, aspect, and mood is not always clear-cut. In English, one point of controversy is what to do with the perfect. In some languages, the perfect is clearly an aspect, whereas in English some authorities say it is an aspect, others say it is a tense, and still others say it is neither one of those but rather that it something called a phase (CGEL, p. 116). As I explained above, CGEL's take is that the contrast perfect vs. non-perfect constitutes a secondary tense system in English.
Controversy 2: the future tense
Another point of controversy in English is the future tense (or the lack thereof). This time the fuzzy boundary is between tense and mood, and CGEL, like many other authorities, says that the construction will/shall + plain form of the verb does not correspond to a future tense, but rather that will/shall are auxilliaries of mood, not tense. Here I will reproduce CGEL's argument in full (pp. 209-210):
Traditional grammar treats will (and, in the 1st person, shall) as a future tense auxiliary, proposing a tense system with three terms:
 PAST PRESENT FUTURE
took takes will take [traditional tense system]
The view taken here, by contrast, is that while there are numerous ways of indicating future time, there is no grammatical category that can properly be analysed as a future tense. More particularly we argue that will (and likewise shall) is an auxiliary of mood, not tense.
The case against the traditional analysis
(a) The three-term system does not cater for the relation between will and would
One major argument against  is that would is the preterite counterpart of will. The relation between would and will is just like that between could and can. We have distinguished three uses for the preterite (past time, backshift, and modal remoteness), and would is found in all three, as seen in §9.8. Will take, therefore, does not belong in a one-dimensional system with took and takes any more than has taken does: the contrast between preterite and present is independent of the presence or absence of will, just as it is independent of the presence or absence of have. Even if we provisionally accept that will is a future tense auxiliary,  must be modified so as to allow for two dimensions of contrast:
 PAST PRESENT
NON-FUTURE took takes
FUTURE would take will take
(b) Will belongs grammatically with can, may, must, etc.
We have seen (§2.4) that a whole cluster of grammatical properties distinguish can, may, must, and a few more from the other verbs of English. They constitute a syntactic class whose central members are strongly differentiated from ordinary verbs—and will belongs very clearly among these central members. This argument is not itself decisive: it would in principle be possible to say that the verbs in question formed a class of tense/mood auxiliaries. But it does place the onus of proof on defenders of the future tense analysis to demonstrate why will (and shall) should be differentiated from the others as tense auxiliaries vs mood auxiliaries.
(c) Will belongs semantically with can, may, must, etc.
The survey in §9 shows that will belongs in the same semantic area as the uncontroversial modal auxiliaries, and the same applies to shall. The difference in interpretation between a simple present tense and its counterpart with will is to a very large extent a matter of modality. Compare, for example:
 PRESENT TIME FUTURE TIME
simple present That is the doctor. They meet in the final in May.
will + plain form That will be the doctor. They will meet in the final in May.
In each pair the time is the same, but the version with will is epistemically weaker than the simple present. Note also that all of the auxiliaries in question can be used with situations that are in past, present, or future time. Compare, then will and may in:
 PAST TIME PRESENT TIME FUTURE TIME
i He will have left already. He will be in Paris now. He will see her tomorrow.
ii He may have left already. He may be in Paris now. He may see her tomorrow.
For [ii] the past time interpretation is attributable to perfect have, while the present and future time interpretations can be accounted for in terms of the mechanism discussed in §7: may allows either a present or future interpretation to be assigned to its complement. Under the analysis proposed for will, [i] will be handled in exactly the same way.
Notice, moreover, that may and will are themselves present tense forms. In examples like  this present tense encodes the time of the modal judgement, and it is possible for the present tense modal to be modified by the time adjunct now: Now we will/may not be in time to see the start. And in dynamic uses of will this present time meaning tends to be more salient. In I've asked him to help us but he won't, for example, won't indicates his present disposition (compare but he can't, indicating present inability), and in This door won't lock I am talking about the present properties of the door (compare You can't lock this door).
Data like that shown in  does not by itself refute a future tense analysis: there are languages (such as French) where the translation equivalents do contrast as present vs future tense. What is decisive is the combination of arguments (a)-(c), and the extent of the grammatical and semantic likeness involved in (b) and (c). If one looks at the verbal system of English without any preconception that the tripartite division between past, present, and future time will inevitably be reflected in a system of three corresponding tenses, then the evidence is overwhelming for grouping will, shall, may, can, must, etc., together as auxiliaries of the same kind.