I don't exactly know how to describe it, but I've heard of this happening in English before. I'm pretty sure the word 'newt' is an example of this. From what I've heard, the word used to be 'ewt', having the indefinite article 'an', but pronunciation/scholars/ I don't know caused the word to become 'newt', having the indefinite article change from 'an' to 'a'. I think something similar happened to the phrase 'an other'. Sometimes I hear phrases using the word 'nother' like 'a whole nother', but that may be informal. I know that 'another' is an accepted word, but I'm 99% sure it came from the phrase 'an other', but pronunciation/scholars/ I don't know changed it to 'another'. What is this phenomenon called?
OED answers this question at newt:
Variant of eft n.1 (see forms at that entry) with metanalysis (see N n.).
From the beginning of the Middle English period, the coexistence of two forms of the indefinite article (an before vowels and a before consonants) often led to metanalysis (a similar phenomenon occurs in other languages where the indefinite article ends in -n, e.g. Dutch, German, French, Italian, etc.). Variants arising by metanalysis sometimes alliterate with words with initial n- in alliterative verse, and in some cases have become established as the regular modern forms (so that e.g. Middle English a nadder became an adder; compare also apron, auger, etc.), while conversely some vowel-initial forms have gained n- (so that e.g. Middle English an ewt became a newt; compare also nickname).
As an earlier answer correctly responded, 'newt' resulted from metanalysis. The rest of a complete answer, regarding 'nother' and 'another', is not entirely other. So, you state
I think something similar happened to the phrase 'an other'.
The phrase 'an other' became 'another' by compounding, no metanalysis involved:
Origin: Formed within English, by compounding.
(From OEDO 'another', entry updated for OED3 June 2022.)
Yet you also seem interested in 'nother', and particularly the phrase 'a whole nother'. So, you also say
Sometimes I hear phrases using the word 'nother' like 'a whole nother', but that may be informal.
You are correct, 'nother' has sometimes been used informally (also known as colloquially) in "positive contexts, in senses of other: different; (the) other (of two alternatives); additional." [OEDO 'nother', A. adj.2 (determiner)]. Recently, this use is "chiefly in a whole nother" (the earliest quote in OEDO using the phrase is from 1910, the latest 1963), and metanalysis produced this 'nother' as a variant of 'another' (op. cit.; the latest quote in OEDO using the bare word in this sense is contextual evidence from the Dictionary of Caribbean English, 1996, reprinted in 2003).
However, depending on which phrases you heard ("sometimes I hear phrases" you said), 'nother' may be the result not of metanalysis or compounding, but rather the result of a third linguistic process, shortening, or of a fourth process, aphesis.
Shortening produced another still in use 'nother', with the sense "neither" (OEDO, entry updated for OED3 December 2003, 'nother', "B. adj.1 (determiner)"). OEDO says this sense of 'nother' is frequently found in the phrase "neither nother.... In later use English regional and Caribbean" (op. cit.).
I don't know if you've heard this phrase, 'neither nother', or 'nother' used in the sense of 'neither', and while it seems unlikely that you have, I hesitate to rule any possibility out without evidence one way or 'nother.
Aphesis produced "'nother, pron.3 and adj.3" as a pronoun in the sense "[c]ontrasted with one...[a]nother one, a different one" as well as in the phrase "one 'nother", and as a determiner in the sense "[a]dditional, further" (OEDO, entry updated for OED3 December 2003).
The linguistic process of aphesis consists of the "gradual and unintentional loss of a short unaccented vowel at the beginning of a word" (OEDO, 'aphesis', "not yet fully updated").
Note that other senses of the shortened 'nother' than are discussed in the foregoing exist; I have omitted discussion of obsolete senses and senses I consider unimportant with respect to your question.