Melville was free to choose the definite or indefinite article here. In general, it means very similar things either way. It's hard to say why Melville chose the indefinite article, but we can guess at distinct nuances of using the indefinite article here.
A world ~ a kind of world
The paragraph before a whale describes ships avoiding a specific place because of the fear struck by the floating corpse, followed by years of habit where other ships will also avoid the place. This obedience to past habit is compared to precedent, tradition, old beliefs, and orthodoxy. It is this place (where the corpse is) or this world (a world of orthodoxy) that the ghost instills powerless panic in.
It could be that a world makes the literal or metaphorical place sound hypothetical or partial, thereby contrasting an imagined world of orthodoxy with another world ungoverned by orthodoxy. (The world beyond orthodoxy? The real?) The OED documents a few uses of "a world" under "world, n." I.1.c. that seem similar, each referring to a possible world with an implicit contrast to this world now:
c. gen. Any state or realm of existence, esp. one regarded in contrast to that of contemporary human life (sense 1a). Cf. other world n. 2, possible world at possible adj. 4.
a1797 M. Wollstonecraft Posthumous Wks. (1798) IV. lxvii. 3 A world in which self-interest..is the principal mobile.
2007 Washington Post 29 July (Home ed.) (Book World section) 5/1 Among Other Things, I've taken up Smoking is a coming-out novel about a world we don't quite live in yet, a world in which the dividing line between straight and gay looks..faint.
So a world would refer to something somewhat defamiliarized from the usual associations with the world, namely a world under the effects of an apparition.
A world as generalizing something specific
It's also possible to read this as a generalization of the effects of one whale corpse and its memory in one place. In that sense, II.6.b. in the OED fits particularly well, since many things are likened to a world:
b. In generalized sense: any material or other system likened to but usually distinct from the earth. Usually with indefinite article or in plural. Cf. universe n. 3a.
1676 J. Dryden Aureng-Zebe iii. 33 Too truly Tamerlain's Successors they, Each thinks a World too little for his sway.
1822 Ld. Byron Heaven & Earth i. iii, in Liberal 1 181 When the hot sun hath baked the reeking soil Into a world.
1917 A. Quiller-Couch Notes Shakespeare's Workmanship xvi. 294 So he sleeps, and awakes—to be rewarded..with the best earthly thing that could betide in a world he has served worldlily yet well.
In this sense, a world may help specify that this is not literally the world but rather something less definite, that could be taken for many things: a period of time, a spot of ocean, the mind of a ship crew, and so on.