The OED gives buggerlugs and bug-a-lugs at the same entry.
Bugger started off as "Bogomil" - a member of a 13th century Bulgarian sect. The sect was heretical and, in order to villianise them, the Catholic Church's propaganda against them accused them of homosexual practices. Bugger then took the meaning of an active homosexual.
In this meaning it was a very offensive word, but by the mid-19th century it had lost a lot of its force when used colloquially and tended to be a rather friendly form, and it is probable that this is because of confusion with "beggar" in the sense of someone who deserves sympathy. It is certainly understood that way and has no sexual connotations.
Currently, "bugger" isn't used to describe heretics. It is very rare as a noun to mean "someone who penetrates in anal sex" It is used with a qualifying adjective as a colloquial/slang that indicates its meaning as a mild invective or endearment.
bugger, n.1 Origin: A borrowing from French. Etymon: French bugre. Etymology: < Anglo-Norman and Middle French bugre, bougre (Chiefly British, Australian, and New Zealand.)
1. Usually with the and capital initial. A heretic; spec. a member of the Albigenses (Albigenses n.). In later use historical and rare.
1340 Ayenbite (1866) 19 He ne belefþ þet he ssolde, ase deþ þe bougre and þe heretike. [...]
1753 Chambers's Cycl. Suppl. The Buggers are mentioned by Matthew Paris..under the name of Bugares... They were strenuously refuted by Fr. Robert, a dominican, surnamed the Bugger, as having formerly made profession of this heresy. [...]
1989 W. Weaver tr. U. Eco Foucault's Pendulum xviii. 126 The bougres were simply Bogomils, Cathars of Bulgarian origin.
The OED notes
"In the Middle Ages, anal sex and various other sexual practices which were deprecated (including bestiality, pederasty, and incest) were attributed to members of various, especially dualist, heresies, such as the Bogomils and the Albigenses (or Cathars). Such charges were common during the period of the Albigensian Crusade (1209–29),"
The following is an example of "bugger" used to mean both a heretic, a homosexual, and as an insult:
1694 in T. Brown & J. Savage tr. C. de St. Évremond Misc. Ess. II. sig. Mm2v I clapt the Pistol to his Head, when a certain Bougre of a Jesuit [Fr. un B.. de Jesuite] pusht my Arm aside, and hinder'd my Design.
This then gave rise to the more general term
2. A person who penetrates the anus of someone during sexual intercourse; a person who performs any act classed as ‘buggery’. Chiefly as a term of abuse or contempt. Cf. buggerer n. Now generally considered offensive but also somewhat archaic in this sense. The term formerly had a more or less taboo status (especially in 19th cent. British use) which is now weakened or lost in most contexts, ...
1540 J. Palsgrave tr. G. Gnapheus Comedye of Acolastus ii. iii. sig. Liiiv He is a bougour [L. cinaedus] or one that is paste shame... He is an ympe of the stewes.
1965 Times 25 May 16/2 Lord Goddard said..if this bill [i.e. the 1965 UK Sexual Offences Bill] goes through it would be a charter for these buggers' clubs.
3.a. slang. Used as a term of abuse or contempt for a person, [animal or thing].
1955 Times 27 Jan. 6/5 A remark of the policeman to him was: ‘Don't argue, get those buggers out of here.’
Now more common in weakened use in sense 3b.
3b. slang and colloquial. Used familiarly or playfully to express affection, compassion, etc.: a person or (occasionally) an animal. Chiefly with modifying adjective as little, lazy, poor, silly, etc. Cf. beggar n. 6b, beggar n.
1854 M. J. Holmes Tempest & Sunshine 203 ‘If I'd known all you city buggers was comin' I'd a kivered my bar feet’.
2014 Sydney Morning Herald (Nexis) 31 May 38 The young of today are just a bunch of lazy buggers who've never had it so good.
Now to Lug
Lug – a stick or pole 1. A long stick or pole; the branch or limb of a tree. (See also log n.1 1d.)
a1250 Owl & Nightingale 1609 An evereuch man is widh me wrodh, An me mid stone and lugge threteth.
1853 Jrnl. Royal Agric. Soc. 14 ii. 441 In Herefordshire the ordinary mode of gathering the fruit is by sending men to beat the trees with long slender poles or rods,..these poles are provincially termed ‘polting lugs’.
Lug - Chiefly Scottish and northern.
1. One of the flaps or lappets of a cap or bonnet, covering the ears.
1495 in T. Dickson Accts. Treasurer Scotl. (1877) I. 225 Item, fra Henry Cant, ij cappis wyth luggis; price xxxvjs.
Lug 3. an object resembling the external ear
a. The handle of a pitcher, etc. Also technical in various uses, denoting an appendage by which an object may be lifted or suspended;
1624 Fairfax Inventory in Archaeologia (1884) 48 151 One copper pan with 2 lugges.
Lug = ear n. Now colloquial or jocular.
1916 ‘Taffrail’ Pincher Martin ii. 28 Give 'im a clip under the lug!
It is very doubtful that the bugger in "buggerlugs" is taken in any of the above meanings: Buggers (of any description/meaning) are not known for having particularly distinctive ears, so "Buggerlugs" would not seem to be the best variant, and thus "bug-a-lugs" seems to be the original and gives the clue as to the origin:
"bug-a-lugs" slang (originally Nautical).
1. In singular. Chiefly English regional (Dorset) and in form bugalug. A likeness or model of a person, typically a person who is hated; an effigy. Also occasionally: something contemptible or undesirable. Now rare.
1839 Wabash Enquirer (Terre Haute, Indiana) 23 Aug. 3/4 Stout's statue of Queen Victoria..was completely destroyed..having slipped..into the hold of one of the Boston packets, whither it was destined... The editor of the Boston Gazette has thus lost his chance of falling in love with her buggerlug!
1886 W. Barnes Gloss. Dorset Dial. 52 Bug-a-lug, an effigy, a bugbear. A scarecrow or gally-bagger. It means a bug or bugbear on a pole; a-lug meaning on a lug or pole.
1983 G. Morley Smuggling in Hampshire & Dorset, 1700–1850 xi. 174 On more than one occasion an over-zealous Customs Officer was warned off permanently by having..his ‘bugalug’, or effigy, burnt in front of his house.
It is unfortunate that the first quote (1839) is "buggerlug". Note that this is American English and I suspect that it is a misspelling. Colloquialisms usually take a long time to be written down and when they are, they are often done phonetically.
The 1886 quote gives the meaning, and in the 1983 quote the author seems to have found an old use.
A bug =
2. A self-important, pompous, or conceited person; a pre-eminent or powerful person. In later use only in big bug n.
1536 T. Revel tr. F. Lambert Summe Christianitie sig. A.viiv The symple people, have the judgement of the spirit above all such proud bugges [L. larvas] the which we call Magistrinostri.
2004 B. Dylan Chronicles I. iii. 115 The big bugs in the press kept promoting me as the mouthpiece, spokesman, or even conscience of a generation.
Additionally, "Bug" has the meaning of
bug, n.1 An imaginary evil spirit or creature; a bogeyman. Also: an object or source of (esp. needless) fear or dread; an imaginary terror. Cf. scare-bug n., bugbear n. 2. Now rare.
In quot. a1425: a scarecrow.
a1425 (▸c1395) Bible (Wycliffite, L.V.) (Royal) (1850) Baruch vi. 69 (MED) As a bugge either a man of raggis [L. formido] in a place where gourdis wexen kepith no thing, so ben her goddis of tree.
1535 Bible (Coverdale) Psalms xc[i]. 5 Thou shalt not nede to be afrayed for eny bugges by night.
1915 D. Beard Amer. Boys' Bk. of Bugs, Butterflies & Beetles 7 Bugs..stood for some imaginary hobgoblins or terrible nightmare things which never had any existence out of dreamland.
This leaves only "-a-"
A - (prefix) 2. With nouns and verb stems, forming adverbs (and derived adjectives and prepositions) expressing activity, position, condition, etc. Now chiefly poetic.A selection of rarely occurring formations is given below.
1673 J. Arrowsmith Reformation Prol. Who with Religions Twang and Mouth a-splay / Should Conventicle now instead of play.
1999 Esquire Mar. 33/3 Cynics often wonder, aloud, and asmirk.
Bug-a-lugs has lost all connection with its origins, and is now a regional word. The exact intent of its meaning is given by the context, but is somewhat similar to "bugger/beggar" above.
From the above, and although little is certain, I would favour the idea that "buggerlugs" is a version of "bug-a-lug" = a scarecrow (or other egify supported by a stick or pole.)
I think it is possible to see the connection between the self-important, pompous, or conceited person; a pre-eminent or powerful person and an object or source of fear or dread in the quote of 1983 The customs officer was the one who would arrest them (smugglers), and so they feared him.
 The idea of a scarecrow is to put fear into crows - hence the like with "bug" - the an object or source of (esp. needless) fear or dread. On the other hand, humans see scarecrows in a positive light, although a little scruffy.