There is nothing essentially wrong with using the perfect past, where it is needed. However, used in fiction, it is easy to overuse or misuse. Three uses are common; introduce new past events, flash back, and referring to past events already discussed in the story,
Introduce new past events
I had met him last June, and was eager to catch up again.
The problem with this is that a single statement does not give a detailed description. You are summarizing a whole meeting in a single sentence (referred to as telling) instead of describing in detail (showing). In fiction showing is usually preferred to telling, because of the greater reader engagement (but not always, as showing reduces the pace of the story).
To summarize the reader may be left with insufficient detail about the past event and feels "had".
Had may be used a "flash back" about a past event.
The shark came towards me. I could see the razor sharp teeth and new that I was about to die. The whiteness of the teeth reminded me of a visit to the supermarket I had made last Tuesday.
Bla bla bla about shopping.
But I would never know how good the next sale was, as the shark bit me cleanly in half.
This kind of flash back breaks the flow of the story. The reader also does not know when the flash back will end. Placing the flash back in its own scene may be a better idea.
To summarize, the reader can easily become confused by your flash back and feel "had".
Referring to past events
If you are simply referring to past events that you have already covered previously in the story, then the use of "had" is usually fine.
Using "had" forces the reader to refer backwards in time which is always going to slow the pace of the story.
Overuse of "had" can lead your writing an archaic, or even a slightly false feel.
As for all words, the word "had" and the use of the perfect past has to be justified by the benefits it brings to your story. Use wisely.