Scared, in addition to being the past-tense and past-participle form of the verb scare, is also used as a "(de)participial adjective". This is an adjective that has the form of a participle, but which is not considered to be a true (verbal) participle.
One test to distinguish verbal participles from departicipial adjectives is to see if you can use the word "very" before the word. "Very" cannot modify verbs. It is possible to say "I've been very scared of…”, which indicates that it is at least possible that "scared" in your original sentence is an adjective and not a verb.
Furthermore, as you said, you can't add a "by phrase" to "I was scared of..." the way that you could with a real passive sentence.
So it looks like "I've been scared of..." has the same structure as a sentence like "I've been anxious about...", with the present perfect form of the copular verb be followed by an adjective.