You are correct that your modified version is more pleasant to the ear. It's less verbose.
"I am very fortunate to be able to have visited other countries such as Colombia or Brazil."
"I am very fortunate to have been able to visit other countries such as Colombia or Brazil."
"I was very fortunate to be able to visit other countries such as Colombia or Brazil."
EDITS: I am making edits per the comments b/c someone really wanted to point out that the first option above is not technically correct. I agree it's not perfectly structured English b/c of the tense of "have visited" with "to be able to". Technically speaking it should be "was able to" and the tense of visited changed to visit dropping have.
The second OR third sentences above are correct. The trouble with the first is the fact that you can't have these together: "have visited" (past) and "to be able to" (present). It will pass muster in AmE if you said it aloud.
The second and third sentences are more clear, direct, and technically grammatically correct. At any rate, "to be able to" is unidiomatic in that first context and sounds strange because we are not used to hearing it this way. "To have been able" is more idiomatic and makes more sense with things that have already occurred. Or you can change it to "was able to" and modify the tense of "visited".
NOTE: I don't like overthinking grammar syntax. English syntax is very meld-able and driven by the masses; it's very nature is completely subjective. We make the rules. There is not some science that dictates what is right or wrong. There are no scientific tests, empirical data, or hypothesis we can use to determine which are right or wrong in the universe (aside from what the populace at large uses in everyday speech). What you need to do is learn the syntactic structure and rules of your language and then begin to learn how it's actually used in your particular locale. There are many idiosyncrasies in every language of the world and nothing involving linguistics is a law or set in stone.