I'll answer as honestly as I can. My comments imply no criticism, just what I hear.
There are several aspects to recognising an accent.
- Whether or not you are a native speaker
Clearly you are not a native English speaker. You have an accent that corresponds to no native accents. You are not as fluent as a native speaker and your vowels are accurate but your consonants give you away. For example the 'l' sound which I imagine led someone to wonder about Japanese.
- The accent of your teachers
I would say that your accent sounds European rather than American and that you learned from British English teachers or from teachers who themselves learned from British English teachers. (or possibly from Germanic non-native English speakers rather than from Latin based languages) Your vowels approximate much more to Received Pronunciation as spoken in middle-class England than to anywhere else I can think of.
- Whether the listener is familiar with the accent of your compatriots
I suspect that the variety of guesses arose, not because your accent is neutral (it isn't) but because it is unfamiliar.
Finally you ask whether it is an annoying accent. That's a loaded question - it also relies on personal opinion which is discouraged here. I think that that the most 'annoying' thing about any accent is a lack of precision that prevents one from understanding certain words. The words I found difficult to hear were 'store' and 'amount' in both of those cases the 't' sound was absent or at least suppressed. It took me several attempts to understand 'store' which still sounds like 'sort' to me even though I now know from the context what it must be.
Once again, I hope you perceive no criticism here. When I ask about my Spanish pronunciation for example I like native speakers to answer as accurately as possible.