When I was in Edinburgh, Scotland, the locals could understand me just fine, but I was flummoxed by their accent, which did not remotely sound like English to me. Necessity forced me to request that the Scottish locals I encountered write out their responses to my questions. Embarrassing! How is this one-way communication misunderstanding possible, given Scotland's proximity to England and Ireland, whose accents are far easier to understand?
I think a full answer to this question would be rather elaborate, taking in history, geography and politics.
I suspect you were talking to working class Scots. Working class accents are often more localised and have greater variation.
In sort, they can understand you but you can't understand them because you are from a more privileged socio-economic group.