Well, it's not incorrect, and it's perfectly comprehensible, but to my ear it's not the most idiomatic way to ask the question. As a native speaker, I'd phrase it as "When do you go to sleep?" or simply "What time do you go to sleep?" (the preposition makes the sentence clunkier, and you don't need it).
Based on the reactions of the people you were speaking to, I'm guessing they either misheard you (perhaps due to your accent?) or were taken aback because the question felt intrusive and/or like a non-sequitur.
It depends on what you were talking about before you asked it, and on what your relationship with these people was. If the context didn't involve talking about sleeping, then I second the suggestion to add "at night" to the end of the sentence to clear that up. Or, if you didn't know them well, it might simply have been that they didn't feel comfortable sharing that kind of information.