Technically, one could use I slept at 11, but it is not the most natural or common expression available. One time I might use it would be when reporting past events or conditions, and then I'd put the times first:
At 6, I awoke.
At noon I played football.
At 5 I partied.
At 11 I slept.
But note at 11 I slept does not mean I went to sleep at 11. It means at 11 I was asleep. This is the past simple equivalent to the past progressive At 11 I was sleeping (or I was sleeping at 11) which is more common to say, because to sleep is an action that occurs over time. We don't normally use it in contemporary English to mean, for instance, I started to sleep.
In everyday (American) English, I wouldn't use I slept at 11 but one of the following alternatives, depending on what I wanted to express:
I went to sleep at 11
I fell asleep at 11
I was asleep at 11
I was sleeping at 11
I used to go to sleep at 11
Etc.