I voted to close this as something you can easily look up in dictionaries, but since it has received an incorrect response, I had better clear it up.
I went to the store near me.
This is OK grammatically. Note that there is some ambiguity. Literally, it means near your physical body (and there is some doubt as to whether that means where you are now, or where you were at the time you are talking about). Of course, your body doesn't stay in one place, so the shops "near" you will change as you move around.
In everyday speech, "near me" can be shorthand for "near my home" or "near my office", or school, etc. It's ambiguous, but not wrong.
I went to the store nearby me.
This not something you would normally hear. Nearby is not a preposition like "near" that can show the relationship between two nouns. It is an adjective.
You could use:
I went to a store nearby.
The sentence doesn't specify where the store is close to. The reader must infer that from context.
Or you could use:
I went to a nearby store.
... which has the same meaning, but places "nearby" in the conventional adjective position.
I went to the store by me.
This is OK grammatically - "me" is the object, the store is "by" you. If "I went to the store by the river" is syntactically correct, then so is "I went to the store by me".
However, it doesn't sound natural, because there are other set phrases that native speakers would always use instead.
"By myself" means "on my own". So:
I went to the store by myself
... means you had no company. It could reasonably be parsed as "I went to the store next to myself", but the familiarity of the phrase "by myself" trumps that.
All of this is irrelevant to the main question of whether "by" and "near" can be used interchangeably. They cannot, because they have different meanings.
10 Downing Street is near 14 Downing Street. But it is not by 14 Downing Street, because 12 Downing Street is between them.
12 Downing Street is both by and near 10 Downing Street.