Does it mean before or after? Or does it mean both before and after? For example, do not drink or eat within an hour of these pills.
Within is a single word. In the sense of your example, it means something like during, or before the end of:
See definition on Merriam-Webster.
'Within' means inside some sort of time frame... so this sentence literally means that you must wait until an hour after eating before taking the pills, and must not eat until an hour after taking the pills.
In my experience, this is very usual advice - I don't think it was intended this way; it could mean you need to wait for an hour after eating before taking the pills, or it could mean that you have to wait for an hour after taking the pill to eat.
I would interpret it as the latter, but it isn't entirely clear.