I vaguely remember a quote floating on the WWW that says
Be careful if everyone else is doing the same.
Tom loves to ride. He has an impeccable passion in riding and motorcycles. However, if he encounters someone or a group who shares the same level of passion as him, he feels inhibited. He reacts as if he isn't as interested for fear if something will go wrong if too many of the likeminded gets together.
How would you describe people like Tom? Is not the same as unconventional, non-conformist just in case you think along that line.