Ditching religion dumps the guilt and fear, freeing your mind to deal with reality instead of myths.
Humans are wired for guilt and fear, so ditching religion just means we project those onto politics or culture instead. You're not freeing your mind, just changing the myths.