Can anyone send me the moderator a mail withough joining the group?

i am the moderator. can anyone who has my group email adress send me an email wihtout joining the group first? It seems no or my settings are wrong. so far if someone sends an email to my group email it comes back as undeliverable and asking to join with a yahoo account first.