I think the important thing is to remember WHY Jesus was here in the first place. Not when he was born.
Jesus came to die for our sins. He even said to remember his death, not his birth. Not that his birth wasn't important, but WHY he was here was the most important and what He wanted people to remember the most.
I celebrate my Saviour's birth. As I don't actually know His day of birth, I celebrate it when everyone else does. December 25th.
Some have used it and the fact that pagans celebrated in December as a reason to attack organized religion.
Organized religion DOES have it's weak points, no doubt, as it is run by MAN. That's why i put my faith in God, and not man, as man will let you down. God warns of that very thing in the Bible also. EVERY man is imperfect. Even the ones you THINK are "Holy men".