Certainly over Advent we should...We are preparing for the Lord to enter into this world, and likewise more spiritually into ourselves. The Lord does not dwell in defiled temples, so we must do all that we can to chastise our weak wills before He comes.
On Christmas day itself, I feel it would be a rejection of the joys that arrive with our new-born Saviour to be penitential. It is, in my opinion (and that of the early Church) a sin to fast on a feast day, as it cuts you off from the universal celebration of the Church and the graces which can be gained from rejoicing in our Lord's presence. I would recommend Christmas eve be the high point of your penance, so that you expiate yourself (should you feel the need), before the coming of appearence of our Lord Jesus Christ.
But on Christmas day itself, it is your Christian duty to feast, be joyous and spread good cheer. This is the day when the boy who would one day bring you salvation entered the world! On this day there can only be rejoicing - so prepare for it, by all means, with penance, but I would advise against a penitential Christmas.