Okay, I think I finally found out the origin of "to the left".
http://history1900s.about.com/od/holocaust/a/auschwitz.htm
Families, who had disembarked together, were quickly and brutally split up as an SS officer, usually a Nazi doctor, ordered each individual into one of two lines. Most women, children, older men, and those that looked unfit or unhealthy were sent to the left; while most young men and others that looked strong enough to do hard labor were sent to the right. Unbeknownst to the people in the two lines, the left line meant immediate death at the gas chambers and the right meant that they would become a prisoner of the camp. (Most of the prisoners would later die from starvation, exposure, forced labor, and/or torture.)
I think we may need to reexamine that "Chris is not worse than Hitler" rule.
Haters To The Left is an established meme, gleaned from the Beyoncé song and some shitbag from 30 Rock. Of course, Chris being Chris made it into a CWCism by constantly using it with no discernible context.
I mean, errr, Hitler confirmed!