The only way I think it works is if you put something in between faggot and nigger, like "faggot ass nigger." That sounds okay, almost as good as nigger faggot.
Linguistically, "niggerfaggot" rolls off the tongue more easily.
In "faggotnigger" the "tn" in the middle is hard to pronounce quickly.
It's the subconscious desire to make the language more comfortable for the speaker that makes "niggerfaggot" the version being used.