You are 100% right, and if you take a look at the history of drag shows you will see that.
In the 19ths century minstrels shows were the shit. Actors first only mocked men, but then they decided to expand their marker and have caricatures of black women as well. Below is a white man in drag and blackface.

It was really well received, but the civil war ended, freed blacks were moving into cities, and it's a stupid idea to offended a potential costumer. So vaudeville shows, comedic acts with lots of variety that took inspiration from minstrel shows, started popping up and brought female impersonators.

This is Julian Eltinge, arguably the most successful female impersonator. Being a female impersonator was a straight white man's job. They had two personalities, one one and out out of drag. Homosexuality was looked down upon at this time and as time went on female impersonators became associated with gay men and sex work, so it fell out of fashion.
There were male impersonators of course, but they were not as popular and did not mock men as hard as women were mocked if I recall correctly. Drag in it current form is directly connected to minstrel shows and it arguably is closer to it than female impersonators because of how many African American Vernacular English mannerisms they adopt. One look at the
phrases used on RPDR will show that. They want to be a sassy black women so bad and it grinds my gears because this isn't how black women act.
This information isnt explicitly stated on the internet but it's there: wikipedia mentions it. Almost all LGBT website that talks about drags history either fails to mention female impersonators or pretends that female impersonators didn't come from the minstrel's potrayal of black women.