Might be a bit revisionist of me but I'll throw my two cents in. I think it might have crippled Hololive in the beginning if they did that, they were just hitting their stride at this point around Sept 2019 and Fubuki was trending upwards (215k or so according to holotracker) when the stars first debuted. Hololive was marketed as idols either to differ from the competition or it was just A-chan's idea on a whim.
The first collab between Hololive and the stars if I recall correctly was a Dead by Daylight with Fubuki, Choco-sensei, and Masturi in Feb 2020. This didn't tank their numbers or anything, but it is worth pointing out that collabs are far enough part that any of those specials fan (they are vocal but tiny) can ignore it as well as the cohesion built so far in Hololive. Fans might not mind but there are some in Hololive that rarely collab or talk to the other genders leading to division in Hololive and ruining that fuzzy feeling people get seeing a small group interact and do well with each other, for the most part.
For example, if there was a big event, there might be awkward silence because some of the Hololive members don't know how to interact with them or even worse, they don't want to show up. Hololive was trending upwards, do you risk that success with a smaller, niche market? Another risk in combining branches is you risk losing that something that separates you from other vtuber companies. Just something to think about.
Edit - wording