With your mention of earlier Civs I really ought to play the second and third. Also you hit somewhat on the nail, Civ used to be about whacky alt-history where the Persia reaches the space age while the Vikings are stuck in the industrial era. It wasn't enjoyed like a GSG.
Grand-Strategy Game, like EU4 or CK2. They're much more focused on actual history compared to Civ. But recently even GSGs have had tons of alt-history and whacky stuff. Civ is just an inferior game on many levels, only advantage is that Civ does not take hundreds of hours for a single game.
Whacky alt-history
is enjoying it like a GSG. The general attitude of playing for an interesting, challenging game versus playing to win a victory condition distinguishes the two classes of Civ players.
Civ's virtue for me over PDX game like EU4 or CK2 has been that while you won't get the individual depth of one, it covers all eras within a single campaign and it's a bit more gamey. This excerpt from the Realism Invictus manual describes this much better than I ever could.
There used to be ways where Civ was simpler yet better, but with Civ6 and now 7 it's just inferior. For example alt-history, Civ4 you had more fun options where EU3 was much more linear. Now, after ten years of EU4 DLC, it has a huge number of options, that are absent from Civ games. The only obstacle is apparent complexity (keyword: apparent).
Civ 7 is even a downgrade from 6 in that your options for reaching victories are much more limited. In 6 the ways to reach a culture victory were varied, you had many options like great works, wonders, relics, natural parks, improvements like resorts, etc. In 7 the culture victory might as well be renamed the "Archaeology Victory", since that's the only way you can win it.
Personally I don’t like civ 5. They solved the global happiness issue it had on release with scaling tech cost with more cities. What this meant is that 4/5 cities are optimal in 90% of games. Firstly, this creates a whole “tall vs wide” debate that didn’t really exist before in civ. Secondly it completely undermines one of the core pillars of civ, which is expansion. A core to the games is the idea of covering the map, and your civ should be overcoming others. It’s the first X in 4 X. In many civ 5 games as a result you have whole stretches of the map empty in the 20th century. I get a lot of people really like civ 5. That’s fine but as someone who played since 2, it didn’t fit to me. My favourite in the series is 4.
Tall vs wide is a persistent debate because it's a proxy for the more fundamental conflict of the harcore boardgamers vs the casual roleplay/sandboxers. As not every civilization in history could be classified as "wide" (like Egypt, Venice, Prussia, etc), ideally a game series that's supposed to be historically
authentic to some extent should accomodate non-wide playstyles. It's a matter of flavor as well as just fundametally good game design: there should be variety in gameplay styles and not every civ should play the exact same.
In a series that's about making impactful decisions, tall vs wide is one of the most consequential. Do you play it safe, meaning that you fall off in the lategame, or do you take an early risk and hope to snowball in the future? Tall play doesn't undermine the core pillar of expansion, it's just another expression of it. Internal expansion with specialists, wonders, and great people vs external expansion that's better with production and faith, that sort of thing.
Civ 5 had issues with the balancing between the two styles but that only meant that it should've been refined, instead of thrown out. Civ 6-esque design where improvements to yields are mostly numerical instead of percentage based leads to mindless infinite city settling and "yield spam" type gameplay, which is the opposite of the series's spirit as being "a series of interesting decisions".
Being forced into playing wide means that nine times out of ten of the time the player with the larger numbers will win. That's not a strategy game at all.
I been reading some steam reviews and a common thread I seen is that by separating leaders and civs, the fantasy of roleplaying a civ to ahistorical success has been irrevocably shattered. People care about the Roman Empire, they dont care so much about Caesar or Trajan. Doubly so when you're forced to change civ in the middle of the game and are now suddenly forced to play some civ you may not like or heard of.
This is actually pretty big because while hardcore multiplayer tards may not mind, the casual crowd may just walk off.
The "hardcore NQ multiplayer community" that plays Civ games like they're turn-based RTSs is absolute cancer. I cannot begin to tell you how much I despise influencers like PotatoMcWhiskey, Spiffing Brit, and FilthyRobot who're obsessed with "balance", "exploits", and "builds". Civ 7 reeks of that sort of person's influence and has completely catered to their desires.
I'm sure that having 27,500 possible combinations of civs and leaders is very exciting and allows all sorts of "zomg muh adjacency bonuses", but the much larger casual playerbase that enjoys Civ as a game series that tries to strike a balance between history and gaminess has been completely ignored.
For me and I'm sure many others, variety was satisfied by things like different map types, map options like world age and rainfall, and scenarios, things which are conspicuously absent at release. It really does seem to be that Firaxis has been in a bubble about what they think their playerbase wants and expects. The "this is just a silly boardgame" mindset is visible in all aspects of 7's design.