It's funny how even in this line of discussion, both the C16 and C128 usually lie completely forgotten. The C128 was even a dual CPU deal, a souped up HMOS version of the 6510 and a Z80 that could run CP/M. Even had the VDC for 80 col. digital RGB output compatible to CGA monitors(with adapter IIRC, commodore put it's "productivity" monochrome composite signal on there too, which frankly was helpful if you wanted the sharpest possible b&w picture on old display hardware) which you can also interpret perfectly with modern hardware. I have a desktop model of it and it kinda looks like the Amiga 1000. That thing somehow had no market segment. The Amiga that came later on was superior, the C64 had more games, the PCs were better for business. Poor odd machine, even if not inherently bad but I got the impression even the TRS-80 has more retro fans.
Yeah, I've got a 128. It's a good system.
Problem is, for most people, it was just a C64. There was never enough market share to produce more than a tiny handful of software really optimized for it.
Other problem is, like you said, and contributing to the first problem, while it was a good machine, it wasn't revolutionary. It was an upgraded C64, when you get right down to it. Which was fine, but you had the Amiga, the Atari, and DOS machines were starting to get better and cheaper as IBM's stranglehold on the "IBM PC" market crumbled... The market was fracturing too much, and the 128 did nothing to really stand out. It was "fine". It maintained almost perfect compatibility with the existing Commodore ecosystem. But it wasn't a game changer. And that's what Commodore needed.
As far as it's lack of presence in the retro community - not enough of them. They're a lot rarer than a C64, and because of the lack of adoption I already talked about, there's basically absolutely no reason to have one. There's almost literally nothing the 128 can do that the 64 can't, as far as interests actual collectors and retro enthusiasts. Actually there are a handful of things the 128's "near perfect" backwards compatibility fails at, so in some ways it's actually worse, if you want to play one of a dozen or two titles.
So no reason to pay the price. It's a novelty, at best, and if you want to get a "novelty" C64, most people are more interested in the luggable. The two piece version of the 128 is at least interesting in form factor, but they seem to be even rarer than the one piece... IIRC they cost something close to twice as much when they came out, because of the built in floppy drive and more involved construction. And those are the worse machines, compatibility wise - the built in disk drive just doesn't work for certain obscure ways 64 disks were written. Like half of the problems with backwards incompatibility are because of the drive itself.