If we agree Battlefield 2024 looks no better than its predecessors from a decade ago (for whatever reason), and we installed both games today, ran them at the same presets and resolution (to make it as fair as possible), which game do you think will have the far better performance?
They'd run exactly the same, because if that were true (and it's not), it would mean that they just used the same exact code from a 2015 and used different assets of the exact same quality. If you modded BF2042 to use BF4 assets, it would probably run faster, since they've likely made a few improvements to the rendering pipeline in 10 years, similar to how MW2019 has a much better optimized graphics engine than MW3.
If you took screenshots of the big hitters of the 90's and 00's and lined them up one after another you would see the blatant advances that justified lackluster performance. The same can't be said of the 2020's where every game looks the same and somehow performs worse.
This happens to every technology. Early improvements are huge and cheap. Late improvements are expensive and marginal. We could be talking about airplanes, steam engines, televisions, cell phones, or video games. Look at the innovations in the first 30 years of the auto industry - windshields, automatic wipers, rearview mirrors electric headlamps, heaters, etc - nothing in the last 75 years changes the driving experience as dramatically or cheaply as being able to drive in the dark without rain hitting your face.
Those early innovations in the first 10 years of 3D gaming (about 1993 through 2003), like colored lighting, gloss maps, Gouraud shading, etc are computationally cheap, conceptually simple, and dramatic in effect. Innovations in the last 10 years are less dramatic and more expensive for the simple reason that all the low-hanging fruit has been picked.
The mistake you are making is thinking that if an improvement feels minor, it can be made computationally small. A simple example is color depth. I first started gaming when PC games were EGA,4 bits per pixel:
Then came the jump to 256 color VGA, 8bpp, 2x more expensive than EGA. The colors were stunning:
Or here's what Doom would have looked like in 4-bit EGA vs 8-bit VGA:
In 30 years, despite going from 8bpp to 16bpp to 32bpp, nothing's ever matched that jump from 4-bit to 8-bit in terms of how dramatic and stunning it was (actually, 2-bit CGA to 4-bit EGA was pretty impressive as well). Going from 16bpp to 32bpp was truly marginal. Color banding a was bit better, and there was less dithering. "Lazy programmers refusing to optimize" was not the issue. A hard-working programmer optimizing the shit out of a 16-bit rasterizer would still not have been able to eliminate color banding and dithering. You needed another doubling of color depth. But it was a doubling in cost with marignal results.
The reason SSAO, raytracing, 4K, and other technologies give modest results at massive expense is that the low-hanging fruit is gone, it's the path of all technologies. There is no way to "optimize" raytracing to make it 10x faster or 10x more visually stunning. Again, the mistake you are making is thinking that a graphical effect that gives marginal improvements in the look & feel of a game should be computationally cheap, and that just isn't true.
That doesn't fix the worse A.I., physics, destruction, netcode, and every other aspect that goes into these games.
DICE not making small objects moveable or concrete barriers destructible in BF 2042 that were dynamic in BF3 has nothing to do with optimizing code.
