Cyberpunk 2077 Grieving Thread

[Jimmy Durante's Ballsack]

Hopefully the aesthetic of the game's design is good. Because in eight months Cyberpunk is going to look like trash if they went for photo realism.

Like others have said though, it's probably a lot to do with audio. A lot of good 70GB will have done.

 

[ZMOT]

Nah, compression should be a net gain for consoles, in terms of on-disk footprint, loading times and memory use. See above for an explanation. Modern gaming hardware has more than enough horsepower to decompress game assets on the fly without any noticeable impact on performance. They often have hardware that can handle a lot of that work at no performance cost at all (there's audio/video decoders on every GPU today, for example, that have their own dedicated space on the die -- using them doesn't involve the rest of the GPU so that work can be offloaded for free onto an otherwise idle component).

cpu cycles and ram aren't infinite tho, especially on consoles when you try to make a 2020 game run on "modern" 2013 hardware. there's a reason mass effect 3 had no holster animation. and since consoles are the target platform almost every time, it's gonna drag everything else down with it to the lowest common denominator due to contractually obligated parity clauses (just as people predicted back then).
could you optimize every version of every platform to the point you get almost similar results? sure, but who's gonna pay for that - especially when consoles sell on the whole "buy new shit to have it run better" idea. can't remember when a console allowed me to turn shit down to 720p to get stable 30+ fps to squeeze a few more years out of the hardware for example.

 

[moocow]

cpu cycles and ram aren't infinite tho, especially on consoles when you try to make a 2020 game run on "modern" 2013 hardware.

Compressed assets use less RAM, not more, until they're unpacked (and then the compressed data is discarded anyway since you no longer need it). And if you need unpacked assets, you're using that memory anyway whether compression was involved on disk or not. If memory is really tight, you can get clever and start discarding compressed blocks as they're decompressed so you never end up consuming more than the uncompressed data size, plus maybe a few kilobytes for the last compressed block. But that's something you'd only really need to worry about on, say, PS2-level hardware where you've only got a few megabytes of RAM. Once the OG Xbox arrived with 64MB of system memory and the PS3 arrived with 256MB of system memory, this stopped being a concern.

And I cannot emphasize enough how little CPU time is required to decompress data compressed with most common algorithms. Obviously lzma and lzma2 are slow going in and out because they focus on high compression ratios above all else, but others (like lz4) are optimized for high-speed decompression (see this benchmark where lz4 spent 0.4 CPU seconds to decompress a 445MB file on a 2.67 GHz i5; the CPUs in the current gen and next gen consoles are faster than this).

It is literally faster to stream compressed data from disk and decompress assets into memory on the fly than it is to just pull in uncompressed data. Keep in mind disk I/O isn't "free" either. It takes CPU cycles too, and in fact I/O is a blocking operation (so the CPU spends some time in "iowait" state waiting for I/O to finish -- that's time it can't use for anything else). Given the same amount of wall clock time, you'll end up with more of your data in memory if it starts out on disk in compressed form than if it's uncompressed.

In the above benchmark, the uncompressed file size was 445MB and it compressed down to 159MB, or 35.6% of the original file size (lz4_hc improves compression ratios even further and still decompresses at the same speed). Loading that compressed data from disk takes just 1/3rd of the I/O activity as loading it in its uncompressed form. lz4 decompression is so fast (described as "multiple GB/s per core (~1 Byte/cycle)" by its developers) that it can decompress data faster than it can come in from the disk, even if it's an SSD. I'd be willing to wager this remains true even with the PS5's "super-mega-hyper-ultra-awesome" SSD architecture. It doesn't matter how fast disk I/O is; the less of it you have to do, the better.

Think of it this way: say you need 445MB of data loaded from disk into memory for whatever purpose. You can either wait for 159MB of data to load from disk and give up one core for less than half a second to decompress it (including I/O overhead), or you can wait three times as long for 445MB of data to load from disk and still give up one core for at least 2/5ths of a second as it waits for 3 times as much blocking I/O to finish. With decompression this fast, it will take less wall clock and CPU time overall to read and decompress 159MB of data than it will to just read 445MB. The CPU is faster than the disk and storage controller by a long shot, no matter how fast they are.

The cool thing is this applies to practically every platform -- game consoles and PCs alike. lz4 is open source, patent free and very portable (it's easy to build and use in tons of languages on lots of operating systems and architectures; there's even a decoder written in assembly for the Z80 CPU used in Texas Instruments' old TI-81 calculators). You don't have to do anything special to have it available on every platform your game is targeting, and it's thread-safe, meaning you can perform multiple decompression operations simultaneously in different threads without locks or mutexes or semaphores or other pain-inducing multithreading trickery.

Using it is a no-brainer and honestly doesn't take much dev time to do it. Yes, the dev time cost is not zero, but the end result is a game with a significantly smaller footprint. There's no real reason not to do this except for simply not giving a shit (or, for the more cynical people out there, wanting to deliberately make the footprint bigger to "squeeze" out competing games on limited storage).

 

[Ginger Piglet]

Compressed assets use less RAM, not more, until they're unpacked (and then the compressed data is discarded anyway since you no longer need it). And if you need unpacked assets, you're using that memory anyway whether compression was involved on disk or not. If memory is really tight, you can get clever and start discarding compressed blocks as they're decompressed so you never end up consuming more than the uncompressed data size, plus maybe a few kilobytes for the last compressed block. But that's something you'd only really need to worry about on, say, PS2-level hardware where you've only got a few megabytes of RAM. Once the OG Xbox arrived with 64MB of system memory and the PS3 arrived with 256MB of system memory, this stopped being a concern.

And I cannot emphasize enough how little CPU time is required to decompress data compressed with most common algorithms. Obviously lzma and lzma2 are slow going in and out because they focus on high compression ratios above all else, but others (like lz4) are optimized for high-speed decompression (see this benchmark where lz4 spent 0.4 CPU seconds to decompress a 445MB file on a 2.67 GHz i5; the CPUs in the current gen and next gen consoles are faster than this).

It is literally faster to stream compressed data from disk and decompress assets into memory on the fly than it is to just pull in uncompressed data. Keep in mind disk I/O isn't "free" either. It takes CPU cycles too, and in fact I/O is a blocking operation (so the CPU spends some time in "iowait" state waiting for I/O to finish -- that's time it can't use for anything else). Given the same amount of wall clock time, you'll end up with more of your data in memory if it starts out on disk in compressed form than if it's uncompressed.

In the above benchmark, the uncompressed file size was 445MB and it compressed down to 159MB, or 35.6% of the original file size (lz4_hc improves compression ratios even further and still decompresses at the same speed). Loading that compressed data from disk takes just 1/3rd of the I/O activity as loading it in its uncompressed form. lz4 decompression is so fast (described as "multiple GB/s per core (~1 Byte/cycle)" by its developers) that it can decompress data faster than it can come in from the disk, even if it's an SSD. I'd be willing to wager this remains true even with the PS5's "super-mega-hyper-ultra-awesome" SSD architecture. It doesn't matter how fast disk I/O is; the less of it you have to do, the better.

Think of it this way: say you need 445MB of data loaded from disk into memory for whatever purpose. You can either wait for 159MB of data to load from disk and give up one core for less than half a second to decompress it (including I/O overhead), or you can wait three times as long for 445MB of data to load from disk and still give up one core for at least 2/5ths of a second as it waits for 3 times as much blocking I/O to finish. With decompression this fast, it will take less wall clock and CPU time overall to read and decompress 159MB of data than it will to just read 445MB. The CPU is faster than the disk and storage controller by a long shot, no matter how fast they are.

The cool thing is this applies to practically every platform -- game consoles and PCs alike. lz4 is open source, patent free and very portable (it's easy to build and use in tons of languages on lots of operating systems and architectures; there's even a decoder written in assembly for the Z80 CPU used in Texas Instruments' old TI-81 calculators). You don't have to do anything special to have it available on every platform your game is targeting, and it's thread-safe, meaning you can perform multiple decompression operations simultaneously in different threads without locks or mutexes or semaphores or other pain-inducing multithreading trickery.

Using it is a no-brainer and honestly doesn't take much dev time to do it. Yes, the dev time cost is not zero, but the end result is a game with a significantly smaller footprint. There's no real reason not to do this except for simply not giving a shit (or, for the more cynical people out there, wanting to deliberately make the footprint bigger to "squeeze" out competing games on limited storage).

Uncompressed audio is in particular unforgivable, frankly. FLAC can reduce to 50% with no loss of fidelity. Ogg Vorbis can hit 36% with minimal fidelity loss. Both are open source. And let's be frank. In a game you aren't going to be "REALLY LISTENING" to the music the way that fucking audiophiles are with their turntables and multi-wired hi-fi setups that cost more than a small car. So you can probably drop the bitrate to CD quality and reserve the FLAC quality files for the soundtrack download package / deluxe edition soundtrack CD.

(Also, I remember seeing in one of the earlier trailers someone buying a Samurai album in-game on vinyl. I am glad to know that we still have vinyl snobs in the future. I suppose they can cope by saying that because an analog format can't be DRM'd effectively they're sticking it to the man, as the man presses discs on an Arasaka manufactured vinyl press.)

 

[ZMOT]

Using it is a no-brainer and honestly doesn't take much dev time to do it. Yes, the dev time cost is not zero, but the end result is a game with a significantly smaller footprint. There's no real reason not to do this except for simply not giving a shit (or, for the more cynical people out there, wanting to deliberately make the footprint bigger to "squeeze" out competing games on limited storage).

but it adds up, that's my point. you are never performing just one operation in a benchmark environment. disk access alone will choke most games (even on pc), and sony has always been cheap/lazy in that regard (never read any xbox comparisons so no idea how much an SSD would improve the old xbones). and with less ram you gonna have to swap more, which means more I/O and cycles reloading assets etc.
I also don't think devs a necessary lazy, just that the amount of assets has exploded, bigger size, higher amount, all adds up (unless you're respawn and distribute the audio files compressed and unpack them during installation...)

 

[Jimmy Durante's Ballsack]

This technical discussion is far more interesting than anything related to Cyberpunk 2077 since the first teaser trailer came out.

 

[Smaug's Smokey Hole]

Think of it this way: say you need 445MB of data loaded from disk into memory for whatever purpose. You can either wait for 159MB of data to load from disk and give up one core for less than half a second to decompress it (including I/O overhead), or you can wait three times as long for 445MB of data to load from disk and still give up one core for at least 2/5ths of a second as it waits for 3 times as much blocking I/O to finish. With decompression this fast, it will take less wall clock and CPU time overall to read and decompress 159MB of data than it will to just read 445MB. The CPU is faster than the disk and storage controller by a long shot, no matter how fast they are.

Change the scenario to games and the 445MB might be desirable, because the partial transfer can be used right away without waiting for the full file/block to load and decompress(remember early UE3 games?). Optimizing and laying out data like this have been a thing for a long time and it leads to a lot of asset redundancy to avoid thrashing.

Compressed textures are doing it the right way being block based, but they are also lossy compression and that won't go over so well with other assets. Well, pre-fab LOD models for characters and objects are lossy but they mean even more repetition with every model and texture existing at different levels of complexity. It trades space and RAM for performance, which is pretty typical for game engines.

It is absolutely textures that is ballooning things up in size, at 32BPP a 4096x4096 texture is 64MB if uncompressed, it will be smaller after compression but a normal map would sit at 48MB and it's something that maybe shouldn't go through lossy compression because it's used for calculations. Using a non-lossy method the normal map might compress great or not well at all.

I'm willing to tolerate even the worst looking games so long as the gameplay is fun (Shores Of Hazeron that's a visual nightmare)

Jeez, you weren't kidding.

 

[Jimmy Durante's Ballsack]

Jeez, you weren't kidding.
View attachment 1749138

Different opinions and all that but that screenshot is gorgeous.

 

[Disc]

Jeez, you weren't kidding.
View attachment 1749138

I didn't know they released a halo game for DOS.

 

[Marissa Moira]

I bought a new computer with 64 gigs of Ram, a 2060 RTX, and an AMD Ryzen 9 3900x just to watch this shitshow destroy any and all performance benchmarks with bloated code and needless post production effects.

 

[moocow]

but it adds up, that's my point. you are never performing just one operation in a benchmark environment. disk access alone will choke most games (even on pc), and sony has always been cheap/lazy in that regard (never read any xbox comparisons so no idea how much an SSD would improve the old xbones). and with less ram you gonna have to swap more, which means more I/O and cycles reloading assets etc.

It doesn't really "add up" the way you think it does. The CPU is always faster than the disk. It doesn't "fall behind" decompressing data coming from the disk -- it will always decompress data faster than the disk can deliver it (especially, as you point out, on consoles with garbage storage performance) and the work will never peg even a single core to 100% utilization. Disk access does indeed choke most games, which is why using on-disk compression is a win: you can load more data with less disk activity.

You will always have more CPU cycles than IOPS, even mid-game when you're trying to stream data from disk in the background without the player noticing. Reloading assets is also faster since you need less I/O to do it. And as I described in a recent post, in-place decompression doesn't have to use much memory at all; with block-based compression algorithms like lz4, you may need no more RAM than the final decompressed data will occupy if you do it right.

ETA:

Change the scenario to games and the 445MB might be desirable, because the partial transfer can be used right away without waiting for the full file/block to load and decompress(remember early UE3 games?). Optimizing and laying out data like this have been a thing for a long time and it leads to a lot of asset redundancy to avoid thrashing.

lz4 is block-based as well, and is suitable for real-time random-access because its compressed format is predictable. It's used for SquashFS on Linux for read-only filesystems meant for very sluggish flash storage like SD and MicroSD and USB 1.x and 2.x thumb drives, and it's got a long track record of strong performance even for random access. I've even seen it used on installation CDs and DVDs where they'll put just enough stuff on the ISO9660 track to make the disc bootable and then cram the rest into a single Squashfs file. It still does pretty well. For the longest time, you could fit the Debian "netinst" ISO onto those business card sized CDs thanks to Squashfs. Those were nifty.

Compression is obviously not a panacea and there are situations where it won't yield beneficial results, but really it should used on any game data that can be successfully reduced in size by a lossless algorithm. I'd also argue that textures should be candidates for lossy compression, especially once they hit 4k or 8k resolution where the loss should be imperceptible. Normal maps can be (and often are) compressed with BC5; this is natively supported by all modern GPUs. It reduces their size by 50% or more (75% if you're willing to give up some precision).

I bought a new computer with 64 gigs of Ram, a 2060 RTX, and an AMD Ryzen 9 3900x just to watch this shitshow destroy any and all performance benchmarks with bloated code and needless post production effects.

God damn, that sounds like a beast. How much did that set you back?

 

[ZMOT]

It doesn't really "add up" the way you think it does. The CPU is always faster than the disk. It doesn't "fall behind" decompressing data coming from the disk -- it will always decompress data faster than the disk can deliver it (especially, as you point out, on consoles with garbage storage performance) and the work will never peg even a single core to 100% utilization. Disk access does indeed choke most games, which is why using on-disk compression is a win: you can load more data with less disk activity.

You will always have more CPU cycles than IOPS, even mid-game when you're trying to stream data from disk in the background without the player noticing. Reloading assets is also faster since you need less I/O to do it. And as I described in a recent post, in-place decompression doesn't have to use much memory at all; with block-based compression algorithms like lz4, you may need no more RAM than the final decompressed data will occupy if you do it right.

but all that happens on top everything else, most open world games are constantly streaming assets while the game is running - there isn't a loading screen where everything sits neatly in ram once that's done. a game already choking will choke even more.

 

[Jimmy Durante's Ballsack]

On the other hand, I have no problem with loading screens or zones. Metroid Prime was fine and dandy and STALKER was a blast.

 

[moocow]

but all that happens on top everything else, most open world games are constantly streaming assets while the game is running - there isn't a loading screen where everything sits neatly in ram once that's done. a game already choking will choke even more.

It shouldn't choke more though. It should choke less. There's less I/O going on when the data is compressed on disk because you don't need to read as much of it to get the final data you want in memory.

lz4 can achieve 5 gigabytes per second on a single thread when decompressing on a late model 2.7GHz i5. The current consoles have faster CPUs than that. At that level of performance, it's like a sneeze in a hurricane on a modern CPU. I can't emphasize enough how little impact it has on performance even on a CPU-bound workload. Even if you are literally pegging all cores to 100% (no game has done this on a modern multi-core console, to my knowledge) you can still decompress at a rate of 1 gigabyte per second even if your decompression thread only gets 0.2 CPU seconds every wall clock second. That still handily beats everything besides a RAM disk.

 

[EyelessMC]

It shouldn't choke more though. It should choke less. There's less I/O going on when [...] lz4 can achieve 5 gigabytes per second on a single thread when decompressing on a late model 2.7GHz i5 [...] on a CPU-bound workload. Even if you are literally pegging all cores to 100% [...] besides a RAM disk.

You really make good points but the most important aspect is thermals and GPU threading. If you don't have at least a quad core supraliminal GPU then your fan noise will easily be above 68dB at any given read of instanced core threading even with a Ryzen 4800ti, and that's with or without a solid SSD that can page down the gigabytes of second-stalled scoring.

As for thermals it's obvious that you can't just strap-on a bottom-down liquid cooling system because the CPU won't be compatible with the internal charging of your motherboard--which, of course, if we're talking about being able to compress and deread data then you need something more than some NVIDIA GTX990. Which is why I always suggest a water-based cooling system adjudicated by an acrylic 9 paste so you can get the performance of an Iz8 from a decompressed 2230 and really make her sing.

 

[murdered meat bag]

I searched around and unlike Bethesda games which also crash like a motherfucker there do not appear to be comprehensive modder made stability/bugfix patches to Gothic games so they are essentially unplayable at least for me.

did you install to your C drive?

would crash at intro movie until i moved it c program files.

 

[Judge Dredd]

I still maintain that developers just aren't using all the techniques at their disposal to reduce game footprints and optimize their games. Maybe it's because they just don't have enough time during development to integrate them, or maybe it's because they've gotten complacent since bandwidth is cheap and so is storage (at least on PCs). It's astonishing though that they're not taking the time to reduce game footprints on the consoles, though.

Just imagine owning a PS5 or Xbox-whatever and only being able to play 6-7 games on demand before running out of space, and having to delete a game to make room for another 100GB+ game that could take hours to download depending on your broadband link. If you've got shitty internet, it could take days to install a game or download its patches at these sizes. That's just god damn awful.

I understand, I'm just hesitant to call developers lazy.

People who have never made a game often underestimate the amount of work that goes into them. It's similar to discussions I've had with Nintendo and Roller Coaster Tycoon fanboys that claim every game should be written in assembly. Yes, the frame rate is better, but it's much more difficult to make and takes longer to make. On the other end of the spectrum, you can download Unity or Unreal and watch those "make a FPS in 1 hour" type tutorials on YouTube, buy some assets, and slap a game together over a long weekend, but that game is going to be bad.


Same goes for compression. I agree that games should return to the minimum install or full install option from back in the day, and I agree that parity clauses are bullshit, and I agree that Ultra settings could be a separate download. Personally, I think console makers should demand that a game must be fully playable from the disc, or put harsh limits on download sizes. But ultimately, they have to make a choice. Inconvenience the player base at release, or piss off the influencers who spent £3000 on the latest 3 letter graphics card. I might not like their choice, but I understand it.

 

[Smaug's Smokey Hole]

It shouldn't choke more though. It should choke less. There's less I/O going on when the data is compressed on disk because you don't need to read as much of it to get the final data you want in memory.

Doesn't matter if it is on a disk or disc, seek times kill performance, it's better to replicate data and put it sequentially along with the area/section that needs to be read. I know what you mean and texture compression is used, textures/bitmaps are still huge though. Not making game audio regional is a bummer, scalping/pirating Halo 2 was really fun because the french version leaked first. Sacrebleu!

 

[jje100010001]

Current concern- it sort of feels like the city is quite underpopulated in many of the videos. This is one of the things that killed Watchdogs 2's setting for me, which was that the city felt quite lifeless despite the detail.

I know CDPR said it was all tied to the time of day/peds 'disappear' when you get into a car, but I can't but think that it's an excuse for a downgrade.

Saw a thread full of webms showing off a stellar drive-by shooting segment (guy gets shot in the head at a phenomenally fast snail's pace, still kicking) and the amazing attention to detail in a POV cutscene (driver manages to turn the car without moving the wheel). PS5 apparently.
Then someone posted this:
View attachment 1748553
I really hope this is legit. Can anyone confirm?

Honestly sounds like cope- why don't they show the city at any other time of day? Or any of the busier places?


At the same time, driving still feels like it lacks momentum and weight for some reason- not sure why, but maybe it's a camera/fov thing?

Also the soundtrack previews seem decent enough, but I think that there could have been some opportunity for a Fifth Element-type cultural influences, considering the diversity in Night City- at the moment it does seem Blade Runner-esque electronic/ambient-heavy. I wonder what the radio will be like?

 

[Cantercoin]

To divert from the technical spergery at the moment, if you preorder Cyberpunk 2077 on steam you get an exclusive short story with some avatars posters and a wallpaper