The Linux Thread - The Autist's OS of Choice

[Overly Serious]

As was mentioned, sync replication is the best solution if you have to be 100% up to date but at a performance hit having to wait for the write to hit the replica. For shit like a shit forum then async replication(log shipping) would get you to the 99.9%. Sure you'll lose any transactions that didn't get in the current log and shipped but that's a fairly minimal loss in most cases if you're not a financial company. But it absolutely has to be a second database/server. It can be a warm standby and importing logs in real time or a cold standby requiring a switch, import all current logs and start up.

DB operations with this forum are going to skew significantly to Reads over Writes. And most writes would likely be appends to some Last Accessed log of some kind to be referenced for showing unread thread status and similar. I would suggest synced slaves would be very desirable because reads can be spread across both master and slaves (or just slaves were this a larger operation) lessening load on master significantly. Both in terms of raw I/O and in terms of table/row locking.

I'd also love to see the partitioning structure and index sizes on these tables. I'm sure I could spend a merry morning nosing through stats and seeing how I could optimise stuff.

 

[Atlas Sneezed]

I tried pitching OpenZFS to Null, but he rejected the idea. Too bad, it would make database corruption recovery a breeze, and with a proper SLOG and some ARC tuning would probably give the DB a big IO boost. Reads come from ARC, writes go to SLOG (backed by optane) and then to the actual drives in a mirror vdev. Backups happen with zfs snapshots rsynced to S3. Done. Mysql shits itself? No worries, zfs promote and you're off. Catastrophic disk failure? rsync the zfs snapshots back from s3 instead of doing costly mysqldump imports.

Even better would be postgres+barman streaming WALs to S3 directly, but IDK if xenforo can do postgres

Also, wtf do new ootb drives that are natively 4k, come set to 512e? It's been, what, a fucking decade since we needed 512 sector drives? I think it's more at this point.

 

[Overly Serious]

Even better would be postgres+barman streaming WALs to S3 directly, but IDK if xenforo can do postgres

A quick search says that Xenforo has no plans to introduce Postgres support. And if it did it would be on a future version that Xenforo is denying Null a licence for. In any case, he's on MySQL now and migration even if it supports it, would be quite the task. And he can't use S3 or any other Amazon service as I highly doubt they'd accept him as a client. Not sure he'd want to anyway - Null likes to roll his own from what I've seen.

I tried pitching OpenZFS to Null, but he rejected the idea. Too bad, it would make database corruption recovery a breeze, and with a proper SLOG and some ARC tuning would probably give the DB a big IO boost. Reads come from ARC, writes go to SLOG (backed by optane) and then to the actual drives in a mirror vdev. Backups happen with zfs snapshots rsynced to S3. Done. Mysql shits itself? No worries, zfs promote and you're off. Catastrophic disk failure? rsync the zfs snapshots back from s3 instead of doing costly mysqldump imports.

Is it confirmed he was re-importing from mysqldump exports? I know he said he was rebuilding tables due to corruption but maybe from binary logs?

Genuine question - does your advice above come from actual practical experience running MySQL (or other DB) on ZFS? Maybe I'm out of date but performance on ZFS used to be significantly slower than recommended file systems by MariaDB such as XFS or ext4.

 

[geckogoy]

Maybe I'm out of date but performance on ZFS used to be significantly slower than recommended file systems by MariaDB such as XFS or ext4.

There isn't a big performance difference. If zfs is using redundancy its slower but you can simply not do that if you so wish and keep the snapshot, mirroring, integrity checking, etc zfs has and afaik just send snapshots to a more redundant/slower pool.

Might work better? IDK

ZFS would for sure eventually facilitate putting all the files back to where they were at the requested snapshot, dunno if that would actually save any time.

 

[Atlas Sneezed]

Genuine question - does your advice above come from actual practical experience running MySQL (or other DB) on ZFS? Maybe I'm out of date but performance on ZFS used to be significantly slower than recommended file systems by MariaDB such as XFS or ext4.

Not MySQL, but Postgres. At company A i ran ZFS on SLES before it was openzfs we had a huge Postgres deployment *and* something called Adobe AEM which implemented something called Oak - a java-specific append-only backingstore using tar files. Both of these were insanely challenging and a disaster recovery procedure would take *days* in the case of psql and *a full fucking week* in the case of AEM.
I initially - on recommendations from Suse - tried btrfs, and it worked somewhat well for postgres, but fell flat on it's fucking face when faced with AEM, because it would only take 2 snapshots for it to spit out "Out of space" which actually meant that it's metadata got so fragmented that it couldn't do anything. That meant a full btrfs defrag routing if you were lucky. To give it it's due AEM was insane in what it did to the filesystem.
So then i first experimented with zfs 0.8, threw everything I could at it and it didn't budge. We went from having WEEKS of recovery time to a mere 6 hours max on a total failure, which we never had since I implemented it. I was touted a company hero, got a big fat raise and they literally kissed my boots to never let me go.
Now I'm at a different place where we're AWS based, and mostly k8s, but there are still places I use openzfs and it's fucking golden. Especially since OpenZFS it's been a breeze. It's sad it'll never go into mainline because ARC could work SO MUCH BETTER if it played nice with the rest of the VFS buffers, but alas in the words of Saint Terry - Linus is a niggerfaggot

A quick search says that Xenforo has no plans to introduce Postgres support

Sucks to suck

And he can't use S3 or any other Amazon service as I highly doubt they'd accept him as a client. Not sure he'd want to anyway - Null likes to roll his own from what I've seen

That was a mental shortcut. S3 = any service that talks S3. You can do Minio which is an S3 server

There isn't a big performance difference. If zfs is using redundancy its slower but you can simply not do that if you so wish and keep the snapshot, mirroring, integrity checking, etc zfs has and afaik just send snapshots to a more redundant/slower pool.

When put against "dumb" FS it will be slower on writes, but it gives you the tools to mask that and even outperform, you just gotta understand your workload and adjust accordingly. Have lots of sync writes? That's gonna suck on a single drive, but slap in a ZIL, now every write is async and batched. Have lots of tiny small file writes? Metadata VDEV and now you're faster than ext4. Need more oomph out of your reads? Mirror+stripe and give ARC lots of memory, now you're playing with power. Can't slap in more RAM but have free PCI slots? Slap an enterprise nvme drive and do L2ARC (but only if you're pressed for it). And this is *on top* of having first class integrity guarantees.

ZFS it still the king of real filesystems, even if it's starting to show it's age a bit, and you can't convince me otherwise

 

[Overly Serious]

@Atlas Sneezed Thanks for the good read. I rarely get down to the file system level (only had to once because of incompetent Sysadmins but to say what they were doing would be so unique in its insanity it would be too much of a P/L).

I shall have to have a proper play around with ZFS sometime. Maybe do some benchmarking, too.

Oh, and when a problem occurs in a set of products where one member of the set is an Adobe product, my starting assumption is that the fault lies with Adobe.

 

[Atlas Sneezed]

rarely get down to the file system level (only had to once because of incompetent Sysadmins but to say what they were doing would be so unique in its insanity it would be too much of a P/L).

I actually like going lower to the kernel level because it's great for job security and prospects, but also it's more interesting then userspace. Anytime something goes wrong I can just jump into what the kernel is getting and immediately pinpoint which developers fault it is. Very useful skill for a DevOps/SRE role

Oh, and when a problem occurs in a set of products where one member of the set is an Adobe product, my starting assumption is that the fault lies with Adobe.

It was originally a Swiss company called Day. Adobe bought them out around version 5.0, which is way after they created Oak. The swiss SBBCFFFFS (their biggest train operator) runs their site on Day/AEM. But your assumption is not wrong.

 

[HootersMcBoobies]

and Percona's MySQL variant with xtradb had a significant performance advantage over stock InnoDB

Percona made a lot of money doing MySQL consulting work too. The last time I was at a shop where we looked at moving to it, the big issue we ran into is Percona didn't support cascadin deletes at the time. (Not sure if they do now). We were using a shitty e-commerce engine called Magento, and it heavily relied on cascading deletes. The Percona recommendation was to create triggers for cascading. Magento was open source, but had no test suite at all in their codebase, so even attempting such a customization would be a nightmare.

Years later I was at a shop where we were using MariaDB because it supported multi-master. I got out of there quick though because they were asking people to constantly work 70~80 hours a week, so I'm not sure how well it actually worked after launch.

 

[AmpleApricots]

I'm glad it was that video, I thought I did it with my 2500 sock puppet accounts

 

[Caesar]

Since I inadvertently started this argument I feel I should weigh in. @davids877 is right, consider where we are for a moment i'm more surprised i didn't get called a faggot for such a question or told to download boku-no-picOS. Anyone that finds themselves in this thread probably has some basic familiarity of linux.

It's been about 4-5 years since i last tried to daily drive a linux system, dunno if manjaro is superior to ubuntu but aside from a few small issues it's going much better than I initially expected.

You should be aware though that Manjaro has borked peoples systems with badly tested updates before, it can break using AUR packages even though it uses them as a default source, and the maintainers still have trouble keeping their websites SSL cert current even with auto tools and them being caught with their pants down multiple times now because of it.

It sounds like you don't wanna move and OK, that's fine, just be prepped at some point to do a reinstall.

 

[Betonhaus]

, and the maintainers still have trouble keeping their websites SSL cert current even with auto tools and them being caught with their pants down multiple times now because of it.

I... How? Even my server proxy does it automatically and I get an email for every domain I stopped using well before the expiry date.

 

[Caesar]

I... How? Even my server proxy does it automatically and I get an email for every domain I stopped using well before the expiry date.

well because they're retards and careless basically and Manjaro only really got popular because they had partnerships with PINE and whoever else for marketing.

At its core though it's just an arch fork so while it works for them, whatever, it'll be fine. I'm just giving them warning.

 

[User names must be unique]

At its core though it's just an arch fork so while it works for them, whatever, it'll be fine. I'm just giving them warning.

I'm all backed up and thinking of looking elsewhere anyway, the wifi driver issue is becoming a ball ache after 6 days of working flawlessly they have decided to stop this morning uninstalling , updating and reinstalling has got them only partly functional and now my VPN is borked.

 

[Caesar]

I'm all backed up and thinking of looking elsewhere anyway, the wifi driver issue is becoming a ball ache after 6 days of working flawlessly they have decided to stop this morning uninstalling , updating and reinstalling has got them only partly functional and now my VPN is borked.

If you want easy arch, endeavourOS is 100% AUR compatible.

 

[prollyanotherlurker]

I mean. These days Arch is easy Arch.

Set up wifi and run archinstall if you can't get a working system from there. You probably won't have much luck using your computer for anything beyond opening a web browser.

 

[User names must be unique]

If you want easy arch, endeavourOS is 100% AUR compatible.

I might give it a shot but i'm thinking a more pedestrian distro is probably going to be more my speed, I don't want to install my own drivers anymore.

 

[Caesar]

I might give it a shot but i'm thinking a more pedestrian distro is probably going to be more my speed, I don't want to install my own drivers anymore.

Then without going too into distro wars I'd try Linux Mint Debian Edition or Pop! _OS

 

[User names must be unique]

Then without going too into distro wars I'd try Linux Mint Debian Edition or Pop! _OS

They're already on my list*. I gave endeavour a shot but after 5 minutes of failing to install Brave I gave up, if i'm too retarded to perform step number 1 of a new OS install it's clearly not for me. Now onto Fedora

*

shout out ventoy btw, best bootable usb thingy I ever did use.

 

[x3firehexx]

They're already on my list*. I gave endeavour a shot but after 5 minutes of failing to install Brave I gave up, if i'm too retarded to perform step number 1 of a new OS install it's clearly not for me. Now onto Fedora

Naw, you are making a mistake, you should go install LFS, much simpler, and you'll get a customized, minimal, de-bloated install w/o anyone's bullshit on it.

Just make sure to do sudo rm -rf /* or sudo rm -rf / --no-preserve-root right after you are finished to remove the french language pack, you have to do this first, then you can move onto the other unnessecary language packs. Y'know, will debloat by a good amount.

 

[Caesar]

They're already on my list*. I gave endeavour a shot but after 5 minutes of failing to install Brave I gave up, if i'm too retarded to perform step number 1 of a new OS install it's clearly not for me. Now onto Fedora

*View attachment 6711487

shout out ventoy btw, best bootable usb thingy I ever did use.

Were you trying to install using the GUI software center? You probably didn't enable the AUR if so. I don't think arch is a good fit for you anyway tbh but if you go back, that's probably why.

Another option is ChimeraOS which is the steam deck OS for generic devices, which is also arch based, as you said you like to game but it is an immutable file system (in simple terms of what that means for you, pacman won't work, you'll have to use flatpaks). That may or may not work for your use-cases. It is designed for gaming, though.