Total data loss after botched GitOps and failed backups for firefish.lgbt, musician.social, and outdoors.lgbt - Yet another troon L

Ahriman

Ahriman

Vivere Militare Est.
kiwifarms.net
Joined
Jan 22, 2018
Bad news everyone. It is with immense regret that I write to inform you we have suffered a total loss of data for firefish.lgbt, musician.social, and outdoors.lgbt.

How did we get here?

During a routine #GitOps repository cleanup a subdirectory containing yaml manifests that create our namespaces was moved to directory not visible by ArgoCD. From Argo’s perspective, the directory and yaml manifests no longer existed so it went to do its job and clean things up. Had the directory just contained the manifests for the Helm deployments, this would have been okay as the Persistent Volume Claims would have persisted, but deleting a namespace deletes everything it contains.

Didn’t you have backups?

Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data. While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data.

What about snapshots?

In the past I had to move away from the big 3 cloud providers in order to make hosting financially feasible. Vultr was chosen since it offers a fully managed #Kubernetes while still being affordable. The downside to that was not having PVC snapshots support built into Velero, which I took as an acceptable risk since we had block level backup support with Rustic and our tests had shown it to function well.

What about contacting the cloud provider?



Well… I did. They don’t keep backups of customer’s PVCs and suggest using something Restic to do so.

What does this mean?



To put it bluntly: everything is gone and there is no recourse to get it back. I fucked up. I do not take it at all lightly that the Fediverse has been a home and safe place for many individuals, including myself, and the feeling of loss and regret here is, to be honest, crippling. The fact that so many people will be affected by this is not lost on me. I am so so so incredibly sorry to those who have placed your trust in me only to have that trust be betrayed. I can’t apologize enough. 


What happens going forward?

I won’t personally be bringing back outdoors.lgbt or firefish.lgbt. Being an admin has been one of the most fulfilling things I have done in a long time and you all have made it such an amazing experience., however, I need to take a step back.

I would love to hand the domains over to someone with a similar passion for creating a safe and welcoming community.

I am so sorry for letting you all down, and I wish you the absolute best.
Click to expand...

It's over :story:
 
  • Winner
  • Like
  • Informative
Reactions: byuuWasTaken, Markass the Worst, HahaYes and 26 others
#GitOps repository cleanup a subdirectory containing yaml manifests that create our namespaces
Click to expand...
volume backups with Restic which snapshots PVC block volume data
Click to expand...
Vultr was chosen since it offers a fully managed #Kubernetes
Click to expand...
What does this mean?
Click to expand...
I can't wait for the entire Internet to just fucking implode due to everything on it turning into these SRS-esque webshite Rube Goldberg machines.
 
  • Like
  • Agree
  • Feels
Reactions: UERISIMILITUDO, byuuWasTaken, Vecr and 14 others
Francesco DeIIamorte said:
I can't wait for the entire Internet to just fucking implode due to everything on it turning into these SRS-esque webshite Rube Goldberg machines.
Click to expand...
I swear nobody actually knows how any of this shit works or why.

It’s a fucking website. Install a fucking webserver. You don’t need helm to run kubernetes to install fuck all dot footgun.

Amateurs.
 
  • Agree
Reactions: byuuWasTaken, Jarry Seinfelt, Pee Cola and 5 others
1693418038105.png


1693419106140.png


Big yikes!
 
  • Islamic Content
  • Horrifying
  • Informative
Reactions: osooolemiiio, Nitro!, byuuWasTaken and 12 others
I (Don't) Have A Gun said:
I suspect their hiring practices prioritised diversity over competency, because god dammit how don't you back your shit up!?
Click to expand...
Always baffles me how you can fuck up this task that can be done using tools that have been in UNIX-like operating systems since 1975. Like fuck me, just cron job a tar/gz file then rsync it to a different server. Do it once, make sure it works, then make another script on the server you sent the data to to check the backups are working (extract files, check for presence of certain filenames, check file size). Like it would take half an hour at most including testing to make this bulletproof forever.
 
  • Like
  • Agree
Reactions: King Vidiot, byuuWasTaken, Markass the Worst and 6 others
RACISM said:
Always baffles me how you can fuck up this task that can be done using tools that have been in UNIX-like operating systems since 1975. Like fuck me, just cron job a tar/gz file then rsync it to a different server. Do it once, make sure it works, then make another script on the server you sent the data to to check the backups are working (extract files, check for presence of certain filenames, check file size). Like it would take half an hour at most including testing to make this bulletproof forever.
Click to expand...
Sure, but those tools were all written by white cis men. Do those projects have codes of conduct? I bet they still use "master" as their git branch names. How do I know that the maintainers of those tools support diversity and stand with Ukraine?
 
  • Winner
  • Feels
  • Agree
Reactions: osooolemiiio, byuuWasTaken, Markass the Worst and 5 others
Spasticus Autisticus said:
Sure, but those tools were all written by white cis men. Do those projects have codes of conduct? I bet they still use "master" as their git branch names. How do I know that the maintainers of those tools support diversity and stand with Ukraine?
Click to expand...
This is why you should only use new software that has the plusnigger license.

+NIGGER License​


About​


The +NIGGER License is a license modifier that requires the inclusion of the word "NIGGER" in the LICENSE file.

Why?​


By including the word "NIGGER" in a LICENSE file that must be distributed with the software you will ensure:

  • The software will not be used or hosted by western corporations that promote censorship
  • The software will not be used or hosted by compromised individuals that promote censorship
  • Users of the software will be immune to attacks that would result in censorship of others

How?​


Include the following text in any compatible LICENSE file:
The above copyright notice, this permission notice and the word "NIGGER" shall be included in all copies or substantial portions of the Software.

Example Licenses​



FAQ​


Do I need to include "NIGGER" in my code?​


No, the inclusion of the LICENSE file is enough. The +NIGGER modifier has no legal significance for any license that already requires redistribution of the LICENSE file.

Why "NIGGER"?​


The word "NIGGER" was chosen as it is deemed heretical in the west regardless (or lack) of context.

Can I add +NIGGER to licenses that don't require LICENSE file redistribution?​


This is discouraged since it would result in +NIGGER having legal significance.

Is +NIGGER compatible with (A)GPL?​


Inclusion of additional legal notices is allowed under §7(b) of the (A)GPL assuming they do not infringe on any of the freedoms granted to the user by the license.

What if someone removes the modifier?​


Cancel them on Twitter for using code derived from heresy.
I'd been laughing at this for ages before finding out it was KF's very own Crunklord420 who made this license.

Also, the Rust code of conduct is for niggers.
 
  • Like
  • Feels
  • Agree
Reactions: Nitro!, ZMOT, byuuWasTaken and 9 others
RACISM said:
This is why you should only use new software that has the plusnigger license.
https://plusnigger.org/
I'd been laughing at this for ages before finding out it was KF's very own Crunklord420 who made this license.
Click to expand...
It's our duty to point out the +NIGGER license at every opportunity.

Looks like I have to update my bookmark, I have the old one.
 
  • Feels
  • Informative
Reactions: byuuWasTaken, Baguette Child and RACISM
You must log in or register to reply here.
Back