# Error 2020: Too Slow



## Null (Jan 30, 2020)

In 2020, Mobile is king, and in more ways than just "a lot of Internet traffic is mobile". The biggest and best search engine in the world, *Bing*, is constantly changing how it evaluates sites for rankings and at some point in 2019 they decided that page speed matters a lot. A lot, like, a lot a lot.

There's a lot of reasons for this, not least of which is that it's genuinely just important for user experience that a page loads quickly. However, unlike in previous iterations, Bing has decided that the actual total load speed of the page matters significantly. That is, the amount of time it takes for the page to fully render and become usable, not just the time it takes to talk to the server. The Bing AI is so good now it can simulate mobile use on web pages and detect automatically usage issues that would impact a mobile user.

Late in 2019, I noticed our coveted Bing Score was tanking massively and pinpointed the likely culprit being A&H's policy of copy and pasting articles for purposes of archival and not requiring users to go to pozload sites for the actual content. Since hiding that board from the public, our Bing Score is improving.




Around the same time, I fixed a major issue in the site causing serious lag on every page. XenForo designed its `xf_session_activity` to be `MEMORY`, which locks _the entire table_ every time there's an update. That particular table is also updated every time anyone loads any page. So, if you have 3000 users doing things that table is locked and unlocked constantly and no page can load while that table is locked because it has to be unlocked to do the commit to load the page. In short, clusterfuck, seriously hurt the site and just by switching it to `InnoDB` it was fixed.

Now that I've seen the results I started looking at what other insights I could to determine why our Bing score was so hurt in 2019 and the big one, outside of just slow webserver, is slow page speed on mobile. Their test conditions use a standard consumer Windows phone like everyone has and checks to see when the page actually renders.



The big issue here is that XenForo just isn't fucking designed to complete this test. I've been trying to find ways to cheat it, but really, it's a huge problem with how the pages are done and I can't fix it easily. Sure, the site is optimized so that on subsequent page loads everything is cached, but for the first load it takes about 10 seconds at least for all the resources to download. The thing it hates the most is avatars, especially _GIF_ avatars_,_ and I'm considering removing avatars entirely for mobile view. I'm using the default skin right now instead UI.X and it's not really any faster so I've ruled that out.

Meanwhile, the server still really isn't that fast and I can't figure out why. It's like there's 500ms of overhead on every page load and I can't pinpoint it to save my life. It's nothing to do with PHP, the actual page generation is within 150ms now.

It's all very tiring and I just wanted to complain.


And honestly, I feel this sort of prioritization is an active effort on Bing to destroy discussion sites. It's not just us, every forum I know is getting fucked by these changes. What they want to do is have web pages designed for mobile that load in 0.5s on enterprise cloud infrastructure (that no independent operator can get without playing by the cloud's rules), and they want those pages to be Wiki articles or Real News :tm: articles with a headline and thesis sentence that impart Goodthink and satisfy the Bing query immediately. Opening these sorts of sites with 1000 weirdos and their avatars talking about people and current events without any kosher editorial policy is bad for their M.O. and they want to get rid of it.


----------



## OwO What's This? (Jan 30, 2020)

have you tried turning it off and on again


----------



## HOMO FOR LIFE (Jan 30, 2020)

What's Bing?


----------



## Gar For Archer (Jan 30, 2020)

Is it feasible to hide avatars for not-logged-in users using XenForo? That may be a solution that satisfies both Bing’s conditions and user usability.


----------



## xX_rAcE_wAr_420_Xx (Jan 30, 2020)

Why is every search engine company such a faggot?

It's good work though, between this and highlights it makes skimming through a big thread much easier.



Gar For Archer said:


> Is it feasible to hide avatars for not-logged-in users using XenForo? That may be a solution that satisfies both Bing’s conditions and user usability.


Possibly an option that's enabled by default? So users who are no fun can just disable avatars for their view?


----------



## Null (Jan 30, 2020)

Gar For Archer said:


> Is it feasible to hide avatars for not-logged-in users using XenForo? That may be a solution that satisfies both Bing’s conditions and user usability.


I've thought about it. I think the simplest things I could do are:
1. Continue to work on the server and figure out what that fucking nightmarish mystery overhead is.
2. Hide user avatars for guests and find a way to enforce thumbnailing for all images.

But even then, I need to
3. Completely redo how XenForo does CSS.

I've tried cutting up the CSS every which way but the site is simply not designed to load fast the first time. It just doesn't seem worth it but that alone is the majority of the Bing Score.


----------



## johnjohn99 (Jan 30, 2020)

just give google access to all of our I.P.'s so they will list the farms again


----------



## Exigent Circumcisions (Jan 30, 2020)

I'm a completely retarded faggot but is having chat enabled on the home page maybe causing part of the issue? My subjective experience on mobile is that the home page won't work properly until the chat is completely loaded; I have to wait quite a while before clicking a link because my phone inevitably decides I was clicking on something else.

Edit: thought about it some more and chat doesn't load for guests so probably not that, but it still sucks.


----------



## Thomas Paine (Jan 30, 2020)

The server overhead is amplified while browsing on a VPN. Response times on interactive fields (chat, thread replies, buttons) and UI elements are insane. Sometimes they fail to load or function entirely.

Before anyone chimps out, it's not my VPN, other sites using similar architecture are much more responsive.
The frustration actually dissuades me from posting at times.


----------



## Ivan Shatov (Jan 30, 2020)

I've noticed the page load time problem going through old threads. I'd be interested to know what the load time is on user profiles and user post pages, those seem to take the longest (and are probably the least frequently used.)

I wonder if it's possible to do the following:

1) Cache pages for anonymous users, ensure search engines only see pages loaded from cache. This could improve response times without degrading the experience for people posting on the site.

2) Put a spider in place to cache old pages. Can be done without eating bandwidth if the spider is on the local network.


----------



## WeWuzFinns (Jan 30, 2020)

Null said:


> I've thought about it. I think the simplest things I could do are:
> 1. Continue to work on the server and figure out what that fucking nightmarish mystery overhead is.
> 2. Hide user avatars for guests and find a way to enforce thumbnailing for all images.
> 
> ...


Seems like you're up to some big brain problem solving. Keep at it null you'll crack it eventually.


----------



## registered 2 hide avatars (Jan 30, 2020)

Not our Bing Score!


----------



## MrTroll (Jan 30, 2020)

Let's go back to phpBB.


----------



## AnOminous (Jan 30, 2020)

Phoneposters are trash and now they're fucking up the site just by existing.  Die Bing die.


----------



## Lurkism (Jan 30, 2020)

Downscale background to 144p.


----------



## byuu (Jan 30, 2020)

Convert every gif avatar to a 16-color palette. It's what God wants and is pleasant to create.
Plus, dithering is comfy.

A third of the file size for @Exigent Circumcision's avatar.


----------



## OwO What's This? (Jan 30, 2020)

garakfan69 said:


> Convert every gif avatar to a 16-color palette. It's what God wants and is pleasant to create.
> Plus, dithering is comfy.
> 
> A third of the file size for @Exigent Circumcision's avatar.
> View attachment 1122767


or just convert them to APNG, which every browser supports now


----------



## Lurkism (Jan 30, 2020)

*Laughs in Bing*


----------



## Kosher Salt (Jan 30, 2020)

Ivan Shatov said:


> 2) Put a spider in place to cache old pages. Can be done without eating bandwidth if the spider is on the local network.


Caches don't work that way. You'd just reduce the cache's performance by doing that.

The cache isn't big enough to hold the whole site; if it was, you'd just put everything there to begin with. It just holds the pages that are accessed most recently/frequently, so that those pages will load faster. By loading a bunch of pages into the cache that aren't actively being accessed, by definition you're pushing pages out of the cache that _were_ being accessed... which means that the site slows down even more.


----------



## King of Not Playing Games (Jan 30, 2020)

InnoDB? Is this rustbucket powered by MySQL?


----------



## Pope Negro Joe the XIIIth (Jan 30, 2020)

Just an odd question related to the search function, is there a reason it tops out at 8 pages?


----------



## hauser (Jan 30, 2020)

install gentoo


----------



## vertexwindi (Jan 30, 2020)

But I love my animated avatar


----------



## Exigent Circumcisions (Jan 30, 2020)

AnOminous said:


> Phoneposters are trash and now they're fucking up the site just by existing.  Die Bing die.


That's fair, but some of us are out here building the basements that the rest of you post your memes from.


----------



## kūhaku (Jan 30, 2020)

Make it so that to access the site at all when not logged in you need a captcha? If the bing AI can only access that then it would think it’s fast.

Also holy shit, shoutbox is so unbelievably laggy on mobile, that is probably a big issue


----------



## HOMO FOR LIFE (Jan 30, 2020)

*get a better server, fgt*


----------



## NOT Sword Fighter Super (Jan 30, 2020)

Bing is the best?
I’m not sure if that was a joke or not.


----------



## Large (Jan 30, 2020)

HOMO FOR LIFE said:


> What's Bing?


*Bings your head* OwO


----------



## mindlessobserver (Jan 30, 2020)

Careful null. If you keep digging into the inner workings of the internet you might fall through into a cave full of mole people.


----------



## Lurkism (Jan 30, 2020)

You could convert all the gif's into mp4's


----------



## CIA Nigger (Jan 30, 2020)

xX_rAcE_wAr_420_Xx said:


> Why is every search engine company such a faggot?


Because they know the world uses them to find information, so they can get away with pushing corporate approved websites in your face and penalizing sites that don't confirm to their idea of the web. As AMP and material design proved, whatever Google wants Google gets.

Google has even more pull on how the internet operates because they run the #1 search engine and the #1 web browser. If Google depreciates some protocol or something you'd better fall in line or else your site gets delisted and Chrome users can't view it. To add to that, even Microsoft Edge and Opera are now based on Chrome. The only big browser that isn't Chrome or Webkit based is Firefox and they've been dropping the ball with shit like the famous plugin expiration issue, killing npapi plugins, or the layout changes. It's even worse than the situation that led to the Microsoft lawsuits in the 90s.


----------



## Julay-chan (Jan 30, 2020)

1. Which web server are you using?
2. If you're using fascgi, check your fastcgi opts.
3. Try profiling PHP's built-in server vs. your production webserver. It might help you figure out whether you need to profile the PHP code or your webserver configuration.
4. Are you profiling the time spent in each function in PHP's side?


----------



## Floop (Jan 30, 2020)

CIA Nigger said:


> To add to that, even Microsoft Edge and Opera are now based on Chrome. The only big browser that isn't Chrome or Webkit based is Firefox and they've been dropping the ball with shit like the famous plugin expiration issue, killing npapi plugins, or the layout changes. It's even worse than the situation that led to the Microsoft lawsuits in the 90s.


----------



## Saladin (Jan 30, 2020)

No offense, but does this actually matter? Like, at all? The Farms isn’t exactly some underground autism cabal (despite what some in the media claim), so it’s not exactly hard to find this place even if the SEO isn’t the best. I assume the Farms gets more traffic from word of mouth than search engines.


----------



## Hecate (Jan 30, 2020)

if I understand correctly, Bing would viewing the site as a "guest" user correct?
would it be feasible to test a guest stylesheet that loads everything in smaller and less detailed? or is the overhead unrelated to file sizes?

I don't know a lot about xenforo or css so this is probably retarded sounding


----------



## Takodachi (Jan 30, 2020)

in the year of our lord, people still use bing?
why?


----------



## Lurkism (Jan 30, 2020)

Saladin said:


> No offense, but does this actually matter? Like, at all? The Farms isn’t exactly some underground autism cabal (despite what some in the media claim), so it’s not exactly hard to find this place even if the SEO isn’t the best. I assume the Farms gets more traffic from word of mouth than search engines.


Null wants more boomers to use the site.


----------



## LTO Data Cartridge (Jan 30, 2020)

Saladin said:


> No offense, but does this actually matter? Like, at all? The Farms isn’t exactly some underground autism cabal (despite what some in the media claim), so it’s not exactly hard to find this place even if the SEO isn’t the best. I assume the Farms gets more traffic from word of mouth than search engines.


I was wondering the same thing. Do we really benefit from going for constant growth?


----------



## Julias_Seizure (Jan 30, 2020)

If the bing algorith tests speed as a guest would it be possible to set it so until you make an account no gifs or even avatars show up and its as bare bones as possible by default for guests unless you change settings or make an account?


----------



## MuuMuu Bunnylips (Jan 30, 2020)

LTO Data Cartridge said:


> I was wondering the same thing. Do we really benefit from going for constant growth?


Cancerous growth is best growth.


----------



## Null (Jan 30, 2020)

LTO Data Cartridge said:


> I was wondering the same thing. Do we really benefit from going for constant growth?


Of course. We're only barely financially stable at the moment, after years, and the way it's set up I can lose it basically any time. Exploring what sort of uses this place can serve is important if I'm to keep running it. I'm only getting older and eventually I'll have to start making choices. I'm not going to allow myself to turn 30 without a plan.


----------



## Trigger Me Timbers (Jan 30, 2020)

Hey Null, I think this is all really insightful and I for one want to stick my flag in the ground as a proud mobile user  

Question, a personal theory I have is that Bing also weighs in a pages age as a big factor, have you found this to be the case?

FYI being so mobile friendly is one of the reasons I post here so much, when I’m at a computer I should be working so it’s nice on my breaks/time off I can open my phone and see what my fav Lolcow is up to.


----------



## Coffee Shits (Jan 30, 2020)

Is it feasible to render the output of css.php to a static file, or would that fall under doing over XenForo's CSS system? I ask because I checked the source and it's called on every page load and potentially doing so dynamically.

If I had to guess, it's concatenating a bunch of pre-minified CSS files and sending them to output, and this takes kind of a while. The response does seem to be cached on the server - if I repeat a request it's complete in 100-400ms but the first time (based on futzing with the k, s, l, and d params - they appear optional?) it's anywhere from 400-2400ms on my connection. The response has a valid Cache-Control header, but if you're concerned with first load rendering the result to a static file might be a good way to shave off a few milliseconds. Is XenForo spinning up a new instance every time it runs this file?


----------



## Looney Troons (Jan 30, 2020)

Mobile user primarily here. I wouldn’t mind if you got rid of the avatars, but ngl, I recognize posters first from their avatars. This would only be a minor inconvenience, in my opinion.  I’d actually have to pay attention.


----------



## Lurkism (Jan 30, 2020)

Another solution to the GIF problem, would be getting the first frame of the GIF that would reduce the to a comfy size under 40kb (for guest users).


----------



## Stoneheart (Jan 30, 2020)

ad victoriam, ex machina, non sibi sed fundus.


----------



## Odessa Norwood (Jan 30, 2020)

I don't know if this will help or hinder, but I've had this message a few times tonight.

Edited to remove IP address.


----------



## James Smith (Jan 30, 2020)

Odessa Norwood said:


> I don't know if this will help or hinder, but I've had this message a few times tonight.


It helps you dox your IP.


----------



## 7proxy (Jan 30, 2020)

Null said:


> It's like there's 500ms of overhead on every page load and I can't pinpoint it to save my life. It's nothing to do with PHP, the actual page generation is within 150ms now.



I assume the "150 ms" figure is from XenForo's debug output at the bottom. How did you measure the overhead? Did you eyeball it by doing TTFB - 150 ms?

If the page generation is fast but there's an unexplained delay somewhere, I'd be thinking of some kind of resource contention. Check FPM's log, see if it's running out of available processes, or if it's having issues spawning new ones. Check out OPCache's status to see if it's not running out of memory somehow. That won't be it because it would also increase the time spent within XenForo but it's worth checking, especially anything related to revalidation and locks because they could hold up page generation *before* XenForo starts the timer.

After running out of low-hanging fruits I'd check the kernel's logs to see if it's not running out of descriptors. That could explain the Cloudflare timeouts... maybe.


----------



## Pond Scum (Jan 30, 2020)

Odessa Norwood said:


> I don't know if this will help or hinder, but I've had this message a few times tonight.
> 
> View attachment 1123035


Come on now, people work hard for those Kiwi IPs!


----------



## Odessa Norwood (Jan 30, 2020)

SoapQueen1 said:


> It helps you dox your IP.



Bother, put up the wrong screenshot.   I'll change it now.

Edit to remove random V.


----------



## A shitty ass clover (Jan 30, 2020)

Mr. Moon: Better SEO optimizer that a fat tranny using google glasses..


----------



## Lurkism (Jan 30, 2020)

Odessa Norwood said:


> I don't know if this will help or hinder, but I've had this message a few times tonight.
> 
> Edited to remove IP address.


That's normal behavior of the farms.


----------



## Operator of the Wired (Jan 30, 2020)

Lurk more said:


> You could convert all the gif's into mp4's
> View attachment 1122837


Second, but vp9 WebMs instead. The following line has slimmed down my videos by insane degrees, but needs AMD graphics hardware acceleration to run fast.
*ffmpeg -i "$1" -c:v libvpx-vp9 -c:a libopus "$1"-slim.webm*


----------



## HOMO FOR LIFE (Jan 30, 2020)

Let me tell you my website traffic breakdown. 

99% Google. 

1% Bing.

Null: omg I need to dedicate 99% of my time to Bing optimization.

Also null is good and mighty. Pls don't smite.


----------



## slartifartblast (Jan 30, 2020)

When I use the site on mobile, I tend to use the Stealth CSS option anyway, because meh avatars, but from what I gather it loads them anyway


----------



## Lurkism (Jan 30, 2020)

Why not switch guest users to a different theme that doesn't have avatars?


----------



## Neil (Jan 30, 2020)

HOMO FOR LIFE said:


> What's Bing?


The best place to save and hide porn searches if you don't want to use incognito mode.
Because no one will ever think to check there.


----------



## Save Goober (Jan 30, 2020)

I can believe Bing is punishing forums as I've casually noticed this too. Bing constantly changing their rules can only harm small sites that don't have a team to keep up with and respond to all the changes and use fancy new technology.
I also really liked the comfy alternate universe feel of this post. I personally love how Windows runs on my Nokia phone, and Blockbuster streaming has fewer issues on it than Blackberries.


----------



## The Fool (Jan 30, 2020)

> And honestly, I feel this sort of prioritization is an active effort on Bing to destroy discussion sites...



I mean, if I was going to assume a conspiracy, it would probably just be for selling more Azure instances. But honestly I take this at face value, I think this is just to make their face a little shinier, to make normies go "sites load faster when I look them up on bing!" because normies really are that stupid. MS knows normies are this stupid. It's the same reason Windows is Windows.

Here's an extreme idea, what if you generated some kind of bare-minimum Web 1.0-style cache of the site that showed up for guests? Periodically generate the cache and show the cache immediately to clients instead of on-demand. It would look like shit and may potentially drive away new users who don't realize there's a much better looking site available after you sign up, though.

I don't know, I'm just spitballing here. I'm just thinking like, what lightweight alternatives could you serve to guests. A guest needs exponentially less interactivity than a user does, so maybe you should use different software entirely for them.


----------



## BOLDYSPICY! (Jan 30, 2020)

Oh, I was gonna tease @Null about breaking the site to fix an inconsequential thing that only bothers him because I haven't had issues with loading until jihad was declared, but then I realized that it's probably because I almost exclusively browse on mobile. I am the cancer that is killing the Bing score. Feels bad man.

. . .actually, why IS that? My smooth woman brain is bad at computer. Is it because the mobile version of the site is a bit more compact? Like an HTML version of a site that loads faster than one with all the bells & whistles?

*EDIT: *or wait, do I have it the wrong way around? I'm very tired.

My point is I don't have issues on mobile. But I also exclusively use Brave. Does that matter? What browser do other mobile users use?


----------



## FuckedUp (Jan 30, 2020)

Try getting the site banned in more countries. Pretty sure the block page loads much faster.


----------



## Angel Dust (Jan 30, 2020)

BOLDYSPICY! said:


> Oh, I was gonna tease @Null about breaking the site to fix an inconsequential thing that only bothers him because I haven't had issues with loading until jihad was declared, but then I realized that it's probably because I almost exclusively browse on mobile. I am the cancer that is killing the Bing score. Feels bad man.
> 
> . . .actually, why IS that? My smooth woman brain is bad at computer. Is it because the mobile version of the site is a bit more compact? Like an HTML version of a site that loads faster than one with all the bells & whistles?
> 
> ...



I exclusively use brave on mobile as well. Smooth woman brain also not good at computer. Plus can browse on phone at work. The site on/off loads like anus for me. Sometimes it's super fast and other times I'm just starting at a screen going "load! Load god damn it!". Mostly though I'd say it's fair to average speed? I agree with maybe hiding the chat or putting it elsewhere. That seems to cause the most issues for me.


----------



## HumanHive (Jan 31, 2020)

The fact @Null spares even a second to think about Bing is extremely troubling and I question the site's long term future.
But optimization is always nice.


----------



## Tealeaf (Jan 31, 2020)

HumanHive said:


> The fact @Null spares even a second to think about Bing is extremely troubling and I question the site's long term future.
> But optimization is always nice.


None of the optimizations are Bing specific. Google prioritizes fast sites in its algorithm as well. Faster site also means happier  farmers.


----------



## BisquitDoughHandsMan (Jan 31, 2020)

I mostly just feel sorry for Null. If you like, Plan B After 30 can be living in the woods behind my house. You can befriend as many dogs as you want!


----------



## Freya (Jan 31, 2020)

BisquitDoughHandsMan said:


> I mostly just feel sorry for Null. If you like, Plan B After 30 can be living in the woods behind my house. You can befriend as many dogs as you want!


Well thats a hard offer to refuse


----------



## BoJack Horseman (Jan 31, 2020)

You all are forgetting about one detail, Bing uses Oath which is what other engines such as DuckDuckGo use so I believe optimizing for Bing will have a positive effect on the other non-Google engines as well. Correct me if I'm wrong tho.


----------



## now watch this drive 911 (Jan 31, 2020)

Seems to be largely TTFB and Queuing: blame Cloudflare?
What do the response times look like when you're hitting the origin directly?
Have you managed to isolate the lag to network or server?

that 900ms wait on index tho lmao


----------



## CaesarCzech (Jan 31, 2020)

force mobile view by default, you want normal view  register.  Cheat bing by basically building potemkin village.


----------



## Null (Jan 31, 2020)

now watch this drive 911 said:


> Seems to be largely TTFB and Queuing: blame Cloudflare?


I don't know. Here's a basic PHP file, so this is time to spin up FPM and get a response.

https://kiwifarms.net/ttfb.php - thru CF
https://kiwifarms.nl/ttfb.php - local reverse proxy

On .nl, TTFB is between 30ms and 600ms, usually on the higher side.

People keep saying "HAVE YOU TRIED TO RAISE THE FPM CHILD COUNT???" like I haven't been doing this shit my entire fucking life and I don't know the first thing about fucking tuning and I haven't been mitigating DDoS attacks and I haven't been doing shit professionally since I was fucking goddamn 19 years old

It's not even that I can look strictly at hardware because I don't just own my servers, I own my router. Could there be something about the router that is causing a huge latency to the server? Between the servers since I distribute the stack over multiple? Maybe the SDD is bad? Maybe there's some weird LXC configuration? I don't know, I don't fucking know. There's now 10,000 moving parts and if any of them take half a second the entire thing is completely fucked.

I don't think it's MySQL. Since my last batch of fixes to MySQL, I see between 17 to 150 idle connections to the database from FPM. Nothing is ever hanging. There's no max_client issues.

I also don't think it's PHP. If I run blackfire via CURL, I get results like this.

blackfire curl --samples 10 https://kiwifarms.net/threads/richard-c-kyanka-v-ashley-k-martin.65737/
Wall Time     154ms
I/O Wait     17.6ms
CPU Time      136ms
Memory       5.36MB
Network         n/a     n/a     n/a
SQL          15.5ms    18rq

Excellent! 154ms. Perfect. Bam boom. I've looked at its traces and bottlenecks and sorted all those out, which is why the site should feel faster as soon as the initial waiting period is over. Once you hit that 600ms wait, the page is loaded.

I also have looked at the different local servers talking to each other and they work fine as well, so I don't think it's FPM to Redis / MySQL wait.

If I profile from my local debian installation I get similar results.

blackfire curl --samples 10 https://kiwifarms.net/threads/richard-c-kyanka-v-ashley-k-martin.65737/
Wall Time     162ms
I/O Wait       23ms
CPU Time      139ms
Memory       5.36MB
Network         n/a     n/a
SQL          17.5ms    18rq

I'm very certain it's not PHP, it's something weird. Why does it take 600ms to connect to the server? The ping is 173ms to the device and that's consistent (+3) with the datacenter from Europe.


----------



## An Ghost (Jan 31, 2020)

LTO Data Cartridge said:


> I was wondering the same thing. Do we really benefit from going for constant growth?


We benefit from people being BTFO.  There’s lots of cows who, when you search their name, you get KF as the first result. Beating out their YouTube channels and Facebook pages. In this day and age it’s so important to be in the top few if you want anybody to see your stuff. Bigger cows, we’re usually on the first page. But imagine being on the _second page_ of google. Then it’s like what’s the point?

Also we can’t gut the guest user experience to up bing score. All the lurkers we already have get the short end of the stick, and new potential users say “ this site looks like aids” and leave.


----------



## Null (Jan 31, 2020)

What's further driving me crazy is this. This bypasses all of Cloudflare and is just hitting an HTML page, so it also eliminates any FPM slowness.

There's 2 kinds of loads.

The Chad Bing Score:




The Virgin 600ms:




#2 is obviously the Jared Leto _Damaged_ that is causing problems and it happens every other load. It's not even a flat increase because of SSL. It takes 180ms to talk to the server, and another 180ms to do the SSL handshake, and then the actual response time for the site is about twice as slow as it should be (again, just to pull an HTML page and send it).

This also has to be an issue with just the KF server because other PHP websites hosted on the same fucking hardware has no issue what so ever.


----------



## Swiss46 (Jan 31, 2020)

Try downloading ram and uploading it to the site! It made my computer run Runescape faster, surely it will work for a website.


----------



## now watch this drive 911 (Jan 31, 2020)

Here is request for single jpg.

TCP establish connection (SYN, SYNACK, ACK) - 235ms (tcp stream 6 closed so 7 is actual (470ms total time))
TLS handshake (Client Hello, Server Hello) - 237ms

And the rest is history.

Latency is a serious thing in real time systems! This is why network file-systems (NFS) can be ass if used wrong.

These 600ms are - from our similarly distant locations - the baseline value for latency. Everything else that happens tacks onto that figure. If the front page is rendered dynamically each time it's requested, that's cpu+db+php time + download time for each asset + 600ms. Six parallel connections on x64 chrome, so there's room to move a lot of data.

There's 63 requests for this page, in chrome that happens across 6 parallel threads. Drop the total number of requests if possible. Turn the stickers into a sprite set? Serve JS and CSS in a minimized bundle. x64 Chrome is 6 threads but perhaps ARM+Cellular is less?

Note: I tested twice and got a TCP-RST in the first tcp stream both times. I think this is actually chrome doing something with hard-clearing the cache. Recommend you do your own tests to isolate and crosscheck my results.


----------



## Otterly (Jan 31, 2020)

Tinfoil hat on, but are you sure the problem is coming from within your systems, and not from without?
Edited to add: search engines can delist you, and that’s obvious, but it wouldn’t surprise me if they had ways of just subtly down rating sites as well. These ‘nudge’ policies are widespread in health and politics. 
  Apologies if that’s ridiculous. I know f all about computers, but a bit about the dark arts of behaviour manipulation


----------



## kiwifarmsfan1 (Jan 31, 2020)

For site accessibility and convenience, typing in a keyword to access the A&H forum as an alternative to logging in would be nice.  It can be simple like "kw" or something more creative.  Only humans and spam bots will enter a word into a wordbox, search engines will just pass over the page.  It was nice browsing incognito without having to type in a long pw.  I wonder if bing will downgrade a site if a bunch of pages load instantly but just contain a wordbox?


*now watch this drive 911's comment about the network stack and the filesystem should really be investigated, always go one level lower than you are comfortable with.*



Null said:


> 3. Completely redo how XenForo does CSS.
> I've tried cutting up the CSS every which way but the site is simply not designed to load fast the first time. It just doesn't seem worth it but that alone is the majority of the Bing Score.



You might just have to think out of the box. I don't think of this as the best solution, but just to prove it can be done. From the worst option, webpages didn't always have css, to the second worse option, there are simple styles like those used in print page mode on forums that allowed you to print discussions.  For the third worse but an improved option, themes for logged in people, logged out get a slightly to moderately uglier but fast version.


----------



## Thomas Paine (Jan 31, 2020)

I'll be completely transparent, I have a barely-functioning grasp on networks. That said, Is there anyway that it might be a throttling issue from the server's physical data center or a provider along the network chain?

Time to crowdfund some server blades for Null.


----------



## Null (Jan 31, 2020)

Thomas Paine said:


> I'll be completely transparent, I have a barely-functioning grasp on networks. That said, Is there anyway that it might be a throttling issue from the server's physical data center or a provider along the network chain?


I do not believe the issue is the actual server.


```
curl -w "@curl_format.txt" -so /dev/null https://kiwifarms.net/
   time_namelookup: 0.000311
      time_connect: 0.000375
   time_appconnect: 0.006830
  time_pretransfer: 0.006910
     time_redirect: 0.000000
time_starttransfer: 0.104205
                   ———
        time_total: 0.104407
```

When done locally, the entire request takes well under 200ms. That is with PHP proccing. There is something specifically, and intermittently, wrong about outside requests. What's weird is this latency isn't found outside of this particular site. Other, small sites I run on the same hardware have no mysterious overhead. I have to be doing something wrong but I can't find it. There's just a flat 150ms lag on the initial connection, another 150ms lag on the SSL handshake, and then another 150ms lag before the page loads. Sometimes. The other half of the time it loads instantly.


----------



## Null (Jan 31, 2020)

after fucking around I'm deadass certain the issue is that:

1. we get 100 req/s now
2. nginx needs at least the same number of processes as php or else there's that mysterious overhead waiting for a nginx process
3. php needs at least 64 static processes otherwise you get sporadic 502 errors
4. the front-end is an 8 core server with 128 processes and the new bottleneck preventing coveted *site fast* status is waiting on a CPU, though our CPUs are mostly idle

tell me if this sounds dumb, because I've tried rolling with fewer of each and it caused serious issues.


----------



## Shoggoth (Jan 31, 2020)

Null said:


> Sometimes. The other half of the time it loads instantly.





Null said:


> tell me if this sounds dumb, because I've tried rolling with fewer of each and it caused serious issues.


Sounds like some queue somewhere is exploding then the system's latency goes through the roof.
If you feel like watching a 20min out of a 40min technical talk, maybe this'll give you some ideas:








						Zach Tellman - Everything Will Flow
					

Queues, it seems, are back in fashion. This is exciting, as queues let us explore problem spaces that were unreachable by pure, immutable primitives. But que...




					www.youtube.com
				



Is there any way you can profile the server and see what each thread is doing when it's hanging?


			CPU Flame Graphs
		



			Off-CPU Flame Graphs
		






						How to generate PHP Flamegraphs
					

In my quest to optimise the PHP sites I work on, I found that the Flame Graph repository contains a script to convert XDebug traces.  Flame graphs, created by Brendan Gregg, visualize the most frequently hit code from profiled software. The x-axis represents the profiled stack traces, ordered...



					daniellockyer.com


----------



## Null (Jan 31, 2020)

It's not PHP, I've used black fire to check the scripts and solve those bottlenecks. If it is, it's the fpm and not the scripts


----------



## LazarusOwenhart (Jan 31, 2020)

@Null you said Internet words which I presume mean you're working hard so people like me can talk shit online. God speed you mad bastard!


----------



## ⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛ (Jan 31, 2020)

Null said:


> after fucking around I'm deadass certain the issue is that:
> 
> 1. we get 100 req/s now
> 2. nginx needs at least the same number of processes as php or else there's that mysterious overhead waiting for a nginx process
> ...


It _sounds _like it makes sense..

Assuming this was the case, is there anything straightforward you can do to reduce requests served? I mean, looking at a fresh page load the fontawesome fonts and stuff could presumably come from a CDN, but unless people's browsers are seriously misconfigured that's not going to account for many real requests.

Do you know if Cloudflare is being allowed to cache avatar images* and stuff, or could something be causing those requests to be passed on through? Probably a stupid question, I know.

* dumb question, they show up as cf-cache-status:HIT when I look


----------



## Null (Jan 31, 2020)

3119967d0c said:


> Do you know if Cloudflare is being allowed to cache avatar images* and stuff, or could something be causing those requests to be passed on through? Probably a stupid question, I know.


Cloudflare caches a lot of stuff. If we ran without it as a CDN, our requests/sec would probably triple.


----------



## ⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛ (Jan 31, 2020)

Null said:


> Cloudflare caches a lot of stuff. If we ran without it as a CDN, our requests/sec would probably triple.


Yeah, definitely not seeing Cloudflare cache misses for avatars, attachments, or proxied images on active threads. Are the requests that make it through that significant (from people browsing totally mothballed threads, or going through every page of some of the ancient ones)? I mean if I read the headers correctly they should be cached on CF for a month, whether it's possible to extend that I don't know.

How about the chat? I see it makes a request every five seconds when you're on the front page, even if you can't see it and don't have the chat message box selected- or 1 second if either of those are true. If you've got 50 people idling on the front page _without viewing the chat _that's only ten requests a second, but maybe it's more? It seems like at least the idle cycle could be slowed down without killing interactivity with the chat.


----------



## Null (Jan 31, 2020)

Chat is probably the biggest culprit of consuming php-fpm cycles. I'll try turning it off tomorrow.


----------



## Shoggoth (Jan 31, 2020)

I wasn't talking just about php, you can profile the entire server to try and find the bottleneck, because it seems like you have one.


----------



## 7proxy (Jan 31, 2020)

@Null Have you looked into the WAF rules in your Cloudflare config? Maybe too many rules or a regexp that backtracks too much like last year when they DoS'ed themselves?

From where I sit, I see the same >600ms TTFB on requests that can't be served from Cloudflare's cache regardless of it being a static resource or a CGI request. On the other hand, the same requests (static or not) on .nl are much faster. If only the requests from CF to you have that big TTFB then it's got to be something specific to CF, which makes me think of WAF.


----------



## אΩ+1 (Jan 31, 2020)

Please Remember that this exists.
URL: https://riot.kiwifarms.net/#/welcome

Here's the thread about our official instance of Matrix / Riot.
URL: https://kiwifarms.net/threads/official-kiwi-farms-riot-matrix-server.53185/


----------



## Suburban Bastard (Jan 31, 2020)

אΩ+1 said:


> Please Remember that this exists.
> URL: https://riot.kiwifarms.net/#/welcome
> 
> Here's the thread about our official instance of Matrix / Riot.
> URL: https://kiwifarms.net/threads/official-kiwi-farms-riot-matrix-server.53185/


But the people on riot were weird and smelly


----------



## Bitch Kitten (Jan 31, 2020)

Null said:


> Chat is probably the biggest culprit of consuming php-fpm cycles. I'll try turning it off tomorrow.


What about just taking it off the homepage and keeping chat on the separate page that we have to click to?


----------



## ⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛ (Jan 31, 2020)

@Null FYI, still seeing some 'server errors' while searching and others (@The Last Stand et al) are experiencing the same.

This is a little difficult to follow as it looks like XenForo gives a new ID to each 'unique search' a user does for a term. So if I search for a term that fails to return a result- I was able to achieve this by searching for some common names until a search failed- even if I try searching again or refreshing, it will fail again. I imagine this is done to keep consistent pagination/ordering for search results or something. I'm sure that probably times out in some period of time, but an impatient KiwiFarms user (like myself) will just repeatedly refresh and get mad.




Now, if I use incognito to make the same search a few seconds later, new search ID and it works fine.



It seems like a XenForo bug really- whatever's causing the search to fail should either allow the same search to be made again immediately, or drastically reduce the time for the search ID to expire and use a useful error message- but it seems a lot more prevalent than prior to the rebuild. Do you have a feel as to what might be causing it? Will the search index be building up as people do searches or something like that?


----------



## AmpleApricots (Jan 31, 2020)

If you really want to have a plan in as "financially stable, not having to think what'll happen the next 1, 2, 3, 5 years" I'd recommend you to stop wasting your time on this and look for financial stability elsewhere. The modern corporate internet doesn't want places like kiwifarms and this is only going to get worse. Eventually you'll run into a roadblock, either created by lawmakers pushed into action by the corporates or by pure technical circumstances of the corporate machinery, that you'll not be able to avoid or overcome, at least not without giving up on ever turning a profit. Not that I don't value what you're doing here and you went farther than most would have gone, I just don't think this site will provide you with a retirement plan.

Comedy reply: Just become a neet in one of the european countries with healthcare and financial support for the unemployed.


----------



## The Fool (Feb 1, 2020)

AmpleApricots said:


> If you really want to have a plan in as "financially stable, not having to think what'll happen the next 1, 2, 3, 5 years" I'd recommend you to stop wasting your time on this and look for financial stability elsewhere. The modern corporate internet doesn't want places like kiwifarms and this is only going to get worse. Eventually you'll run into a roadblock, either created by lawmakers pushed into action by the corporates or by pure technical circumstances of the corporate machinery, that you'll not be able to avoid or overcome, at least not without giving up on ever turning a profit. Not that I don't value what you're doing here and you went farther than most would have gone, I just don't think this site will provide you with a retirement plan.
> 
> Comedy reply: Just become a neet in one of the european countries with healthcare and financial support for the unemployed.



If null were a coward he would have stayed in the US and faithfully consumed like anyone else.


----------



## Vecr (Feb 1, 2020)

Null said:


> after fucking around I'm deadass certain the issue is that:
> 
> 1. we get 100 req/s now
> 2. nginx needs at least the same number of processes as php or else there's that mysterious overhead waiting for a nginx process
> ...



How many nginx worker threads do you have? Are GGI/PGP threads not the same thing?

You could try `worker_processes auto;` in your nginx config, if you are not already doing that.


----------



## Null (Feb 1, 2020)

Vecr said:


> You could try `worker_processes auto;` in your nginx config, if you are not already doing that.


auto would be 8


----------



## AbyssStarer (Feb 1, 2020)

please nuke all these fags' gif avatars. Trying to read with them there is a pain in the ass and I have a shit-top so thread content takes forever to load.


----------



## now watch this drive 911 (Feb 1, 2020)

Null said:


> 4. the front-end is an 8 core server with 128 processes and the new bottleneck preventing coveted *site fast* status is waiting on a CPU, though our CPUs are mostly idle



What is the system load? Low CPU utilization with high load is a good indication of hardware exhaustion.


----------



## Null (Feb 2, 2020)

now watch this drive 911 said:


> What is the system load? Low CPU utilization with high load is a good indication of hardware exhaustion.


4.53 3.98 3.92, not high.

No fucking clue what the issue is. It's depressing.


----------



## Kosher Salt (Feb 2, 2020)

Null said:


> When done locally, the entire request takes well under 200ms. That is with PHP proccing. There is something specifically, and intermittently, wrong about outside requests. What's weird is this latency isn't found outside of this particular site. Other, small sites I run on the same hardware have no mysterious overhead. I have to be doing something wrong but I can't find it. There's just a flat 150ms lag on the initial connection, another 150ms lag on the SSL handshake, and then another 150ms lag before the page loads. Sometimes. The other half of the time it loads instantly.


Have you tried emailing the FBI and politely asking them if they can optimize the datacenter that they've got spying on us?

only partially joking....


----------



## now watch this drive 911 (Feb 2, 2020)

Null said:


> 4.53 3.98 3.92, not high.
> 
> No fucking clue what the issue is. It's depressing.



If CPU is 8 core, this may be 4 physical/4 logical - in which case 4.5 would be a bit high. Or it might be 8physical, in which it would be as you said "not high".

If not that, perhaps load on a router above is high.


----------



## Null (Feb 2, 2020)

now watch this drive 911 said:


> If CPU is 8 core, this may be 4 physical/4 logical - in which case 4.5 would be a bit high. Or it might be 8t, in which it would be as you said "not high".














						Intel® Xeon® Processor E3-1240 v3 (8M Cache, 3.40 GHz) Product Specifications
					

Intel® Xeon® Processor E3-1240 v3 (8M Cache, 3.40 GHz) quick reference guide including specifications, features, pricing, compatibility, design documentation, ordering codes, spec codes and more.




					ark.intel.com
				




Performance
# of Cores: 4
# of Threads: 8


----------



## now watch this drive 911 (Feb 2, 2020)

Null said:


> # of Cores: 4
> # of Threads: 8



I suspect this is significant to your scenario, but this is at the end of my experience. All I know for sure is that you have only 4 physical execution units.

Here's a cool tip for you or anyone following along at home:

Htop -> F2 -> display options, turn this on:



Legend is in F1. You can also turn on Kernel threads from Display Options. It also makes htop more colorful, which is good for hacking.

Good luck!


----------



## Null (Feb 2, 2020)

now watch this drive 911 said:


> Htop -> F2 -> display options, turn this on:









Is it saying all of my CPUs are choking on IOwait? How is that even possible? This is all on SSD.





wtf, what is doing this? I checked it before and it was fine


----------



## now watch this drive 911 (Feb 2, 2020)

Null said:


> wtf, what is doing this? I checked it before and it was fine



Could be a period of higher intensity web traffic, it is 10am in Australia. I see in ~1 minute you server went +30 threads +500mb ram.
How 2 find processes w/ IO wait
See if you can view IRQ and SIRQ stats (looks like mpstat may be a good tool)
It may worth reducing the swappiness to stop paging to disk? (assuming the iowait is disk related)


----------



## Null (Feb 2, 2020)

After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.

4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to milk the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.

Also, CrunkLord420 runs the Matrix/Riot and Pleroma instances on Server 4, and that server itself is running out of disk space.

Consider donating via crypto if that's an option for you.





						Supporting the Forum
					

There are concerted efforts to demonetize this site and punish me for hosting it. I cannot use traditional payment processors like PayPal, Stripe, Braintree, Patreon, et cetera. Donating to the site is very convoluted but is the only way to do it. The site's operating expenses exceed $1000/mo...




					kiwifarms.net
				




I'm probably going to do a merch run to finance this expansion soon. Will likely try selling stickers since it was highly requested last time. Keep your eyes out for that.


----------



## Dante Alighieri (Feb 2, 2020)

I await stickers as I need a swag way to dogwhistle my shitposter status!


----------



## GarthMarenghisDankplace (Feb 2, 2020)

New merch? Oh God I'm gonna, I'm gonna  CONSOOM!!


----------



## Fatal Cardiac Infarction (Feb 2, 2020)

Maybe you could run a patreon that all the troon commie kiwis could donate to then demand that you ban all the fun and cool posters cause NAZIS.


----------



## Cubanodun (Feb 2, 2020)

did not pornhub got cucked by paypal and then they invented some kind of new payment system to bypass them? totally forgot about it because pornhub is blocked here


----------



## glossdrop (Feb 2, 2020)

I can't wait to hand a marine an official KF semper fi sticker.


----------



## spiritofamermaid (Feb 2, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...


Will you ever re-use past designs for the merch run? Not specifically this one, but any?


----------



## General Disarray (Feb 2, 2020)

spiritofamermaid said:


> Will you ever re-use past designs for the merch run? Not specifically this one, but any?


I re-nominate the True & Honest Fan in the Coca-Cola script! I give my little BAT to MATI, but I could scrounge some extra shekels for additional swag.


----------



## BarberFerdinand (Feb 2, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...


How much do you anticipate it costing?


----------



## Daisymae (Feb 2, 2020)

BarberFerdinand said:


> How much do you anticipate it costing?


About the same as plane fare to LA, along with money to split rent for 6 months and ubereats for about the same timeframe.


----------



## spiritofamermaid (Feb 2, 2020)

General Disarray said:


> I re-nominate the True & Honest Fan in the Coca-Cola script! I give my little BAT to MATI, but I could scrounge some extra shekels for additional swag.


I want the kiwi halloween one, I bought the wrong size like an idiot so I want a chance to buy the right one.


----------



## AmpleApricots (Feb 2, 2020)

Don't forget to throw more harddrives at this so you get more IOPS out of your system, also as a SSD is nearing fullness the way it works will make it slower. Also tons of RAM will help indirectly with your access times on predictable things because of caching by the OS even if/when tasks don't *directly* use it. No such thing as free/unused memory in a modern OS. (to a degree)

Also check if your choice of filesystem is the most efficient one for this workload. There's also lots to tweak there too. Won't solve your upgrade problem but might shove the next upgrade a bit out into the future.


----------



## Haesindang Park (Feb 2, 2020)

At this rate, we might have the servers over in Iceland and hope they got enough IT people to manage them. We could technically guarantee 0% unemployment there due to demands of Kiwi Farms.


----------



## goodbudweiser (Feb 2, 2020)

Null said:


> I'm probably going to do a merch run ... Will likely try selling stickers.


T shirts? or mugs?


----------



## Spawn (Feb 2, 2020)

A bit over kill but what about something like this?


----------



## AlexGroans (Feb 2, 2020)

>All of this on a single hard disk
I can't believe that it's only 1.5tb, and that the poor thing isn't dead yet.


----------



## NOMYBITCASH (Feb 2, 2020)

Dude just buy a 4tb SSD 

But in all seriousness need a new flare or merch.

I'd buy the fuck out of a mug.


----------



## Give Her The D (Feb 2, 2020)

Fuck, I'll give you money for Supporter status when my next paycheck comes rolling in on Thursday.

I'll help the fuck out of you.


----------



## whatever I feel like (Feb 2, 2020)

I still want a hat, even though you refuse to let me buy one.


----------



## mindlessobserver (Feb 2, 2020)

Sooo, the Kiwifarms logo, only as a car window sticker.  Be funnier then getting a confederate flag bumper sticker!


----------



## YooA (Feb 2, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...



I've pledged $10 a month on MATI.com. All that I ask is you bring back 2:3 ratio avatars, and I'll stay subscribed forever.


----------



## Smug Cat (Feb 2, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...


Sent some shex. Keep up the good work mate.


NOMYBITCASH said:


> Dude just buy a 4tb SSD
> 
> But in all seriousness need a new flare or merch.
> 
> I'd buy the fuck out of a mug.


I, too, would buy the fuck out of some mugs.


----------



## NOT Sword Fighter Super (Feb 2, 2020)

I want my "The Void" sexy man undies.


----------



## HOMO FOR LIFE (Feb 2, 2020)

HOMO FOR LIFE said:


> *get a better server, fgt*





Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...




I am not saying I was right but. 


I was right.


----------



## Pee Cola (Feb 2, 2020)

I'd wear the fuck out of a KF t-shirt.  Mainly to trigger any spergs that happen to see me wearing it in public.  It's not like any normies know what KF is.


----------



## Stranger Neighbors (Feb 2, 2020)

Null said:


> I'm probably going to do a merch run to finance this expansion soon. Will likely try selling stickers since it was highly requested last time. Keep your eyes out for that.


Merch run. Heard that. If you don't mind disclosing how much are we needing to pool together in order to get the hardware?


----------



## Captain Hastings Official (Feb 2, 2020)

Since it hasn't been posted on this thread yet, I'm going to shill for Brave and point out that it makes it _incredibly _convenient to acquire BAT and then funnel them to Null. Getting enough BAT to send him $5 or $6 a month takes almost no effort.

I especially like it when charities advertise, so that you get the very particular pleasure of taking the Rainforest Foundation's money and giving it to a weirdo who makes fun of fat people.


----------



## IAmNotAlpharius (Feb 2, 2020)

mindlessobserver said:


> Sooo, the Kiwifarms logo, only as a car window sticker.  Be funnier then getting a confederate flag bumper sticker!



Then put it on random cars...


----------



## Optimus Prime (Feb 2, 2020)

mindlessobserver said:


> Sooo, the Kiwifarms logo, only as a car window sticker.  Be funnier then getting a confederate flag bumper sticker!



I'd put one on my car.


----------



## Right To Bear Blarms (Feb 2, 2020)

Spawn said:


> A bit over kill but what about something like this?


Server hardware running older Xeons are pretty widely available for pennies by comparison. 

Something like a Dell Poweredge Rx20 which run two E-5 Sandy Bridge sockets or
HP servers like the Proliant DL series 3xx from generations 6-8 are also dirt cheap running the same sockets. 
Might require a bit more of scrounging for parts but the market is supersaturated with those things.


----------



## GethN7 (Feb 2, 2020)

Well, that sucks.

Brave was kind enough to drop a ton of internet shekels in my lap recently, just sent some here.


----------



## one_time_user (Feb 2, 2020)

@Null too bad you can't find a place to make stickers or whatnot that would let you do an additional sticker (or whatnot) and you could charge $25 for it so people who don't do the currency you take could get you straight up profit money.


----------



## Heinous Fuckery (Feb 2, 2020)

I'd tip heavy for a second chance at "the void" tshirts.


----------



## soft kitty (Feb 2, 2020)

I'm all good on shirts, but I would love to see some stickers, maybe a mouse pad, or a travel mug. Or shower curtains, that'd be dope.


----------



## Rhysuu (Feb 2, 2020)

Subscribed to your Podcast, love you Nool

If you bring back the shotglasses for another merch run I'll buy them


----------



## ??? (Feb 2, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...


Ban uploading of videos, music files, and images in off-topic boards. We can use media hosting sites to bitch about black people.


----------



## Jaded Optimist (Feb 3, 2020)

I bought a void hoodie last time, i will buy all the stickers this time.


----------



## BONE_Buddy (Feb 3, 2020)

I am dropping the 137 BAT I saved up by being too lazy to donate earlier to madattheinternet.com

So that is like $30 into the bucket.

Still too cheap to donate anything other than "free" internet money.

Edit to signal my virtue like the internet attention whore I am.


----------



## HumanHive (Feb 3, 2020)

As foretold, YHWH has cursed this site and will soon drain all financials from the false and dishonest unbelievers under Null's thrall. Praise to the mountain jew for our deliverance from the accuser.


----------



## BlancoMailo (Feb 3, 2020)

BONE_Buddy said:


> I am dropping the 137 BAT I saved up by being too lazy to donate earlier to madattheinternet.com
> 
> So that is like $30 into the bucket.
> 
> ...



You forgot to shout "Match me! ...and no refunds"


----------



## Leibowitz (Feb 3, 2020)

RightToBearBlarms said:


> Server hardware running older Xeons are pretty widely available for pennies by comparison.
> 
> Something like a Dell Poweredge Rx20 which run two E-5 Sandy Bridge sockets or
> HP servers like the Proliant DL series 3xx from generations 6-8 are also dirt cheap running the same sockets.
> Might require a bit more of scrounging for parts but the market is supersaturated with those things.


Reddit isn't good for much, but HomeLabSales is one of the good things still left on that site.

There's also LabGopher.


----------



## ⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛ (Feb 3, 2020)

dinoman said:


> I'm all good on shirts, but I would love to see some stickers, maybe a mouse pad, or a travel mug. Or shower curtains, that'd be dope.


I think we're all missing an opportunity here.

Do you not want a Void bedspread? Perhaps some cardboard box covers with the KF logo for your side-tables?


----------



## Cheesetoast (Feb 3, 2020)

I already donate by BAT and to MATI but will happily throw in additional shekels for access to the Semper Fi and Drink! emojis.


----------



## TwinkLover6969 (Feb 3, 2020)

2020 and not running everything on the CLOUD smh


----------



## Null (Feb 3, 2020)

AlexGroans said:


> >All of this on a single hard disk
> I can't believe that it's only 1.5tb, and that the poor thing isn't dead yet.


To clarify, it's just the static images, and that's backed up to cloud constantly. The scripts are on SSD and the database is on an entirely separated dedicated device.



TwinkLover6969 said:


> 2020 and not running everything on the CLOUD smh


the cloud don't like us much tbh fam, I can't put anything on it besides archives.


----------



## Null (Feb 3, 2020)

Spawn said:


> A bit over kill but what about something like this?


That looks good. What I'm probably going to do is RAID6 a 256GiB SSD and then RAID6 4TiB HDD, so I'm going to need 8 SATA slots. I'd also like more logical cores.

I'm very tempted to get two new servers, actually. One strictly for PHP and then an AMD Threadripper for reverse-proxying, and offer to reverse-proxy any contentious sites that'd like to have me handling their email complaints.


----------



## An Ghost (Feb 3, 2020)

3119967d0c said:


> I think we're all missing an opportunity here.
> 
> Do you not want a Void bedspread? Perhaps some cardboard box covers with the KF logo for your side-tables?


KF rainbow thigh highs I mean uh programming socks. And thongs. Kiwi thongs.


----------



## WeWuzFinns (Feb 3, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...


Who wouldn't want to drink their tard cum from kiwifarms's very own cum chalice


----------



## drunk "Jedi master" (Feb 3, 2020)

ya epyc 7002 and 7001 both are good 7001 is the older of the two you can also get some serious core counts on servers with up to dual 64 core cpus so 128 cores in a single 2U system with 256 threads and up to I think it was 4tb of ram ... if you want to have a ram disk for all the things and another mortgage on your house if you have one.


----------



## Maurice Caine (Feb 3, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...



Where we're going we don't need servers.


----------



## Piss Clam (Feb 3, 2020)

If I ever see one of you wearing a KiwiFarms t-shirt. I'll hug the shit out of you and whisper "calm down, faggot".


----------



## Pepito The Cat (Feb 3, 2020)

AlexGroans said:


> I can't believe that it's only 1.5tb, and that the poor thing isn't dead yet.


I'm thinking a 2 Tb WD purple. A green series would have died a long time ago.


----------



## MarinaAnnDanzig (Feb 3, 2020)

Spoiler: Merch shill: T-Shirt idea 



Any t-shirt bearing "Support The Farms" on one side and logo on the back would sell well. Alternatively a more specific shirt here in Canada reading " I Support The Farms" or "Oil Country Loves the Farms" and something resembling a negative take on how Trudeau would view the farms ie Trudeau's face Crossed out in the back simply stating "Didn't you know? Trudeau Hates Farmers/Oil Country"  would sell even more. Since the hatred for Trudeau runs so deep out here in Western Canada farmers (no relief after early winter has crippled harvest yields) and people who work in the oil field (Still stalling on the Keystone pipeline which will bring 4000* temporary and up to 20,000* long term jobs to western Canadians) will latch on to any thing remotely showing a negative take on Justina. They would support this place blindly without knowing or caring about the real source of the "fundraiser". I'd gladly prepay for ten or so just for my employees and a couple friends as gifts. Cryptocurrencies though are quite confusing to me tbh (Shit I am a fucking boomer) and wouldn't even know where to begin (Can you search Bing for something other then porn?) *These numbers have been highly debated but you get the jist. Western Canadians fn hate Trudeau] Merch shill: T-Shirt idea


I have a very small 500 member forum that has only been online for just over three months. It's non vpn/proxy registration only. I run no ads and don't throw the name around so it's all been word of mouth. My members are mostly old PWL members, Reddit rejects and Chan boomers. They are loyal as fuck though and if it helps I could put a link up and direct some people this way.  If this interests you, or if you can think of any other way I can help please hmu. Regards Luke


----------



## Cosmos (Feb 3, 2020)

I’d love to see another merch run! You should bring back some of the old stuff, too, I remember there being really cool designs I wasn’t able to buy at the time.


----------



## Mikoyan (Feb 3, 2020)

Null said:


> That looks good. What I'm probably going to do is RAID6 a 256GiB SSD and then RAID6 4TiB HDD, so I'm going to need 8 SATA slots. I'd also like more logical cores.
> 
> I'm very tempted to get two new servers, actually. One strictly for PHP and then an AMD Threadripper for reverse-proxying, and offer to reverse-proxy any contentious sites that'd like to have me handling their email complaints.



Speaking from significant experience running databases on arrays, be a little careful since 6 will significantly limit your writing performance.

Of course, going SSD will make up for that a bit...


----------



## CreamyHerman’s (Feb 3, 2020)

You should just get a epyc processor, should be enough for 50 cycles


----------



## Null (Feb 3, 2020)

Broadwing said:


> Speaking from significant experience running databases on arrays, be a little careful since 6 will significantly limit your writing performance.


As I said, the database is on its own SSD hardware already. It got way too big for just the one.



xr95 said:


> You should just get a epyc processor, should be enough for 50 cycles


How would 64x2.0GHz be faster than 8x4.1GHz or something? Would imagine faster/fewer is better for PHP. Like I said, maybe an epyc for an NGINX reverse-proxy shared with sites like ED.


----------



## Leibowitz (Feb 3, 2020)

Null said:


> That looks good. What I'm probably going to do is RAID6 a 256GiB SSD and then RAID6 4TiB HDD, so I'm going to need 8 SATA slots. I'd also like more logical cores.
> 
> I'm very tempted to get two new servers, actually. One strictly for PHP and then an AMD Threadripper for reverse-proxying, and offer to reverse-proxy any contentious sites that'd like to have me handling their email complaints.


Are there any blade servers that use Threadripper? It's more of a prosumer product.


----------



## HOMO FOR LIFE (Feb 3, 2020)

If we are doing a Merc run I want a shirt that says " I was right "


----------



## Love Machine (Feb 3, 2020)

Very excited for the stickers, they better be good fucking quality null.


----------



## Smug Cat (Feb 3, 2020)

Cosmos said:


> I’d love to see another merch run! You should bring back some of the old stuff, too, I remember there being really cool designs I wasn’t able to buy at the time.


I'm still mad I missed the void tbh


----------



## one_time_user (Feb 3, 2020)

I like magnets rather than stickers / in addition to stickers.


----------



## Consenticles (Feb 3, 2020)

So I have this 3d printer I've been playing around with. If you want to, I can help with the fundraiser by providing a small knick knack people can purchase from your merch shop. My ability to fulfill orders would be limited to the capabilities of my printer (as in, how fast I can print).

If I get your permission, @Null, I can design a small kiwi farms model, which you can sell and I can ship via the US (though I would need to be reimbursed for shipping, since that can get expensive).

Something like this

Kiwi Bird found on #Thingiverse https://www.thingiverse.com/thing:2122334

Or maybe a KF logo in 3d. I have discord or I can discuss details in PM. Whichever works. Or ignore this if you aren't interested. I just really like this site and printing filament is pretty damn cheap.

Examples of shit I've made


----------



## j666 (Feb 3, 2020)

Consenticles said:


> So I have this 3d printer I've been playing around with. If you want to, I can help with the fundraiser by providing a small knick knack people can purchase from your merch shop. My ability to fulfill orders would be limited to the capabilities of my printer (as in, how fast I can print).
> 
> If I get your permission, @Null, I can design a small kiwi farms model, which you can sell and I can ship via the US (though I would need to be reimbursed for shipping, since that can get expensive).
> 
> ...



I would buy the shit out of little KF figures


----------



## Love Machine (Feb 3, 2020)

Consenticles said:


> So I have this 3d printer I've been playing around with. If you want to, I can help with the fundraiser by providing a small knick knack people can purchase from your merch shop. My ability to fulfill orders would be limited to the capabilities of my printer (as in, how fast I can print).
> 
> If I get your permission, @Null, I can design a small kiwi farms model, which you can sell and I can ship via the US (though I would need to be reimbursed for shipping, since that can get expensive).
> 
> ...





j666 said:


> I would buy the shit out of little KF figures



We can create Dungeon Dice Monsters but the monsters are lolcow figures. Can anyone defeat my 3 Blue/Green-eyed Stripped Dragons?


----------



## soft kitty (Feb 3, 2020)

3119967d0c said:


> I think we're all missing an opportunity here.
> 
> Do you not want a Void bedspread? Perhaps some cardboard box covers with the KF logo for your side-tables?


Hell yeah.


j666 said:


> I would buy the shit out of little KF figures


1000% yes.


----------



## Haramburger (Feb 3, 2020)

DanteAlighieri said:


> I await stickers as I need a swag way to dogwhistle my shitposter status!


I want stickers to put on the cars of my liberal friends so their LGBT allies riot.


----------



## Love Machine (Feb 3, 2020)

Tagging Yaniv's street signs with the stickers to establish our dominance is the only true way.


----------



## the seeker (Feb 3, 2020)

Can you make a dog collar with spikes for merchandising? this is for my dog, not for my secret furry kink


----------



## GaryGrey (Feb 3, 2020)

Maybe act fast and do a limited run of the KF logo in its hazmat suit.  There has to be some Chinese version of Teespring but for those cloth face masks, get a run of those with the KF logo.


----------



## karz (Feb 3, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...



Just as a suggestion, you could research into Single Layer Cell (SLC) Solid Storage drives.

SLC is "generally used in commercial and industrial applications and embedded systems that require high performance and long-term reliability.
SLC uses a high grade of flash technology that provides good performance and endurance, but the tradeoff is its high price.
SLC flash is typically more than twice the price of MultiLayer Cell (MLC) flash [memory]." (Source)

Getting a 10TiB SLC Drive would not only solve storage issues for many years to come, it would also last for many years (expected lifespan of these drives is 20-30 years under moderate load).

It's the most expensive solution, but will have the least headache (in theory). But you sail this boat, so you determine a Final Solution for The Error 2020 Question.


----------



## HOMO FOR LIFE (Feb 3, 2020)

j666 said:


> I would buy the shit out of little KF figures


I will have one for every place I frequent. My room. My basement. My attic. My bathroom. My car. My second basement. My cage. My closet.


----------



## Virgo (Feb 3, 2020)

Shirts, mugs, stickers, figurines, maybe even a canvas tote? I will buy if any of these pop up.


----------



## Troonos (Feb 3, 2020)

I can't wait for more T-shirt designs. Take my money!


----------



## Situation Type Deal Gorl (Feb 3, 2020)

Null said:


> That looks good. What I'm probably going to do is RAID6 a 256GiB SSD and then RAID6 4TiB HDD, so I'm going to need 8 SATA slots. I'd also like more logical cores.
> 
> I'm very tempted to get two new servers, actually. One strictly for PHP and then an AMD Threadripper for reverse-proxying, and offer to reverse-proxy any contentious sites that'd like to have me handling their email complaints.



I'd definitely get two new servers to split resource handling (then again, that's just spending your money, LOL). Do you use load balancers on the public-facing side, or just rely on CF to do it job? I know CF can be a real dick relaying traffic forward for things they may not have cached, resulting in a zillion requests bouncing in (usually from bots, scrapers, and fuckers versus legit traffic). It's enough to make you crazy.


----------



## one_time_user (Feb 3, 2020)

magnets (the bendy kind) and totes and figurines. plus an option to ridiculously over pay for... a stamp or something. I will buy all of these


----------



## goodbudweiser (Feb 3, 2020)

Consenticles said:


> I have discord



Kiwi Bird on Thingiverse is cute :3
MATI has a discord _*but*_ I'm banned and can't re-figure out the chat pop up so RIP.


----------



## Jeb-sama (Feb 4, 2020)

Null said:


> After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
> We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
> 
> 4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an _entire server_ dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
> ...



if you delete my awful avatar, the website will run seamlessly


----------



## Ashley Lynne Coulter (Feb 4, 2020)

I, for one, look forward to advertising my internet gay Nazi hive status. I'm going to need a whole roll of stickers for this upcoming election cycle.


----------



## JimmerSnail (Feb 4, 2020)

Let's make a lemonade stand, but without the lemonade. Then we call everyone a racist bigot and a kiwiphobe.


----------



## NoFeline (Feb 4, 2020)

I can't wait to vandalize high voltage boxes and traffic signs with the Kiwifarms decals that will definitely be released, definitely.

By the way, for anyone wondering about shirt quality I got one of the Nool shirts last time around and it's excellent. Very soft, good print makes a great cumrag, survives my awful washing machine practices.


----------



## goodbudweiser (Feb 4, 2020)

JimmerSnail said:


> kiwiphobe


unless you want to say farmphobic? But both sound silly.


----------



## CockPockets (Feb 4, 2020)

Will you restore some of the taxing website features you removed after upgrading the server?


----------

