Error 2020: Too Slow

[BOLDYSPICY!]

Oh, I was gonna tease @Null about breaking the site to fix an inconsequential thing that only bothers him because I haven't had issues with loading until jihad was declared, but then I realized that it's probably because I almost exclusively browse on mobile. I am the cancer that is killing the Bing score. Feels bad man.

. . .actually, why IS that? My smooth woman brain is bad at computer. Is it because the mobile version of the site is a bit more compact? Like an HTML version of a site that loads faster than one with all the bells & whistles?

EDIT: or wait, do I have it the wrong way around? I'm very tired.

My point is I don't have issues on mobile. But I also exclusively use Brave. Does that matter? What browser do other mobile users use?

 

[FuckedUp]

Try getting the site banned in more countries. Pretty sure the block page loads much faster.

 

[Angel Dust]

Oh, I was gonna tease @Null about breaking the site to fix an inconsequential thing that only bothers him because I haven't had issues with loading until jihad was declared, but then I realized that it's probably because I almost exclusively browse on mobile. I am the cancer that is killing the Bing score. Feels bad man.

. . .actually, why IS that? My smooth woman brain is bad at computer. Is it because the mobile version of the site is a bit more compact? Like an HTML version of a site that loads faster than one with all the bells & whistles?

EDIT: or wait, do I have it the wrong way around? I'm very tired.

My point is I don't have issues on mobile. But I also exclusively use Brave. Does that matter? What browser do other mobile users use?

I exclusively use brave on mobile as well. Smooth woman brain also not good at computer. Plus can browse on phone at work. The site on/off loads like anus for me. Sometimes it's super fast and other times I'm just starting at a screen going "load! Load god damn it!". Mostly though I'd say it's fair to average speed? I agree with maybe hiding the chat or putting it elsewhere. That seems to cause the most issues for me.

 

[HumanHive]

The fact @Null spares even a second to think about Bing is extremely troubling and I question the site's long term future.
But optimization is always nice.

 

[Tealeaf]

The fact @Null spares even a second to think about Bing is extremely troubling and I question the site's long term future.
But optimization is always nice.

None of the optimizations are Bing specific. Google prioritizes fast sites in its algorithm as well. Faster site also means happier farmers.

 

[BisquitDoughHandsMan]

I mostly just feel sorry for Null. If you like, Plan B After 30 can be living in the woods behind my house. You can befriend as many dogs as you want!

 

[Freya]

I mostly just feel sorry for Null. If you like, Plan B After 30 can be living in the woods behind my house. You can befriend as many dogs as you want!

Well thats a hard offer to refuse

 

[BoJack Horseman]

You all are forgetting about one detail, Bing uses Oath which is what other engines such as DuckDuckGo use so I believe optimizing for Bing will have a positive effect on the other non-Google engines as well. Correct me if I'm wrong tho.

 

[now watch this drive 911]

Seems to be largely TTFB and Queuing: blame Cloudflare?
What do the response times look like when you're hitting the origin directly?
Have you managed to isolate the lag to network or server?

that 900ms wait on index tho lmao

 

[CaesarCzech]

force mobile view by default, you want normal view register. Cheat bing by basically building potemkin village.

 

[Null]

Seems to be largely TTFB and Queuing: blame Cloudflare?

I don't know. Here's a basic PHP file, so this is time to spin up FPM and get a response.

https://kiwifarms.net/ttfb.php - thru CF
https://kiwifarms.nl/ttfb.php - local reverse proxy

On .nl, TTFB is between 30ms and 600ms, usually on the higher side.

People keep saying "HAVE YOU TRIED TO RAISE THE FPM CHILD COUNT???" like I haven't been doing this shit my entire fucking life and I don't know the first thing about fucking tuning and I haven't been mitigating DDoS attacks and I haven't been doing shit professionally since I was fucking goddamn 19 years old

It's not even that I can look strictly at hardware because I don't just own my servers, I own my router. Could there be something about the router that is causing a huge latency to the server? Between the servers since I distribute the stack over multiple? Maybe the SDD is bad? Maybe there's some weird LXC configuration? I don't know, I don't fucking know. There's now 10,000 moving parts and if any of them take half a second the entire thing is completely fucked.

I don't think it's MySQL. Since my last batch of fixes to MySQL, I see between 17 to 150 idle connections to the database from FPM. Nothing is ever hanging. There's no max_client issues.

I also don't think it's PHP. If I run blackfire via CURL, I get results like this.

blackfire curl --samples 10 https://kiwifarms.net/threads/richard-c-kyanka-v-ashley-k-martin.65737/
Wall Time 154ms
I/O Wait 17.6ms
CPU Time 136ms
Memory 5.36MB
Network n/a n/a n/a
SQL 15.5ms 18rq

Excellent! 154ms. Perfect. Bam boom. I've looked at its traces and bottlenecks and sorted all those out, which is why the site should feel faster as soon as the initial waiting period is over. Once you hit that 600ms wait, the page is loaded.

I also have looked at the different local servers talking to each other and they work fine as well, so I don't think it's FPM to Redis / MySQL wait.

If I profile from my local debian installation I get similar results.

blackfire curl --samples 10 https://kiwifarms.net/threads/richard-c-kyanka-v-ashley-k-martin.65737/
Wall Time 162ms
I/O Wait 23ms
CPU Time 139ms
Memory 5.36MB
Network n/a n/a
SQL 17.5ms 18rq

I'm very certain it's not PHP, it's something weird. Why does it take 600ms to connect to the server? The ping is 173ms to the device and that's consistent (+3) with the datacenter from Europe.

 

[An Ghost]

I was wondering the same thing. Do we really benefit from going for constant growth?

We benefit from people being BTFO. There’s lots of cows who, when you search their name, you get KF as the first result. Beating out their YouTube channels and Facebook pages. In this day and age it’s so important to be in the top few if you want anybody to see your stuff. Bigger cows, we’re usually on the first page. But imagine being on the second page of google. Then it’s like what’s the point?

Also we can’t gut the guest user experience to up bing score. All the lurkers we already have get the short end of the stick, and new potential users say “wow this site looks like aids” and leave.

 

[Null]

What's further driving me crazy is this. This bypasses all of Cloudflare and is just hitting an HTML page, so it also eliminates any FPM slowness.

There's 2 kinds of loads.

The Chad Bing Score:

The Virgin 600ms:

#2 is obviously the Jared Leto Damaged that is causing problems and it happens every other load. It's not even a flat increase because of SSL. It takes 180ms to talk to the server, and another 180ms to do the SSL handshake, and then the actual response time for the site is about twice as slow as it should be (again, just to pull an HTML page and send it).

This also has to be an issue with just the KF server because other PHP websites hosted on the same fucking hardware has no issue what so ever.

 

[Swiss46]

Try downloading ram and uploading it to the site! It made my computer run Runescape faster, surely it will work for a website.

 

[now watch this drive 911]

Here is request for single jpg.

TCP establish connection (SYN, SYNACK, ACK) - 235ms (tcp stream 6 closed so 7 is actual (470ms total time))
TLS handshake (Client Hello, Server Hello) - 237ms

And the rest is history.

Latency is a serious thing in real time systems! This is why network file-systems (NFS) can be ass if used wrong.

These 600ms are - from our similarly distant locations - the baseline value for latency. Everything else that happens tacks onto that figure. If the front page is rendered dynamically each time it's requested, that's cpu+db+php time + download time for each asset + 600ms. Six parallel connections on x64 chrome, so there's room to move a lot of data.

There's 63 requests for this page, in chrome that happens across 6 parallel threads. Drop the total number of requests if possible. Turn the stickers into a sprite set? Serve JS and CSS in a minimized bundle. x64 Chrome is 6 threads but perhaps ARM+Cellular is less?

Note: I tested twice and got a TCP-RST in the first tcp stream both times. I think this is actually chrome doing something with hard-clearing the cache. Recommend you do your own tests to isolate and crosscheck my results.

 

[Otterly]

Tinfoil hat on, but are you sure the problem is coming from within your systems, and not from without?
Edited to add: search engines can delist you, and that’s obvious, but it wouldn’t surprise me if they had ways of just subtly down rating sites as well. These ‘nudge’ policies are widespread in health and politics.
Apologies if that’s ridiculous. I know f all about computers, but a bit about the dark arts of behaviour manipulation

 

[kiwifarmsfan1]

For site accessibility and convenience, typing in a keyword to access the A&H forum as an alternative to logging in would be nice. It can be simple like "kw" or something more creative. Only humans and spam bots will enter a word into a wordbox, search engines will just pass over the page. It was nice browsing incognito without having to type in a long pw. I wonder if bing will downgrade a site if a bunch of pages load instantly but just contain a wordbox?


now watch this drive 911's comment about the network stack and the filesystem should really be investigated, always go one level lower than you are comfortable with.

3. Completely redo how XenForo does CSS.
I've tried cutting up the CSS every which way but the site is simply not designed to load fast the first time. It just doesn't seem worth it but that alone is the majority of the Bing Score.

You might just have to think out of the box. I don't think of this as the best solution, but just to prove it can be done. From the worst option, webpages didn't always have css, to the second worse option, there are simple styles like those used in print page mode on forums that allowed you to print discussions. For the third worse but an improved option, themes for logged in people, logged out get a slightly to moderately uglier but fast version.

 

[Thomas Paine]

I'll be completely transparent, I have a barely-functioning grasp on networks. That said, Is there anyway that it might be a throttling issue from the server's physical data center or a provider along the network chain?

Time to crowdfund some server blades for Null.

 

[Null]

I'll be completely transparent, I have a barely-functioning grasp on networks. That said, Is there anyway that it might be a throttling issue from the server's physical data center or a provider along the network chain?

I do not believe the issue is the actual server.

curl -w "@curl_format.txt" -so /dev/null https://kiwifarms.net/
   time_namelookup: 0.000311
      time_connect: 0.000375
   time_appconnect: 0.006830
  time_pretransfer: 0.006910
     time_redirect: 0.000000
time_starttransfer: 0.104205
                   ———
        time_total: 0.104407

When done locally, the entire request takes well under 200ms. That is with PHP proccing. There is something specifically, and intermittently, wrong about outside requests. What's weird is this latency isn't found outside of this particular site. Other, small sites I run on the same hardware have no mysterious overhead. I have to be doing something wrong but I can't find it. There's just a flat 150ms lag on the initial connection, another 150ms lag on the SSL handshake, and then another 150ms lag before the page loads. Sometimes. The other half of the time it loads instantly.

 

[Null]

after fucking around I'm deadass certain the issue is that:

1. we get 100 req/s now
2. nginx needs at least the same number of processes as php or else there's that mysterious overhead waiting for a nginx process
3. php needs at least 64 static processes otherwise you get sporadic 502 errors
4. the front-end is an 8 core server with 128 processes and the new bottleneck preventing coveted site fast status is waiting on a CPU, though our CPUs are mostly idle

tell me if this sounds dumb, because I've tried rolling with fewer of each and it caused serious issues.