Error 2020: Too Slow

Status
Not open for further replies.
Null said:
Sometimes. The other half of the time it loads instantly.
Click to expand...
Null said:
tell me if this sounds dumb, because I've tried rolling with fewer of each and it caused serious issues.
Click to expand...
Sounds like some queue somewhere is exploding then the system's latency goes through the roof.
If you feel like watching a 20min out of a 40min technical talk, maybe this'll give you some ideas:
www.youtube.com

Zach Tellman - Everything Will Flow

Queues, it seems, are back in fashion. This is exciting, as queues let us explore problem spaces that were unreachable by pure, immutable primitives. But que...
www.youtube.com www.youtube.com
Is there any way you can profile the server and see what each thread is doing when it's hanging?

How to generate PHP Flamegraphs

In my quest to optimise the PHP sites I work on, I found that the Flame Graph repository contains a script to convert XDebug traces. Flame graphs, created by Brendan Gregg, visualize the most frequently hit code from profiled software. The x-axis represents the profiled stack traces, ordered...
daniellockyer.com
 
  • Like
Reactions: The Fool
It's not PHP, I've used black fire to check the scripts and solve those bottlenecks. If it is, it's the fpm and not the scripts
 
  • Informative
Reactions: BONE_Buddy
Null said:
after fucking around I'm deadass certain the issue is that:

1. we get 100 req/s now
2. nginx needs at least the same number of processes as php or else there's that mysterious overhead waiting for a nginx process
3. php needs at least 64 static processes otherwise you get sporadic 502 errors
4. the front-end is an 8 core server with 128 processes and the new bottleneck preventing coveted site fast status is waiting on a CPU, though our CPUs are mostly idle

tell me if this sounds dumb, because I've tried rolling with fewer of each and it caused serious issues.
Click to expand...
It sounds like it makes sense..

Assuming this was the case, is there anything straightforward you can do to reduce requests served? I mean, looking at a fresh page load the fontawesome fonts and stuff could presumably come from a CDN, but unless people's browsers are seriously misconfigured that's not going to account for many real requests.

Do you know if Cloudflare is being allowed to cache avatar images* and stuff, or could something be causing those requests to be passed on through? Probably a stupid question, I know.

* dumb question, they show up as cf-cache-status:HIT when I look
 
Null said:
Cloudflare caches a lot of stuff. If we ran without it as a CDN, our requests/sec would probably triple.
Click to expand...
Yeah, definitely not seeing Cloudflare cache misses for avatars, attachments, or proxied images on active threads. Are the requests that make it through that significant (from people browsing totally mothballed threads, or going through every page of some of the ancient ones)? I mean if I read the headers correctly they should be cached on CF for a month, whether it's possible to extend that I don't know.

How about the chat? I see it makes a request every five seconds when you're on the front page, even if you can't see it and don't have the chat message box selected- or 1 second if either of those are true. If you've got 50 people idling on the front page without viewing the chat that's only ten requests a second, but maybe it's more? It seems like at least the idle cycle could be slowed down without killing interactivity with the chat.
 
I wasn't talking just about php, you can profile the entire server to try and find the bottleneck, because it seems like you have one.
 
@Null Have you looked into the WAF rules in your Cloudflare config? Maybe too many rules or a regexp that backtracks too much like last year when they DoS'ed themselves?

From where I sit, I see the same >600ms TTFB on requests that can't be served from Cloudflare's cache regardless of it being a static resource or a CGI request. On the other hand, the same requests (static or not) on .nl are much faster. If only the requests from CF to you have that big TTFB then it's got to be something specific to CF, which makes me think of WAF.
 
  • Like
  • Informative
Reactions: BoJack Horseman and The Fool
@Null FYI, still seeing some 'server errors' while searching and others (@The Last Stand et al) are experiencing the same.

This is a little difficult to follow as it looks like XenForo gives a new ID to each 'unique search' a user does for a term. So if I search for a term that fails to return a result- I was able to achieve this by searching for some common names until a search failed- even if I try searching again or refreshing, it will fail again. I imagine this is done to keep consistent pagination/ordering for search results or something. I'm sure that probably times out in some period of time, but an impatient KiwiFarms user (like myself) will just repeatedly refresh and get mad.
1580530896007.png

Now, if I use incognito to make the same search a few seconds later, new search ID and it works fine.
1580530905395.png

It seems like a XenForo bug really- whatever's causing the search to fail should either allow the same search to be made again immediately, or drastically reduce the time for the search ID to expire and use a useful error message- but it seems a lot more prevalent than prior to the rebuild. Do you have a feel as to what might be causing it? Will the search index be building up as people do searches or something like that?
 
  • Informative
  • Like
Reactions: Angel Dust and Cryonic Haunted Bullets
If you really want to have a plan in as "financially stable, not having to think what'll happen the next 1, 2, 3, 5 years" I'd recommend you to stop wasting your time on this and look for financial stability elsewhere. The modern corporate internet doesn't want places like kiwifarms and this is only going to get worse. Eventually you'll run into a roadblock, either created by lawmakers pushed into action by the corporates or by pure technical circumstances of the corporate machinery, that you'll not be able to avoid or overcome, at least not without giving up on ever turning a profit. Not that I don't value what you're doing here and you went farther than most would have gone, I just don't think this site will provide you with a retirement plan.

Comedy reply: Just become a neet in one of the european countries with healthcare and financial support for the unemployed.
 
  • Optimistic
  • Feels
  • Dumb
Reactions: Cowboy Kim, The Pink Panther, Cryonic Haunted Bullets and 4 others
AmpleApricots said:
If you really want to have a plan in as "financially stable, not having to think what'll happen the next 1, 2, 3, 5 years" I'd recommend you to stop wasting your time on this and look for financial stability elsewhere. The modern corporate internet doesn't want places like kiwifarms and this is only going to get worse. Eventually you'll run into a roadblock, either created by lawmakers pushed into action by the corporates or by pure technical circumstances of the corporate machinery, that you'll not be able to avoid or overcome, at least not without giving up on ever turning a profit. Not that I don't value what you're doing here and you went farther than most would have gone, I just don't think this site will provide you with a retirement plan.

Comedy reply: Just become a neet in one of the european countries with healthcare and financial support for the unemployed.
Click to expand...

If null were a coward he would have stayed in the US and faithfully consumed like anyone else.
 
  • Feels
Reactions: The Pink Panther, In Memoriam and war has changed
Null said:
after fucking around I'm deadass certain the issue is that:

1. we get 100 req/s now
2. nginx needs at least the same number of processes as php or else there's that mysterious overhead waiting for a nginx process
3. php needs at least 64 static processes otherwise you get sporadic 502 errors
4. the front-end is an 8 core server with 128 processes and the new bottleneck preventing coveted site fast status is waiting on a CPU, though our CPUs are mostly idle

tell me if this sounds dumb, because I've tried rolling with fewer of each and it caused serious issues.
Click to expand...

How many nginx worker threads do you have? Are GGI/PGP threads not the same thing?

You could try `worker_processes auto;` in your nginx config, if you are not already doing that.
 
Status
Not open for further replies.
Back