Error 2020: Too Slow

Status
Not open for further replies.
Null said:
When done locally, the entire request takes well under 200ms. That is with PHP proccing. There is something specifically, and intermittently, wrong about outside requests. What's weird is this latency isn't found outside of this particular site. Other, small sites I run on the same hardware have no mysterious overhead. I have to be doing something wrong but I can't find it. There's just a flat 150ms lag on the initial connection, another 150ms lag on the SSL handshake, and then another 150ms lag before the page loads. Sometimes. The other half of the time it loads instantly.
Click to expand...
Have you tried emailing the FBI and politely asking them if they can optimize the datacenter that they've got spying on us?

only partially joking....
 
Null said:
4.53 3.98 3.92, not high.

No fucking clue what the issue is. It's depressing.
Click to expand...

If CPU is 8 core, this may be 4 physical/4 logical - in which case 4.5 would be a bit high. Or it might be 8physical, in which it would be as you said "not high".

If not that, perhaps load on a router above is high.
 
Last edited:
  • Like
Reactions: Ningen
now watch this drive 911 said:
If CPU is 8 core, this may be 4 physical/4 logical - in which case 4.5 would be a bit high. Or it might be 8t, in which it would be as you said "not high".
Click to expand...
1580681366464.png

ark.intel.com

Intel® Xeon® Processor E3-1240 v3 (8M Cache, 3.40 GHz) Product Specifications

Intel® Xeon® Processor E3-1240 v3 (8M Cache, 3.40 GHz) quick reference guide including specifications, features, pricing, compatibility, design documentation, ordering codes, spec codes and more.
ark.intel.com ark.intel.com

Performance
# of Cores: 4
# of Threads: 8
 
  • Informative
Reactions: Operator of the Wired and Ningen
Null said:
# of Cores: 4
# of Threads: 8
Click to expand...

I suspect this is significant to your scenario, but this is at the end of my experience. All I know for sure is that you have only 4 physical execution units.

Here's a cool tip for you or anyone following along at home:

Htop -> F2 -> display options, turn this on:
1580681956054.png

Legend is in F1. You can also turn on Kernel threads from Display Options. It also makes htop more colorful, which is good for hacking.

Good luck!
 
  • Informative
Reactions: Ningen and greengrilledcheese
  • Like
  • Informative
Reactions: Ningen and Vecr
After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.

4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to milk the server for everything it's worth and now we're at a limit. I have an entire server dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.

Also, CrunkLord420 runs the Matrix/Riot and Pleroma instances on Server 4, and that server itself is running out of disk space.

Consider donating via crypto if that's an option for you.

I'm probably going to do a merch run to finance this expansion soon. Will likely try selling stickers since it was highly requested last time. Keep your eyes out for that.
 
  • Feels
  • Informative
  • Like
Reactions: Knyttet, Buer, Ashley Lynne Coulter and 153 others
Null said:
After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.

4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an entire server dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.

Also, CrunkLord420 runs the Matrix/Riot and Pleroma instances on Server 4, and that server itself is running out of disk space.

Consider donating via crypto if that's an option for you.

I'm probably going to do a merch run to finance this expansion soon. Will likely try selling stickers since it was highly requested last time. Keep your eyes out for that.
Click to expand...
Will you ever re-use past designs for the merch run? Not specifically this one, but any?
 
  • Like
Reactions: CashMoney!, Nguyen Van Phuoc, Cheesetoast and 4 others
Null said:
After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.

4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an entire server dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.

Also, CrunkLord420 runs the Matrix/Riot and Pleroma instances on Server 4, and that server itself is running out of disk space.

Consider donating via crypto if that's an option for you.

I'm probably going to do a merch run to finance this expansion soon. Will likely try selling stickers since it was highly requested last time. Keep your eyes out for that.
Click to expand...
How much do you anticipate it costing?
 
  • Like
Reactions: Cheesetoast
Don't forget to throw more harddrives at this so you get more IOPS out of your system, also as a SSD is nearing fullness the way it works will make it slower. Also tons of RAM will help indirectly with your access times on predictable things because of caching by the OS even if/when tasks don't *directly* use it. No such thing as free/unused memory in a modern OS. (to a degree)

Also check if your choice of filesystem is the most efficient one for this workload. There's also lots to tweak there too. Won't solve your upgrade problem but might shove the next upgrade a bit out into the future.
 
  • Like
Reactions: Kosher Salt and Marvin
Status
Not open for further replies.
Back