Build a Farm: Hardware considerations

[Null]

I've tried every optimization. Here's how fucked it is.

I've spent all morning implementing an X-Accel-Redirect header in XenForo. I learned about this during Infinity Next.

infinity-next/infinity-next

Infinity Next is an imageboard suite utilizing the Laravel framework. - infinity-next/infinity-next

github.com

Basically, if you use PHP for sending images, you're going to have a bad time. Literally the exact fucking opposite of web scale. For some reason, XenForo does this. It's become apparent XenForo is not optimized for large sites and especially not for large sites that want to cache content.

So I set to work getting this feature in XF2 and by the end of it, proxy.php is so fast that the new iowait bottleneck is on nginx and the entire server is down, effectively DoS'd by early morning traffic.

fuck this, I'm going to buy a server tomorrow and hope the merch run finances it, we can't wait anymore

 

[DNA_JACKED]

I've tried every optimization. Here's how fucked it is.

I've spent all morning implementing an X-Accel-Redirect header in XenForo. I learned about this during Infinity Next.

infinity-next/infinity-next

Infinity Next is an imageboard suite utilizing the Laravel framework. - infinity-next/infinity-next

github.com

Basically, if you use PHP for sending images, you're going to have a bad time. Literally the exact fucking opposite of web scale. For some reason, XenForo does this. It's become apparent XenForo is not optimized for large sites and especially not for large sites that want to cache content.

So I set to work getting this feature in XF2 and by the end of it, proxy.php is so fast that the new iowait bottleneck is on nginx and the entire server is down, effectively DoS'd by early morning traffic.

fuck this, I'm going to buy a server tomorrow and hope the merch run finances it, we can't wait anymore

I still think a Xeon E-2288G is going to be your best bet for a new server CPU, with a 3.7 base and 5.0 turbo. There is a 4.0 base xeon 6 core option as well, but that about as fast you are going to get single core performance in a server today.

 

[Null]

I still think a Xeon E-2288G is going to be your best bet for a new server CPU, with a 3.7 base and 5.0 turbo. There is a 4.0 base xeon 6 core option as well, but that about as fast you are going to get single core performance in a server today.

There are some 1U dual sockets that are available.

 

[DNA_JACKED]

There are some 1U dual sockets that are available.

Are you thinking of going with a single dual socket server instead of two separate servers?

 

[Null]

Are you thinking of going with a single dual socket server instead of two separate servers?

Don't mobos with 2 sockets instance a single OS and stack?

 

[Slav Power]

If there was some kind of forum software that would be much more optimized and SEO friendly, how hard would it be to port the entire forum to it, if such thing would be even possible in the first place?

 

[DNA_JACKED]

Don't mobos with 2 sockets instance a single OS and stack?

Yes they do. I was running off your comment you wanted high single threaded performance for PHP, and all the dual socket servers I could find new for under $5K have 8 core CPUs that top out at only 4-4.2 GHz boost, with lower base clocks, compared to the 5GHz boost features on Xeons in single socket servers.

Of course, a single dual socket server could host twin VMs, one for each of your current servers. There are plenty of free VM applications that would work for hosting, and it would make system restore easier then running on the metal does.

I'd still recommend a R240 from dell, with a Xeon E-2288G, which will give you the single threaded performance you want for PHP, possibly buy two R240s for less then you can get a single dual socket server, then use your current box for hosting VM instances that you could bounce between the two new servers for better uptime, specially in case of hardware failure or sudden spike in load.

 

[Null]

Yes they do. I was running off your comment you wanted high single threaded performance for PHP, and all the dual socket servers I could find new for under $5K have 8 core CPUs that top out at only 4-4.2 GHz boost, with lower base clocks, compared to the 5GHz boost features on Xeons in single socket servers.

I can't find any processors over 4GHz that aren't $10k. What website are you using?

I'm most mystified by what HD setup to get. I don't really know the difference between NVMe / SAS / SATA. I thought I wanted to get 2 RAID10s, one of 4x256GB SSD and another of 4x2TB HDD with HW RAID, but I have no fucking clue anymore.

 

[DNA_JACKED]

I can't find any processors over 4GHz that aren't $10k. What website are you using?

I'm most mystified by what HD setup to get. I don't really know the difference between NVMe / SAS / SATA. I thought I wanted to get 2 RAID10s, one of 4x256GB SSD and another of 4x2TB HDD with HW RAID, but I have no fucking clue anymore.

None of Intel's xeon processors come with a base clock over 4.0 GHz. They all rely on boost to hit those higher clocks. Server configuration sites dont do a good job of explaining what boost clocks look like, I've been looking up CPU boost stats from intel's website individually. Xeons with metal names (xeon bronze/silver/gold) are more aimed at parallel tasks, and favor cache over clock rates. Stick to numbered xeons (Xeon E-2288G for instance) for higher per core performance.

If you are not bandwidth limited on the storage space, NVMe will be an expensive waste of time. The big difference is speed. SAS/SATA use the same protocol, you can use SATA drive on a SAS port (but not the other way around). SAS/SATA is limited to about 550MB/s in real world use per drive, NVMe can hit as high as PCIe allows, PCIe NVMe hits around 3-3.4GB/s max, where PCIe NVMe hits right now around 6.5GB/s.

I would recommend you stick with HDDs for storage, given how large the site is and how expensive SSDs are at larger capacities. A 2TB SSD will run around $200-250 for a consumer model, figure around $100 for a 1TB consumer model, higher prices for enterprise drives.

Most servers dont have native NVMe drive support. The name of the game today is using NAS full of HDDs that host VMs that run on host servers outfitted with a single SSD to hold the host OS. If we are running on a single HDD now for hosting data storage, even a RAID 5 of 4 HDDs would be a massive boost to performance. If you felt like spending soem cash, 4x2TB SATA III SSDs would provide upwards of 1.5GB/s performance in RAID 5, but the constant read/writes will wear them out quicker. I have no idea what your daily read/write rates are on the current machines.

 

[Null]

Xeons with metal names (xeon bronze/silver/gold) are more aimed at parallel tasks, and favor cache over clock rates. Stick to numbered xeons (Xeon E-2288G for instance) for higher per core performance.

God, is that why the processors are so expensive and suck? Thanks for that

 

[DNA_JACKED]

God, is that why the processors are so expensive and suck? Thanks for that

Yes, precious metal Xeons come in higher core count and higher cache varients, along with a quad channel memory controller. Precious metal Xeons are based on the same tech that goes into Intel's HDET line.

Their numbered Xeons are amped up versions of the arch/chipset tech used in their consumer i3/5/7 line. These servers are even compatible with consumer chips, the dell x240 can be had with celeron, core i3, or xeon chips. These have dual channel memory controllers and smaller cache setups meant for quicker access.

For your purposes, stick with numbered Xeons.

 

[Null]

For your purposes, stick with numbered Xeons.

Do Xeons rule out all dual socket mobos? i.e. I can't have two E-2288G?

 

[DNA_JACKED]

Do Xeons rule out all dual socket mobos? i.e. I can't have two E-2288G?

Yes. The consumer based Xeon chipsets are single socket only. If there is a dual socket version, I havent seen it, it would require twin chipsets and custom firmware that Intel would not support.

Also, here is a breakdown of numbered xeon clock speeds, turbo speeds, and turbo bins per cores used:

Intel Launches Xeon-E 2200 Processors For Servers

Mores cores and higher boost clock speeds!

www.tomshardware.com

 

[Coffee Anon]

Monk determined that the CPU is not the bottleneck, but the HDD seemed to be. If it's an issue with need for quick random access (i.e. requiring low-latency, not necessarily high-throughput), then a high capacity SSD might be the only solution required.
The Samsung 860 PRO has a good review:

https://www.storagereview.com/samsung_860_pro_ssd_review

(4 TB model is $930 on Amazon)

https://www.amazon.com/Samsung-Inch-SATA-Internal-MZ-76P4T0BW/dp/B0786ZQ1PJ/ref=sr_1_3?keywords=samsung+4tb&qid=1581008211&sr=8-3

(2 TB model is $490)

https://www.amazon.com/Samsung-Inch-SATA-Internal-MZ-76P4T0BW/dp/B07879KC15/ref=sr_1_3?keywords=samsung%2B4tb&qid=1581008211&sr=8-3&th=1

The EVO model has a comparitively bad review, even though it's cheaper, so I wouldn't go with that.

Samsung 860 EVO SSD Review - StorageReview.com

Along with the Samsung 860 PRO that was announced at CES 2018 for high-end…

www.storagereview.com

To the point where I wonder if they got a faulty drive.

The PRO also comes with a limited 5 year warranty which is nice.

Note that this is still considered a CONSOOMER drive, not ENTERPRISE™, but those high end SAS SSDs can be pricey if you want top tier performance.

Toshiba PX04P NVMe SSD Review (2.5") - StorageReview.com

The Toshiba PX04P is a series of NVMe SSDs (Non-Volatile Memory Express Solid State…

www.storagereview.com

($3540 for 1.6 TB! Hard to believe. I wonder if this is some sort of scammy reseller getting 300% markup?)

Amazon.com: Toshiba PX04SMB160 Phoenix-M3 ME 1.6TB, SAS 12Gb/s MLC, 2.5", 15mm, 19nm 10DWPD: Computers & Accessories

Amazon.com: Toshiba PX04SMB160 Phoenix-M3 ME 1.6TB, SAS 12Gb/s MLC, 2.5", 15mm, 19nm 10DWPD: Computers & Accessories

www.amazon.com

 

[DNA_JACKED]

Monk determined that the CPU is not the bottleneck, but the HDD seemed to be. If it's an issue with need for quick random access (i.e. requiring low-latency, not necessarily high-throughput), then a high capacity SSD might be the only solution required.
The Samsung 860 PRO has a good review:

https://www.storagereview.com/samsung_860_pro_ssd_review

(4 TB model is $930 on Amazon)

https://www.amazon.com/Samsung-Inch-SATA-Internal-MZ-76P4T0BW/dp/B0786ZQ1PJ/ref=sr_1_3?keywords=samsung+4tb&qid=1581008211&sr=8-3

(2 TB model is $490)

https://www.amazon.com/Samsung-Inch-SATA-Internal-MZ-76P4T0BW/dp/B07879KC15/ref=sr_1_3?keywords=samsung%2B4tb&qid=1581008211&sr=8-3&th=1

The EVO model has a comparitively bad review, even though it's cheaper, so I wouldn't go with that.

Samsung 860 EVO SSD Review - StorageReview.com

Along with the Samsung 860 PRO that was announced at CES 2018 for high-end…

www.storagereview.com

To the point where I wonder if they got a faulty drive.

The PRO also comes with a limited 5 year warranty which is nice.

Note that this is still considered a CONSOOMER drive, not ENTERPRISE™, but those high end SAS SSDs can be pricey if you want top tier performance.

Toshiba PX04P NVMe SSD Review (2.5") - StorageReview.com

The Toshiba PX04P is a series of NVMe SSDs (Non-Volatile Memory Express Solid State…

www.storagereview.com

($3540 for 1.6 TB! Hard to believe. I wonder if this is some sort of scammy reseller getting 300% markup?)

Amazon.com: Toshiba PX04SMB160 Phoenix-M3 ME 1.6TB, SAS 12Gb/s MLC, 2.5", 15mm, 19nm 10DWPD: Computers & Accessories

Amazon.com: Toshiba PX04SMB160 Phoenix-M3 ME 1.6TB, SAS 12Gb/s MLC, 2.5", 15mm, 19nm 10DWPD: Computers & Accessories

www.amazon.com

How much data do these drives write monthly? If the random read is way too low on a HDD, but the total amount written is also comparatively low, the crucial MX 500 is half the price and also features a 5 year warranty (but has 1/3rd the TBW rating, for what thats worth, so if we write massive amounts of data the 860 pro would likely be better).

MX 500 2TB $209: https://www.amazon.com/Crucial-MX500-NAND-SATA-Internal/dp/B078C515QL/ref=sxin_2_ac_d_pm?ac_md=4-2-QWJvdmUgJDIwMA==-ac_d_pm&cv_ct_cx=crucial+MX500&keywords=crucial+MX500&pd_rd_i=B078C515QL&pd_rd_r=36a47d67-4feb-45b1-abd5-e70a26dd22f4&pd_rd_w=Zr0YA&pd_rd_wg=q7qXT&pf_rd_p=0e223c60-bcf8-4663-98f3-da892fbd4372&pf_rd_r=B36NPRDKDZRH757Z33MW&psc=1&qid=1581012098&sr=1-3-22d05c05-1231-4126-b7c4-3e7a9c0027d0

 

[Null]

Discuss

Thinkmate® RAX-1208-SH 1U Chassis - 8x Hot-Swap 2.5" SATA/SAS3 - 600W Single Power
Intel® C246 Chipset - 6x SATA3 - 2x M.2 - Dual Intel® 1-Gigabit Ethernet (RJ45)
Six-Core Intel® Xeon® Processor E-2286G 4.0GHz 12MB Cache (95W)
2 x 16GB PC4-21300 2666MHz DDR4 ECC UDIMM

LSI MegaRAID 9341-8i SAS 12Gb/s PCIe 3.0 8-Port Controller
4 x 240GB Intel® SSD D3-S4510 Series 2.5" SATA 6.0Gb/s Solid State Drive
4 x 1.8TB SAS3 12.0Gb/s 10000RPM - 2.5" - Seagate Exos 10E2400 Series (512e/4Kn)

CONFIGURED PRICE:
$4,160.00

($377/mo)

 

[Looney Troons]

Are you building this from Thinkmate’s site, or elsewhere? I don’t know... it’s solid enough, I just feel as if it’s egregiously overpriced, even at that monthly price point.

 

[Null]

Are you building this from Thinkmate’s site, or elsewhere? I don’t know... it’s solid enough, I just feel as if it’s egregiously overpriced, even at that monthly price point.

Thinkmate. What would you suggest? I checked ebay for that processor and didn't find much.

 

[DNA_JACKED]

Discuss

Thinkmate® RAX-1208-SH 1U Chassis - 8x Hot-Swap 2.5" SATA/SAS3 - 600W Single Power
Intel® C246 Chipset - 6x SATA3 - 2x M.2 - Dual Intel® 1-Gigabit Ethernet (RJ45)
Six-Core Intel® Xeon® Processor E-2286G 4.0GHz 12MB Cache (95W)
2 x 16GB PC4-21300 2666MHz DDR4 ECC UDIMM

LSI MegaRAID 9341-8i SAS 12Gb/s PCIe 3.0 8-Port Controller
4 x 240GB Intel® SSD D3-S4510 Series 2.5" SATA 6.0Gb/s Solid State Drive
4 x 1.8TB SAS3 12.0Gb/s 10000RPM - 2.5" - Seagate Exos 10E2400 Series (512e/4Kn)

View attachment 1133069

CONFIGURED PRICE:
$4,160.00

($377/mo)

Looks like a good setup.

Are you building this from Thinkmate’s site, or elsewhere? I don’t know... it’s solid enough, I just feel as if it’s egregiously overpriced, even at that monthly price point.

You have to figure the server itself, from someone like Dell, would be 1100-1200$, with only 8GB of RAM and 1 HDD. This thing has 32GB, quad SSDs and quad 10K HDDs with a hardware SAS RAID controller. That ain't cheap.

Thinkmate. What would you suggest? I checked ebay for that processor and didn't find much.

I'd be amazed if you could find them at all, given these chips came out 3 months ago.

 

[soft kitty]

Looks good.