Home Server and Self Hosting General - Technological Self-Sufficiency

  • 🐕 I am attempting to get the site runnning as fast as possible. If you are experiencing slow page load times, please report it.
Major Discord

Major Discord

kiwifarms.net
Joined
Jun 3, 2021
Since /g/ is down and there's no /hsg/, I thought it can be a prime time to start a thread on here. I have seen a few other threads having tidbits of the information, but I thought centralizing everything can be beneficial.
I am by no means an expert on this topic, so any contributions will be greatly appreciated.

What is this thread for?
I would like this thread to cover discussions both about home servers/homelabs, as well as Virtual Private Servers (VPS). Both have their uses. I know that "self hosting" is sometimes strictly reserved for home servers, but I would like to include VPS as well.


Why should you self host?
Hosting your own services helps to both de-bloat your life and learn a few new skills in the process. A lot of services can be relatively easily hosted privately. You are also less-likely to be affected by some ToS changes or price hikes. I think hosting stuff yourself is often cheaper.

Resources
Where to host
  • VPS: I think the general advice is - if the service you want to host requires extensive internet access, use a VPS. This includes websites and email servers. Specifically email servers require reverse DNS (rDNS), which will not work for the majority on a homelab server, since people tend to have dyanmic IP addressess and not static IP ones. VPS are also useful as a kind of an off-site backup.
  • Home server: everything else can be hosted/experimented on a home server, such as: Network-Attached Storage (NAS), Plex/Jellyfin server for your media, automated backups, CCTV.

Useful Links
  • Hosting on a VPS
    • Landchad.net: Contains useful tutorials on how to host your own website, email server and other stuff. I think some information is outdated/wrong, but I don't remember what exactly - I think some stuff will just not work. And I also dislike that Luke (site's author) does everything as a root instead of making a su account.
    • Vultr, Hetzner, Netcup: Some companies from which you can get a VPS. I honestly don't have experience with many. I have seen another thread on the forum discussing VPS providers - I know obviously that KF itself can't be hosted with these providers. But most likely the majority reading this thread will just want to host normie stuff and I think the mentioned providers will do. I believe all 3 unblock ports for email servers on request (or have them unblocked to begin with). I would not recommend Njalla, especially since I find their entire selling point sort of moot. If the VPS provider is in Europe, you are covered by the GDPR anyway, and as far as I know your data will be always anonymized.
    • MX Toolbox: If you are hosting an email server or a website, this can be useful to check for possible problems and issues.
    • https://appmaildev.com/en/spf & https://www.learndmarc.com/ (provided by @Riftrunner ) : Another tool for checking SPF, DKIM and DMARC for email deliverability.
    • Cheap VPS Deals (provided by @dumbledore ): Normie providers, mostly. Some, such as Crunchbits or BuyVM, would be also suitable for "free speech" content. You should check the ToS/AUP of the provider to understand if they will unblock ports (notably 25) for the email server. At this moment BuyVM, Crunchbits and CloudFanatic (the latter according to @Jeff Q. Anime ) seem to unblock the email server ports on request.



Various guides/manuals/collections of information.

Courtesy of @Riftrunner
Virtualization
Whether you’re a homelabber or a Fortune 100 corporation, virtualization is an important concept to understand and an essential skill. While you could get away with running everything on bare metal hardware, virtualization gives you the ability to run as many virtual servers as your hardware can support.

The benefit of going virtual is that you can take point-in-time snapshots of the disk for easy rollbacks and backups, segregate services and applications, and manage resources such that no rogue application can suck up all your disk, CPU and RAM.

  • Containers
Whether Docker, LXC, or something exotic, these are a form of user-space isolation rather than full virtualization. Generally, the kernel is shared between container guests and the OS, which makes for much faster initialization and lower resource usage, but does restrict your OS choices (e.g. no Windows guests if you're on Linux)​
​
Another benefit of containers (mainly Docker) is it gives you access to a large library of pre-configured images with software like MariaDB pre-installed and ready to go.​
​
Some hypervisors (fancy word for virtual machine host software) support containers as well as full virtualization, but this is not the case for most.​

  • VMware ESXi (vSphere)
ESXi used to be the go-to option for nerds at home wanting to gain marketable skills. It was a darling in virtualization from small business all the way up to the largest corporations in the world, but thanks to its recent acquisition by Broadcom, it has lost a lot of favor.​

The strength of VMware ESXi is that it’s easy to use, performant, well supported and reliable. This is mostly still true today, but recent anti-consumer changes such as the removal of perpetual licensing, increased minimum core counts for licensing, locking down updates, and firing customers who reduce their licensing consumption, has meant its reputation has soured immensely.​
​
This isn’t to say that corporations aren’t using VMware anymore, they absolutely are, but smaller players (such as you) are not welcome to use it anymore. There is no longer a free version, and pirates can no longer receive updates online. At the moment there’s no DRM on offline updates, and the latest version still accepts perpetual keys which are easily obtainable with a Google search, but I would not recommend it for new homelabs.​

  • VMware Workstation
One of the few good things to come out of the Broadcom acquisition is that VMware Workstation is now completely free. Unlike ESXi which runs bare metal, this runs as an application on your Windows or Linux (read the DIY section before going down this path) PC and is an easy way to get started with virtualization.​

  • Microsoft Hyper-V
Similar to VMware Workstation, Hyper-V can run atop the Windows operating system and virtualize any x86 guest. Enterprise users will typically setup dedicated hosts using Windows Server, but you can run it on consumer versions of Windows so long as you’re using Pro, Enterprise or Education editions.​
​
In my opinion VMware Workstation is easier to use, but Hyper-V is a much more marketable skill than Workstation if you’re interested in developing skills for a career in IT.​

  • Proxmox Virtual Environment
In the wake of VMware completely shitting itself, many smaller enterprises have been migrating to Proxmox VE as it offers a compelling feature set at a fraction of the cost. For autists at home, the price is $0 as Proxmox is FOSS, you need only suffer a nag screen when you logon.​
​
For those familiar with VMware or Hyper-V, it’s a learning experience, and for total newbies it’ll take a while to get to grips with its somewhat convoluted UI. For those familiar with Debian Linux, you’ll feel right at home as PVE is built on Debian, though you do need to install it using their customized ISO.​
​
This is my recommended option for savvy autists who have moved beyond desktop virtualization such as VMware Workstation and are looking to virtualize on dedicated hardware. It has native support for Linux containers (LXC) so it’s a good option for those who are starved of resources or just prefer containers.​
​
There's also out of the box support for full backups and Proxmox Backup Server is available for free if you need incremental backups.​

  • XCP-ng
While I would wager Proxmox VE is the most popular vSphere alternative, another one I’ve seen mentioned a lot is XCP-ng. This is a community supported distribution of Citrix XenServer which bundles Xen Orchestra and uses Xen HVM as the hypervisor.​
​
I have not used it and cannot comment on its user friendliness, the main complaint I’ve seen is that it has a 2 TiB limitation for virtual disks at the moment. This isn’t likely to cause issues for homelabbers and they aware committed to fixing this.​
​
It's definitely worth considering if you're shopping around for a hypervisor.​

  • DIY
Fundamentally all the components you need to virtualize are available in most Linux distributions. KVM (Linux kernel virtualization module) + QEMU (device and hardware emulator) is all you technically need for a fully-fledged Linux-based virtualization machine.​
​
There are GUIs like virt-manager which provide an interface for KVM-based VMs and this is what I'd suggest to anyone wanting to do desktop virtualization on Linux as it's far better supported than running VMware Workstation.​

  • In Conclusion...
Don't ignore virtualization when building up your homelab. It's a critical component to a well functioning lab and an invaluable skill to demonstrate if you want a career in IT.​
​
You may ask why I didn't mention Oracle VirtualBox, and it's mainly because I don't understand why you'd use VBox when Workstation is free on Windows and you have KVM on Linux.​
​
I hope you enjoyed my essay and find it useful.​
 
Last edited:
  • Informative
  • Winner
  • Like
Reactions: raspberry mocha, Harvey Danger, Mr Macintosh and 15 others
You can get a list of cheap VPS deals on https://lowend-deals.xbit.win/
Normie providers, mostly. Some, such as Crunchbits or BuyVM, would be also suitable for "free speech" content.

If you are looking for Email hosting on your own domain: https://mxroute.com/ (currently used by KF) and https://namecrane.com/ are great. You won't have issues with deliverability (common on self-hosted email) or censorship of any kind.
 
Last edited:
  • Informative
  • Like
Reactions: gentoofag and Major Discord
Major Discord said:
Do most allow opening the required ports for an email server? As far as I've heard that is often a problem.
Click to expand...

I would check in TOS or AUP of the provider, its often stated there. Some of hosts on this list open ports on request. BuyVM and Crunchbits do for sure.
But I wouldn't expect amazing IP reputation from cheap VPS providers, for Email I would go for a managed solution(mxroute or namecrane) or a provider that isn't $2/3 per month, If you want usable deliverability of Emails to providers such as Outlook or Gmail.
 
Last edited:
  • Agree
Reactions: 888Flux
dumbledore said:
If you are looking for Email hosting on your own domain: https://mxroute.com/ (currently used by KF) and https://namecrane.com/ are great. You won't have issues with deliverability (common on self-hosted email) or censorship of any kind.
Click to expand...
The deliverability is an interesting issue. I have a simple postfix+dovecot setup. It works very well for most domains. The only problems that I am having are with Microsoft's consumer domains - @Outlook, @live etc. If I check the DMARC reports, it says that the DKIM check is failing. However, it doesn't fail for any other domain.
 
Major Discord said:
The deliverability is an interesting issue. I have a simple postfix+dovecot setup. It works very well for most domains. The only problems that I am having are with Microsoft's consumer domains - @Outlook, @live etc. If I check the DMARC reports, it says that the DKIM check is failing. However, it doesn't fail for any other domain.
Click to expand...

Microsoft is the notorious blacklister of Emails, and dealing with them is a nightmare. Unless you really want to put up a fight to have good deliverablity on your box, I would go for a managed solution and forget about it. There are trusted & small providers such as these two.

It is not related to your configuration, rather them blocking entire prefixes and networks affecting many people.
 
I've personally used BuyVM and CloudFanatic and they both unblocked port 25 on request. Outlook doesn't refuse my emails, but they do get sent to spam sometimes... *sigh*
 
  • Like
Reactions: Major Discord
dumbledore said:
Microsoft is the notorious blacklister of Emails, and dealing with them is a nightmare. Unless you really want to put up a fight to have good deliverablity on your box, I would go for a managed solution and forget about it. There are trusted & small providers such as these two.

It is not related to your configuration, rather them blocking entire prefixes and networks affecting many people.
Click to expand...
I did not know about the providers you mentioned. Thanks for the links! I will update the OP with the links later.

However, I am not sure how big of an issue the deliverability is.
In my personal case I am just hosting private stuff for myself. I think me and most people mostly need an email address to receive stuff rather than to send. And if I need to send, well, it works with most domains, and for the domains it doesn't work, people just whitelist me. If it is some super important email and there's a strict time constraint (which happens almost never), I would just send from my protonmail.

Mxroute seems too expensive for small private use tbh.
 
Major Discord said:
Mxroute seems too expensive for small private use tbh.
Click to expand...

https://namecrane.com/store/email-hosting-deals check these deals out, 100GB plan 3 years for $20
Or go for the lifetime, the person behind NameCrane is Francisco Dias, founder of BuyVM. He sold BuyVM last year and now operates NameCrane, which is ICANN accredited and will soon provide domain registration. There is a good chance that $69 will be a good long term investment.
 
Last edited:
  • Like
  • Thunk-Provoking
Reactions: 888Flux and Major Discord
dumbledore said:
https://namecrane.com/store/email-hosting-deals check these deals out, 100GB plan 3 years for $20
Click to expand...
Yeah this seems cool, especially the lifetime plans. I might go for it, if I have more issues with deliverability. Though at the moment it is literally just consumer Microsoft domains, so I don't bother. And I kind of want to keep my stuff centralized on a VPS, instead of having an additional provider for just email.
 
  • Like
Reactions: Palmer Bangs
Major Discord said:
Though at the moment it is literally just consumer Microsoft domains
Click to expand...
Microsoft has this concept of "warming up" IPs, where basically anything that's new will take at least 2 weeks to not just immediately hit junk folders. They'll also want you to sign up for SNDS but this is only useful if you actually own your IPs. Ditto JMRP.

They mention Return Path Certification but again that's not a thing for normies. You're really left in a bad spot if you want to self-host your external mail delivery. MXroute I can vouch for, I've e-mailed Gmail and Yahoo people with success using that service. Can confirm the e-mail to Yahoo was received based on the wailing that soon followed on that furfag's Twitter account.
Major Discord said:
If I check the DMARC reports, it says that the DKIM check is failing. However, it doesn't fail for any other domain.
Click to expand...
Give it a test using https://www.learndmarc.com/

It's maybe a bit overly verbose given it's intended for absolute beginners, but it does a good job of pointing out subtle problems like alignment issues. Craziest e-mail delivery issue I ever had related to a recipient domain failing ~20% of the time with nonsensical DMARC errors. Turns out that behind their loadbalancer they had one legacy MX alongside a bunch of modern ones and it could not validate DKIM keys of >1024 bits.

The previous e-mail admin decided to use strict SPF alignment without realizing that doesn't work when you use transactional mailers like AWS SES. So our SPF was constantly failing, but DKIM would validate ultimately receiving a DMARC pass most of the time, except with this one badly configured domain.
 
  • Like
Reactions: Major Discord
Virtualization
Whether you’re a homelabber or a Fortune 100 corporation, virtualization is an important concept to understand and an essential skill. While you could get away with running everything on bare metal hardware, virtualization gives you the ability to run as many virtual servers as your hardware can support.

The benefit of going virtual is that you can take point-in-time snapshots of the disk for easy rollbacks and backups, segregate services and applications, and manage resources such that no rogue application can suck up all your disk, CPU and RAM.

Containers
Whether Docker, LXC, or something exotic, these are a form of user-space isolation rather than full virtualization. Generally, the kernel is shared between container guests and the OS, which makes for much faster initialization and lower resource usage, but does restrict your OS choices (e.g. no Windows guests if you're on Linux)

Another benefit of containers (mainly Docker) is it gives you access to a large library of pre-configured images with software like MariaDB pre-installed and ready to go.

Some hypervisors (fancy word for virtual machine host software) support containers as well as full virtualization, but this is not the case for most.

VMware ESXi (vSphere)
ESXi used to be the go-to option for nerds at home wanting to gain marketable skills. It was a darling in virtualization from small business all the way up to the largest corporations in the world, but thanks to its recent acquisition by Broadcom, it has lost a lot of favor.

The strength of VMware ESXi is that it’s easy to use, performant, well supported and reliable. This is mostly still true today, but recent anti-consumer changes such as the removal of perpetual licensing, increased minimum core counts for licensing, locking down updates, and firing customers who reduce their licensing consumption, has meant its reputation has soured immensely.

This isn’t to say that corporations aren’t using VMware anymore, they absolutely are, but smaller players (such as you) are not welcome to use it anymore. There is no longer a free version, and pirates can no longer receive updates online. At the moment there’s no DRM on offline updates, and the latest version still accepts perpetual keys which are easily obtainable with a Google search, but I would not recommend it for new homelabs.

VMware Workstation
One of the few good things to come out of the Broadcom acquisition is that VMware Workstation is now completely free. Unlike ESXi which runs bare metal, this runs as an application on your Windows or Linux (read the DIY section before going down this path) PC and is an easy way to get started with virtualization.

Microsoft Hyper-V
Similar to VMware Workstation, Hyper-V can run atop the Windows operating system and virtualize any x86 guest. Enterprise users will typically setup dedicated hosts using Windows Server, but you can run it on consumer versions of Windows so long as you’re using Pro, Enterprise or Education editions.

In my opinion VMware Workstation is easier to use, but Hyper-V is a much more marketable skill than Workstation if you’re interested in developing skills for a career in IT.

Proxmox Virtual Environment
In the wake of VMware completely shitting itself, many smaller enterprises have been migrating to Proxmox VE as it offers a compelling feature set at a fraction of the cost. For autists at home, the price is $0 as Proxmox is FOSS, you need only suffer a nag screen when you logon.

For those familiar with VMware or Hyper-V, it’s a learning experience, and for total newbies it’ll take a while to get to grips with its somewhat convoluted UI. For those familiar with Debian Linux, you’ll feel right at home as PVE is built on Debian, though you do need to install it using their customized ISO.

This is my recommended option for savvy autists who have moved beyond desktop virtualization such as VMware Workstation and are looking to virtualize on dedicated hardware. It has native support for Linux containers (LXC) so it’s a good option for those who are starved of resources or just prefer containers.

There's also out of the box support for full backups and Proxmox Backup Server is available for free if you need incremental backups.

XCP-ng
While I would wager Proxmox VE is the most popular vSphere alternative, another one I’ve seen mentioned a lot is XCP-ng. This is a community supported distribution of Citrix XenServer which bundles Xen Orchestra and uses Xen HVM as the hypervisor.

I have not used it and cannot comment on its user friendliness, the main complaint I’ve seen is that it has a 2 TiB limitation for virtual disks at the moment. This isn’t likely to cause issues for homelabbers and they aware committed to fixing this.

It's definitely worth considering if you're shopping around for a hypervisor.

DIY
Fundamentally all the components you need to virtualize are available in most Linux distributions. KVM (Linux kernel virtualization module) + QEMU (device and hardware emulator) is all you technically need for a fully-fledged Linux-based virtualization machine.

There are GUIs like virt-manager which provide an interface for KVM-based VMs and this is what I'd suggest to anyone wanting to do desktop virtualization on Linux as it's far better supported than running VMware Workstation.

In Conclusion...
Don't ignore virtualization when building up your homelab. It's a critical component to a well functioning lab and an invaluable skill to demonstrate if you want a career in IT.

You may ask why I didn't mention Oracle VirtualBox, and it's mainly because I don't understand why you'd use VBox when Workstation is free on Windows and you have KVM on Linux.

I hope you enjoyed my essay and find it useful.
 
  • Like
  • Feels
  • Winner
Reactions: WinRAR Free Trial, CoolSixtyNiner, GreenTips and 4 others
I REALLY need to learn how to make a cheap JBOD and get it hooked up to my HP Proliant. Storage is an issue for me (using an external hard drive, don't @ me to tell me I'm dumb, I already know). I went balls deep and everything is held together with duct tape and super glue. Now I'm not home to play with it all the time, so I've been relegated to doing maintenance/updates when I go home every other month because I never learned ubuntu well enough to be able to install a VPS or learn how to do it remote. Oh well, at least I can access my files through plex.
 
Barry McKockner said:
I REALLY need to learn how to make a cheap JBOD and get it hooked up to my HP Proliant.
Click to expand...
How many drives are you thinking? If it's =< 8 then consider an HBA with those SAS/SATA fan out cables. Most cards have 8 SAS lanes but you can get some which are 16, or put multiple cards into a system. (They're usually PCI-E x8 )

If you're in the realm of >16 drives then you'll want to look at external JBOD disk shelves. NetApp DS4246 is popular (do not get a 4243 unless it's super cheap and you can source an IOM6 controller without the total cost exceeding a DS4246) and is a 24 bay shelf with a SAS expander built in. What the expander does is multiplexes the 4 SAS lanes to make all 24 drives accessible over them.

There's other options from Dell and HP as well which are available on the used market. All these options come with the caveat that they are loud since they're designed to be in a server room. You cannot replace the fans easily in those NetApp shelves as they're actually inside the PSU modules.

I did a bit of searching and found this thread which has a lot of good info
TheBallPit

Thread 'LSI / Broadcom HBAs and IT mode firmware - where to get for the 9400-16i?'

Now that HDMI 2.1 has FINALLY come to iGPUs and DDR5 has become user friendly (more or less), I'm preparing a system build and the LSI / Broadcom 9400-16i is on the list, as I'll have a lot of drives to connect.

Does anyone have any familiarity with flashing LSI cards to IT mode? Can it be done under DOS or Windows or does it have to be in a Linux environment? Assuming this particular card needs it since it claims "tri-mode" OOTB, can anyone direct me to the proper firmware?
 
You must log in or register to reply here.
Back