I’m in the process of wiring a home before moving in and getting excited about running 10g from my server to the computer. Then I see 25g gear isn’t that much more expensive so I might was well run at least one fiber line. But what kind of three node ceph monster will it take to make use of any of this bandwidth (plus run all my Proxmox VMs and LXCs in HA) and how much heat will I have to deal with. What’s your experience with high speed homelab NAS builds and the electric bill shock that comes later? Epyc 7002 series looks perfect but seems to idle high.
125W (Less than $15/month) or so for
- Ryzen 9 3900X
- 64GB RAM
- 2x4TB NVMe (ZFS Mirror)
- 5x14TB HDD (ZFS RAID-Z2)
- 2.5GBe Network Card
- 5-port 2.5GBe Network Switch
- 5-port 1GBe POE Network Switch w/ one Reolink Camera attached
I generally leave
powerManagement.cpuFreqGovernor = "powersave"in my Nix config as well, which saves about 40W ($4/mo or so) for my typical load as best as I can tell, and I disable it if I’m doing bulk data processing on a time crunch.5950x in an matx board with 15 x 3.5in drives 1 x sata sad 1 x optane u.2 drive (pulls like 10watts) 1 x Nvidia A2000 1 x Lsi 9305 16i 1 x 2.5gbe intel nic 3 x 140 mm fans at full tilt
Runs at like 120 watts at idle, like 220 watts with a good amount of work and peaks at like 320 watts if I make it do a lot of work
What in the world…
Around 100 Watts for
- NAS with 4x3.5" HDD,
- Minisforum HM90 for Proxmox with 2x2.5" HDDs,
- 16 Port TP Link PoE Switch,
- TP Link router
- 2x Raspberry Pi 4b
But everything with gigabit speed. Doesnt need more at home
My pi costs probably around 20 a year lol.
I’ve got a 3 node Proxmox/ceph cluster with 10G, plus a separate Nas. They are all rack mount with dual PSU. Add in the necessary switching, and my average load is about 800w. Throw my desktop (also on 10G) into the mix and it runs 1.1kw.
That’s roughly $50-60 extra in electricity costs for me monthly.
Would be around 300€ in Germany, on a cheap contract. Limiting myself to one combined NAS/application server atm, with the others turned on only if I want to try sth out.
Let’s see…
My servers (tiny/mini/micros) in total are about… 600W or so. Two NASs, about 15-20W a piece.
I spend a out $150/mo in electricity, but my hot water/HVAC/etc are the big power draw. I’d say about $40-50/mo is what I’m spending on powering the servers in my office.
Definitely puts off some heat, but that’s partially because it’s all in one rack, and I’ve got a bunch of other work hardware in there. It’s about 2 degrees warmer in my office than the rest of my home, but I also have air cycling all the time since it’s a single unit HVAC and I need to keep the air moving to keep it all the right temp in the other rooms anyway (AC will come on more often otherwise, even without my rack).
You most likely won’t utilize these speeds in a home lab, but I understand why you want them. I do too. I settled for 2.5GBit because that was a sweet spot in terms of speed, cost and power draw. In total, I idle at about 60W for following systems:
- Lenovo M90q (i7 10700, 32GB, 3 x 1 TB SSD) running Proxmox, 15W idle
- Custom NAS (Ryzen 2400G, 16GB, 4x12TB HDD)v running Truenas (30W idle)
- Firewall (N5105, 8GB) running OPNsense (8W idle)
- FritzBox 6660 Cable, which functions as a glorified access point, 10W idle
With 25 GbE, even 10, I’d be tempted to PXE boot client systems. Maybe still have a local PCIe SSD for windows game files.
Dunno how that would actually work with Windows, but it was fun when I did it for beowulf nodes. Setting RPis to netboot is a little involved, but you can create an OSMC image and give all your TVs a consistent ‘smart’ interface. You don’t even need 10GbE to be pretty functional for the Pi, but my experience is that WiFi is not fast enough.
I have an ITX Ryzen 2700X with an arc A380. 3 HDDs and 1 SSD boot drive.
Before some kernel improvements for the A380, my idle wattage was 60W. Without the A380 it was around 35W idle. I am hoping that it is around 45W now because of fixing the high idle wattage of the GPU but I have to measure again.
Performance is great though. Perfect Jellyfin streaming, home automation, document and media management, file sync, recipe management, etc…
People tend to over-spec their servers, in my opinion. Unless you are dealing with more than a few dozen clients or so on one server (or having a many-user dedicated streaming server), you really don’t need much.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters AP WiFi Access Point HA Home Assistant automation software ~ High Availability LXC Linux Containers NAS Network-Attached Storage NUC Next Unit of Computing brand of Intel small computers NVMe Non-Volatile Memory Express interface for mass storage PCIe Peripheral Component Interconnect Express PSU Power Supply Unit PoE Power over Ethernet RPi Raspberry Pi brand of SBC SATA Serial AT Attachment interface for mass storage SBC Single-Board Computer SSD Solid State Drive mass storage Unifi Ubiquiti WiFi hardware brand VPS Virtual Private Server (opposed to shared hosting)
[Thread #782 for this sub, first seen 4th Jun 2024, 04:35] [FAQ] [Full list] [Contact] [Source code]
My real server (Nextcloud/NAS/several more vm’s) uses 28 Watts on average. In addition, there is one Pi 4B running, and I don’t even know it’s wattage.
I’m planning on replacing the real server with a new one, with lots of cores and approx. 50 Watts then.
If you’re just running home automation, you do not need an Epyc 🤣
Get a low power anything to just run what you need.
I looked at Epyc because I wanted to bandwidth to run u.2 drives at full speed and it wasn’t until Epyc or Threadripper that you could get much more than 40 lanes in a single socket. I’ve got to find another way to saturate 10g and give up on 25g. My home automation is run on a Home Assistant Yellow and works perfectly, for what it does.
Some unsolicited advice then: don’t go LOOKING for reasons to use the absolute max of what your hardware is capable of just because you can. You just end up spending more money 🤑
For real though, just get an N100 or something that does what you need. You don’t need to waste money and power on an Epyc if it just sits idle 99% of the time.
What I need is a 10g storage for my Adobe suite that I can access from my MacBook. I need redundant, fault tolerant storage for my precious data. I need my self hosted services to be high availability. What’s the minimum spec to reach that? I started on the u.2 path when I saw enterprise u.2 drives at similar cost per GB as SATA SSDs but faster and crazy endurance. And when my kid wants to run a Minecraft server with mods for him and his friends, I better have some spare CPU cycles and RAM to keep up.
