HomeLab: Hybrid Cloud | LAB2PROD

Scaling out the Datacentre

After posting about my latest homelab acquisitions, I’ve gotten a lot of interest to understand what I’m running and how I got there. This blog explores my homelab journey. However, if you’re just interested in the actual hardware specs, you can find them either at the end of this article or here in the ‘Lab’ page on this website.

My lab started pretty small.. a single HP micro server, which I still had up until a couple of years ago. I will admit, for a while it was just a large paperweight.

As my job started to get more challenging, I took it upon myself to invest in a proper homelab kit. It started with a single Dell R710 (good old go to) then quickly to a second one as well as a 12-bay R510 for storage.

This kit served me well for many years, till I started to run nested VCF. It was a huge accomplishment to run nested VCF with this hardware, but that’s as far as it could take me. The HDDs and processors were the biggest bottlenecks.

State of the HomeLab late 2019

Last year, I got rid of the R510 (bottom) and one of the R710’s and replaced them with two R720’s with a few SSD’s each. The purpose of these were to run a 2-node stretched vSAN with a witness sitting on the other host. I also could run a better nested VCF environment on these hosts.

All well and good.. but something just didn’t feel right with my homelab..

2020 Upgrades

I was missing some critical networking, compute and storage. So I bought a Brocade ICX6610 which I use as the core switch in my homelab + another server (R730XD). I repurposed the R710 for storage, running baremetal FreeNAS with 16TBs, the R730XD replaced the R710 as the dedicated management box. It has since run everything from monitoring, media, NSX-T, some virtual routers, and backups. Plus it has it’s own array of SAS disks, so runs a second instance of FreeNAS presented to the network.

The 2021 upgrade and expansion to AWS

So….. started hitting similar issues as before.. some major storage latency to run all those nested VM’s for nested VCF. I finally had enough of it and snatched up 2 extra Fujitsu Primergy RX2540’s from a mate that was leaving town.

I fit 2 NVMe disks into each Fujitsu and the R720’s, and an SSD each, and gave them each 2 x 2 port 10gbit NICs. This is so I can run a 4 node consolidated architecture VCF instance with multi-nic. I deal with it every day for clients, so it made sense to invest in something that I could test and enhance my skills on. Let’s not forget that sweet velocloud box and expansion to AWS!

Racked, Stacked and Ready to Roll

And we finally made it to the day of install… racking all this kit in a hot garage isn’t recommended, however it really got my virtualization juices flowing!

Below are some pics setup and end result.

Lab Logical Design

With the history of my lab and physical set up out of the way, it’s time to get to the fun stuff.

Below is a diagram of my homelab’s the logical cluster configuration.

And finally we have a logical network diagram, for you other network nerds. It shows at a high level, how I utilize VyOS for NSX-T and VCF and the link to my VPC’s in AWS.

Homelab Hardware Specs

Network

• velocloud at the edge – can’t be seen in the picture. There will be followup posts on this
• Sophos XG430 running PFsense
• Brocade ICX6610, running OSPF between PFsense and this
• 2 QSFP+ to SFP+ DAC breakout cables to the two Fujitsu hosts
• 8 twinax 10gbit links to the R720’s
• A whole heap of virtual routers running on the R730
• UniFi AC Pro’s for wireless

Storage

R710

• Dual Xeon X5670’s
• 96gb RAM
• 6x6tb WD RED’s

Management

R730XD

• Dual Xeon E5-2630v3’s
• 256 FG DDR4 RAM
• Intel 1tb PCIE NVMe SDD – for services such as AD, vCenter, vSAN Witness and NSX-T Manager
• Nested FreeNAS pool 8 x 1.2TB SAS HDDs for storage and 2 x SAS 256GB SSD’s for slog
• Dell H730 mini in passthrough mode

Compute

2 x R720’s

• 2 x E5-2670’s and 2 x  E5-2620 v2’s
• 256GB RAM in each server
• 250GB NVMe vSAN cache + 2 1tb SSD + NVMe for vSAN capacity
• 2 x Intel x520-DA2 NICs
• HBA330

2 x Fujitsu Primergy RX2540’s

• 2 x E5-2640 v3’s
• 256GB GB DDR4 RAM
• 250GB NVMe vSAN cache + 2 1tb SSD + NVMe for vSAN capacity
• 2 x Intel x520-DA2 NICs
• PRAID / SAS controllers

Hope you enjoyed the read!

A summary of the hardware specs can be found here Lab.