<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/thumbnail.PNG#center" alt=“r710 Lives”;>
TL;DR: The Dell R710 is a staple in the homelab community. Many individuals have indicated that their 11g (generation) CPU’s were not allowing them to upgrade to vCenter 6.7 due to the disallowed list located here. I was able to upgrade to vSphere 6.7 on my R710’s which use an L5630 (Westmere) CPU. I’ve seen evidence that many of the 56xx series CPU’s are still allowing upgrades, despite a warning being presented. I was unable to use VUM for the upgrade as it presented an incompatible error message. Instead, I used a workaround involving the uploading the depot ZIP file and issuing a VIB command (shown below) to complete the upgrade. There is peace in the lab once again. If you are looking to the 11g generation of servers; ensure you are getting CPUs that the community has validated are still working. Many of the 56xx series appear to be functional - specifically the L5630 in my case, or X5650 in Brian Graf’s case.
Update - Updated to reflect leveraging software profile update instead of software vib install command
Second Update - Added process from Christian Mohn around using tail -f /var/log/esxupdate.log to track update progress
First, as of now, the most success has been had with 56xx series Westmere processors. 55xx have been failing almost entirely. I mention this a few times in the blog post below; but it’s come up in a few questions - So I wanted to put it “Front and Center”.
Second, everything being done in this post is in the context of unsupported homelabbery. Yes, its a word. It’s absolutely not supported by GSS, and if you say “Well Cody said it works on his blog” is NOT going to get you the response you are hoping for.
Lastly, these CPU’s are largely throwing warnings about functionality ending soon. The release notes indicate that upgrades are going to be disallowed. Start planning now to replace these with newer generation servers/CPU architectures. It’s happening. Move past the denial.
It’s a pretty common understanding in the homelab community that R710’s and DL380 G6’s are some of the most well-rounded hardware you can get. They are enterprise grade, so you can make yourself feel like you are running a real datacenter. They are pretty cost effective (Because of 1) old hardware, 2) DDR3 Memory, 3) Aged CPU’s) but they still pack a pretty solid punch performance wise. There is a MASSIVE community around these systems. So what happens when that hardware just gets too old to remain supportable? Purple screens of death. That’s what. Children crying in the streets. Shattered dreams. Chaos.
Last week, vSphere and subsequently vCenter 6.7 were released. I don’t want to get into the weeds on all of the new features - but it’s absolutely a must have upgrade. The HTML5 client has gotten tons of love, the return of the embedded PSC, ESXi fast boot…summed up - it’s a big deal. A few great blog posts around the release are below:
Tucked into the release notes for vSphere 6.7 was this sad bit of information surrounding the treatment of newly unsupported and disallowed CPU’s:
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/01-unsupported.PNG#center" alt=“11g dies”;>
This list effectively covers many of the popular 11th generation (11g) of CPU’s. As I mention above, the low cost of these CPU upgrades, and hardware, made this generation the defacto standard for homelab builds with specifically the 54xx-56xx series (E/X/L models) being the most popular. The troubling part of this was the call-out that upgrades and installations were DISALLOWED for these unsupported CPUs. In past versions, we would receive a nag message that was able to be bypassed. This was different treatment entirely. Many of us started researching the availability of r720’s as a replacement kit for the homelab - however were very unhappy to find the cost to upgrade was substantial.
Hanging around the VMware Code Slack, vExpert Slack, and Reddit - I started to see a few rumblings of people having success running X/E 56xx series CPU’s in nested environments. The problem was, there wasn’t enough chatter to feel confident one way or another. I spoke with my good buddy Brian Graf and found that he had done a fresh install of 6.7 for his environment successfully using an X5650. I decided to give it a shot - but I needed to do an upgrade. Not a fresh install.
I spent the afternoon upgrading my vCenter environment to 6.7. I’ll document a few “gotchas” I found in another post - but for the most part it went off without issue. With VMware Update Manager (VUM) being built into the HTML5 interface in this release, I decided to give it a shot to upgrade my hosts to 6.7. I hopped into VUM and rolled up my sleeves.
I had already pulled down the ESXi 6.7 image, which I uploaded into Update Manager.
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/02-vum-image.PNG#center" alt=“New VUM Image”;>
I created a new baseline and attached it to one of my vSphere 6.5 hosts.
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/03-vum-baseline.PNG#center" alt=“New VUM Baseline”;>
I ran a scan, and was prompted that the current install was non-compliant with the new baseline I attached. This was a good sign!
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/04-check-compliance.PNG#center" alt=“Check compliance”;>
I checked the bubble for my ESX 6.7 baseline, and selected remediate.
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/05-select-remediate.PNG#center" alt=“Remediate”;>
Failure ensued.
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/02-vum-issue.jpg#center" alt=“11g dies”;>
At first, I assumed that my CPU was just incompatible - but then I thought about a workaround I picked up years ago over at Paul Brarens TinkerTry. Would it succeed if I uploaded the update depot ZIP onto a datastore, and ran the upgrade from command line? I pulled down the offline bundle upgrade from MyVmware
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/04-offline-bundle.PNG#center" alt=“Update Depot”;>
and uploaded it on one of my clusters shared datastores. I put my hltenesxi01 host into maintenance mode (also evacuating my vSAN data from it), started SSH, and connected to the host. Once connected, I ran the following command:
Update - After throwing this post up on reddit, I received a response from a user who commented that the better path to go is to update the software profile instead of doing a clean install, which will prevent VIBs from being removed/overwritten. It also moves the profile “up” to 6.7 instead. Def the cleaner path to go. I’ve changed the directions below to indicate this instead. The screenshot below still reflects using the VIB install command. I didn’t have a host to grab a new one for; and the screens look extremely similar.
esxcli software profile update -d /vmfs/volumes/hl-flash-ds01/VMware-ESXi-6.7.0-8169922-depot.zip -p ESXi-6.7.0-8169922-standard
After several minutes of hanging around waiting, I was greeted with a huge glob of text. Scrolling up, I found that the update had completed successfully, and was pending a reboot. All the VIBs that were upgraded were listed; and the final line indicated that no VIBs were skipped. Exactly what I was hoping to see.
Note: If you are running this with a later version of 6.7, and you are unsure about how to determine the profile name, I suggest you give a read to William Lam’s post here around using esxcli software sources profile list -d /path/to/depot/bundle.zip to list out profiles from the bundles.
<img src="/images/2018-04-22-vsphere-67-homelabs-unsupported-cpu/03-success.PNG#center" alt=“Command Successful”;>
I ran this upgrade process across my 2 node vSAN hosts, and my management host (not a concern; this one runs a supported CPU). I had no issues with the upgrade. And all hosts are now functional running 6.7.
Update : Thanks to Christian Mohn for the heads up about a slick way to monitor the upgrade process over on his blog. Essentially, if you start a second SSH session to the host, and run a tail -f /var/log/esxupdate/log - you can watch the progress live. Click here to check out his process @ vNinja!.
This is all totally unsupported. That being said, I wasn’t entirely thrilled about having to shell out some serious cash to upgrade to new hosts to consume the latest and greatest upgrades. I’m happy enough with this workaround that I can keep using my R710 workhorses for a bit longer. I’ll absolutely be putting some money aside to pick up a few R720’s in the near future, but in the short term - that’ll do cluster. That’ll do. I’ve heard mixed reviews of some specific 55xx series CPU’s being functional with upgrades too - but as time goes on, I’m hearing A LOT more around the 56xx series being more reliable.
After a sunday spent doing this migration, I’m having a blast labbing around with vCenter/vSphere 6.7 and finding enhancements left and right. Keep an eye out for a few posts where I dig a bit deeper into what these are!
(c) 2021 Copyright TheHumbleLab