Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Potato Salad
Oct 23, 2014

nobody cares


skipdogg posted:

Anyone here with a very large Nutanix deployment? (3K+ VM's?).
...

I'm sure it's gotten better in the last 4 years, but is it really ready for big enterprise?

an MSP adjacent to me just got done moving a maaaaaaassive customer OFF of Nutanix in order to have some essentials (like uptime) and better features (like useful support). Idk man.

Adbot
ADBOT LOVES YOU

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

Potato Salad posted:

an MSP adjacent to me just got done moving a maaaaaaassive customer OFF of Nutanix in order to have some essentials (like uptime) and better features (like useful support). Idk man.

Yeah, I'm not comfortable at all with it, but I'm not in that department. Trying to decide if I get involved or not. Internal stuff I've read... some folks think Nutanix is going to be our saviour. I'll run it by my boss, he's pretty saavy at the political stuff.

The NPC
Nov 21, 2010


skipdogg posted:

The PRISM software left a lot to be desired

:nsa:

Pile Of Garbage
May 28, 2007



I've always been skeptical of HCI solutions in general as all the implementations seem to be extremely fragile. They're often reliant on one or more black-box appliances which hide a lot of complicated moving-parts that you can't easily diagnose or troubleshoot without vendor support.

Also as mentioned upgrades can take an extremely long time which is an issue not unique to Nutanix. A couple years ago I upgraded a pair of three-node Cisco HyperFlex (It's a HCI layer on top of VMware ESXi running on Cisco UCS) clusters and each one took about 16 hours to complete. Admittedly it was simple and fully orchestrated but still I had to stare at progress bars for 16 hours.

Maybe there's a niche for HCI, maybe running VDI (The HyperFlex clusters I upgraded were solely for Horizon VDI), but for workloads that you care about it seems like a big gamble.

Kaddish
Feb 7, 2002
I feel like the best use case for HCI is smaller installations with no or little existing infrastructure.

SamDabbers
May 26, 2003



Anybody considering Red Hat OpenShift Virtualization (or kubevirt on some other flavor of k8s) as a VMware alternative? Seems worth kicking the tires especially if your org is already running clusters.

Potato Salad
Oct 23, 2014

nobody cares


Kaddish posted:

I feel like the best use case for HCI is smaller installations with no or little existing infrastructure.

agreed, and that's where we use it

propping up a new environment with VMware and, say, Pure storage introduces a lot of single points of failure (that also prevent maintenance during business hours) unless you are spending above half a million dollars. ESXi + VSAN or Hyper-V + S2D allows you to give somebody storage and compute at, heck, $30-60k per node depending on how much storage and ram density is needed. replace old nodes on a rolling basis and suddenly you have a situation that resembles a cloud deployment but way cheaper.

Thanks Ants
May 21, 2004

#essereFerrari


It's cool how MS don't give a single poo poo about developing that product any further unless you fancy tying it all into Azure

Saukkis
May 16, 2003

Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.
Does it really matter if the upgrades take a long time. That's just how things work with a multinode clusters where the upgrades can and should be done one piece at a time. Last month I upgraded a few ESXi clusters and it took days, but it was just several steps of clicking few buttons and then going off to do something else while all the services were working fine the whole time. Just set an alarm in half an hour so you remember to check if you can proceed to the next step.

Are there any non-trivial clusters that aren't just collection of black boxes. Even if it was open source it will be complicated enough only few people will understand how it works. A breakroom axiom from work is that when you build a fancy high-availability cluster you just get a new and unexpected ways for everything to break down. But I have to admit I am really surprised how well VMware has worked for the past decade that I've been dealing with it. It has been more realiable than many of the simple clusters I've had to deal with during that time. Thankfully that intraweb cluster where both nodes had to be started exactly at the same time was before my time. I've heard the story how two guys were standing in front of turned off servers and yelling "1, 2, 3, START!" and hoping they timed it right. At least nowadays the simple clusters work pretty well, but it hasn't been long time since I still had a cluster where the service IP moved between the nodes right as planned, it just forgot to tell the rest of the world where the IP was now.

rufius
Feb 27, 2011

Clear alcohols are for rich women on diets.

Thanks Ants posted:

It's cool how MS don't give a single poo poo about developing that product any further unless you fancy tying it all into Azure

I mean… why would they? It’s a gateway drug to azure and it’s great for the bajillion enterprise customers migrating over time from WinServer.

Subjunctive
Sep 12, 2006

✨sparkle and shine✨

I'm a babe in the proxmox woods, but I haven't found anything about this by searching or reading config files, so I'll supplicate here to the virtualization gods:

I want to set up login to the proxmox web interface such it searches multiple auth domains ("Linux PAM standard authentication" and "Proxmox VE authentication server") and the user doesn't have to select the one they're in. Is that viable, or should I find a way to unify the users somehow?

(I don't care about how conflicting usernames are handled, because I won't have any that aren't the same user.)

Wibla
Feb 16, 2011

Can you tell us a bit more about what you are really trying to do here? That sounds cumbersome...

Thanks Ants
May 21, 2004

#essereFerrari


Skimming the documentation suggests that multiple auth sources means configuring them as different realms, and those are presented in a dropdown on the login page.

Subjunctive
Sep 12, 2006

✨sparkle and shine✨

Wibla posted:

Can you tell us a bit more about what you are really trying to do here? That sounds cumbersome...

I have some users who use the web interface to log in and do things like restart a specific VM or access one VM’s console, and I’m trying to simplify the login process so they don’t have to pick a realm. These users are defined with the proxmox UI and live in that realm, but root and I live in the PAM realm. I just want to hide that complexity because it’s not relevant to them.

Thanks Ants posted:

Skimming the documentation suggests that multiple auth sources means configuring them as different realms, and those are presented in a dropdown on the login page.

Yeah, that’s the situation I’m in right now that I want to simplify.

Wild EEPROM
Jul 29, 2011


oh, my, god. Becky, look at her bitrate.

skipdogg posted:

Anyone here with a very large Nutanix deployment? (3K+ VM's?). Obviously we're looking at a potential exit from Vmware like every other company.

theres a guy in this very thread who had an absolute dog poo poo time with nutanix, it's a wild read

Harry_Potato
May 21, 2021

Subjunctive posted:

I have some users who use the web interface to log in and do things like restart a specific VM or access one VM’s console, and I’m trying to simplify the login process so they don’t have to pick a realm. These users are defined with the proxmox UI and live in that realm, but root and I live in the PAM realm. I just want to hide that complexity because it’s not relevant to them.

Yeah, that’s the situation I’m in right now that I want to simplify.

Put a Netscaler in front of it and I could do that all day long for you with a AAA vip..

H2SO4
Sep 11, 2001

put your money in a log cabin


Buglord

Saukkis posted:

Does it really matter if the upgrades take a long time.

it does when your hyperconverged infrastructure isn't smart enough to account for an extended delay and starts ejecting nodes from the storage and metadata rings because they didn't reboot quickly enough during a "one click" upgrade.

Harry_Potato posted:

Put a Netscaler in front of it and I could do that all day long for you with a AAA vip..

Absolutely do not suggest touching Netscaler right now given all the bullshit going on at Citrix. They were already a very neglected business unit before going private and it definitely hasn't gotten better. Simple reverse proxy (and even basic AAA) is such a solved problem odds are you can do it on a firewall you already got on the edge in the first place.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer

Kaddish posted:

Looks like my org is just going to eat the cost for the next 3 years. I think they still want to explore options though.
This is my org as well. Our problem is compounded by the fact that we JUST transitioned a lot of networking and security into NSX and now are grappling with continuing on that path, migrating the network back out to more physical devices, or what.

We are reducing our licensed core count to reclaim a couple hosts to start playing with alternative platforms. It’s likely we might move to a multi-tier environment where VMware provides a shrinking tier 1 platform for critical systems and everything else winds up on something else.

Anyone else in a similar boat? Especially with NSX? We also have very consistent demand and not a lot of growth (local government) but the growth we do have is heavily biased toward storage consumption, so HCI and its less flexible scaling has never been a great fit.

Pikehead
Dec 3, 2006

Looking for WMDs, PM if you have A+ grade stuff
Fun Shoe
My org was looking to transition to the entire stack (VCF/NSX/whatever) so were well into a full bore Greenfield build when Broadcom completed the purchase and then put the boot in.

There's nothing like having bought completely new hardware and a third party starting to implement things, to get stopped hard on licenses. We requested the licenses we needed from VMware Broadcom, only to be told after a couple of weeks of waiting "lol, we aren't issuing licenses at this time, come back in a couple of months".

Worked through that but we are fully into VMware and have no other viable options. It's been an interesting couple of months.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer
The licensing “migration” has been insane. It coincided with our maintenance ending on our old perpetual licenses and they won’t sell us new maintenance and are just like “it’s OK we’ll toooootally extend your support without a contract or anything if you promise you’ll be buying VCF.” We had no choice but to take them at their word.

Clowns.

Nitrousoxide
May 30, 2011

do not buy a oneplus phone



Today I learned that Proxmox has (experimental) support for SDNs. That's pretty neat. I might have to play around with that to break out my vm's into vlans all inside the computer without ever having to loop out to my home network. Hopefully would be a lot faster.

repiv
Aug 13, 2009

vmware has made workstation pro and fusion pro free for non-commercial use

https://blogs.vmware.com/teamfusion/2024/05/fusion-pro-now-available-free-for-personal-use.html

maybe a partial reversal on their killing off the entry level products? wonder if they'll bring back esxi free

SlowBloke
Aug 14, 2017

repiv posted:

vmware has made workstation pro and fusion pro free for non-commercial use

https://blogs.vmware.com/teamfusion/2024/05/fusion-pro-now-available-free-for-personal-use.html

maybe a partial reversal on their killing off the entry level products? wonder if they'll bring back esxi free

On the surface this is good but it messes up everyone who used player in a commercial context. This is kinda like oracle making jre a paid for item.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer
The only thing I can think is that they realized they hosed up their reputation/goodwill pipeline with education by taking away free ESXi and this is their attempt to course correct

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

SlowBloke posted:

On the surface this is good but it messes up everyone who used player in a commercial context. This is kinda like oracle making jre a paid for item.

Audit bait.

the spyder
Feb 18, 2011

skipdogg posted:

Anyone here with a very large Nutanix deployment? (3K+ VM's?). Obviously we're looking at a potential exit from Vmware like every other company.

My last experience with Nutanix was 4 years ago at a previous employer, and while they sold it as this amazing mature solution, reality was there were way too many unsupported scenarios and unexpected issues that ultimately ended up scrapping the entire project.

According to my friend that still works at my previous employer, the entire Nutanix project was scrapped. Roughly 15M worth of hardware and licenses never saw a production load.

From what I remember our major issues were lack of support from RedHat for AHV necessitating a Nutanix managed ESX-i cluster anyway (I believe AHV is supported now), insane licensing fees for flash storage and nutanix files (A loaded DL380 was about 55K and needed 200K worth of licenses making a single node 250K w/5year support), The PRISM software left a lot to be desired, and 16 node cluster upgrades would take days to finish and you had little control over the upgrade process. If your workload and VM's were fairly standard sized it wasn't a problem, but some of our much larger VM's we had issues

I'm sure it's gotten better in the last 4 years, but is it really ready for big enterprise?

I seem to be one of the few willing to talk about our NX environment.
We're a large healthcare system, going on 3 years now with 15 clusters.

Currently running 1k prod vm's + 1k PVS (app presentation) on AHV/AOS 6.5. All NX hardware. Moving an additional 1.5k this summer.
Primarily a large windows shop with ~100 RHEL VM's tossed in. We run everything from 2vcpu 4GB up to 32vcpu and 512GB.

Current bugs:
1) Windows machines will reboot fast enough on our NVME nodes that ARP does something funky between the vswitch and our nexus TOR switches, resulting in the VM grabbing a 169 IP. Does not happen on our ACI network in the DC, only at our campus sites. Happens 1:200, so it's not a big deal. But annoying during patching reboots. Networking does not want to change their switch side config, so we just wrote a script to check for the IP state and reboot if necessary. NX has an engineering bug filed.
2) LCM upgrade will freeze on G7 hardware. It's a super micro issue. Can't fault NX here. No issues on G8 or G9.
3) RHEL VM's behave 99.7 percent of the time - difficult to tell if the issues we see are due to our CIS hardening or cranky RHEL admins. NX can not replicate on their HW/Lab. Again, no evidence to blame NX.
4) Prism Central sizing - there's was an advisory emailed out last month linking to a KB due to our experience (and others)- their documentation was wrong and has since been updated/improved. NX troubleshot and resolved. Took longer than I wanted, but did not impact prod.

AOS 6.5 brought forth major changes to the platform, even the latest CE is based on it. We saw major performance improvements in both storage and guest VM. We've never had stability problems or any form of outage outside of a rouge node failure and reboot of the VMs.

I would 100% never buy a hybrid cluster, unless your workload + 3yr forecast could handle it. We are having to add NVME nodes at our campuses - but only due to increased workloads beyond design. Having said that, Files on those hybrid clusters is working well for BCDR, but we do not run any prod file shares at this time. We have several multi petabyte Qumulo clusters for SMB, and imo it's a better solution.

YMMV with support. Your overall experience really depends on your account team and involvement in the solution design. I have a great team and they've helped navigate any support issues we've had. We plan to keep these around through HW EOS and after that, we'll see. Every shop, no matter how big or small should be regularly evaluating their solutions/partners.

Pricing wise, I can't publish our pricing, but with the updated core based model you no longer license Flash/TiB's. It's significantly less than what you posted. Let's estimate that 1 node would run about $75k with 5yr terms. When we were faced with refreshing our UCS, MDS, and PURE three-tier - then you add in the new VMware licensing and NX was a cost savings, even with excellent pricing from our other partners.

Our team spends less time managing the individual components and patching/troubleshooting our large Hyper-V environment - enough that it reclaimed 1.5FTE. This was after significant automation efforts.

Let me know if you have any specific questions.

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

the spyder posted:

I seem to be one of the few willing to talk about our NX environment.
We're a large healthcare system, going on 3 years now with 15 clusters.

Currently running 1k prod vm's + 1k PVS (app presentation) on AHV/AOS 6.5. All NX hardware. Moving an additional 1.5k this summer.
Primarily a large windows shop with ~100 RHEL VM's tossed in. We run everything from 2vcpu 4GB up to 32vcpu and 512GB.

Current bugs:
1) Windows machines will reboot fast enough on our NVME nodes that ARP does something funky between the vswitch and our nexus TOR switches, resulting in the VM grabbing a 169 IP. Does not happen on our ACI network in the DC, only at our campus sites. Happens 1:200, so it's not a big deal. But annoying during patching reboots. Networking does not want to change their switch side config, so we just wrote a script to check for the IP state and reboot if necessary. NX has an engineering bug filed.
2) LCM upgrade will freeze on G7 hardware. It's a super micro issue. Can't fault NX here. No issues on G8 or G9.
3) RHEL VM's behave 99.7 percent of the time - difficult to tell if the issues we see are due to our CIS hardening or cranky RHEL admins. NX can not replicate on their HW/Lab. Again, no evidence to blame NX.
4) Prism Central sizing - there's was an advisory emailed out last month linking to a KB due to our experience (and others)- their documentation was wrong and has since been updated/improved. NX troubleshot and resolved. Took longer than I wanted, but did not impact prod.

AOS 6.5 brought forth major changes to the platform, even the latest CE is based on it. We saw major performance improvements in both storage and guest VM. We've never had stability problems or any form of outage outside of a rouge node failure and reboot of the VMs.

I would 100% never buy a hybrid cluster, unless your workload + 3yr forecast could handle it. We are having to add NVME nodes at our campuses - but only due to increased workloads beyond design. Having said that, Files on those hybrid clusters is working well for BCDR, but we do not run any prod file shares at this time. We have several multi petabyte Qumulo clusters for SMB, and imo it's a better solution.

YMMV with support. Your overall experience really depends on your account team and involvement in the solution design. I have a great team and they've helped navigate any support issues we've had. We plan to keep these around through HW EOS and after that, we'll see. Every shop, no matter how big or small should be regularly evaluating their solutions/partners.

Pricing wise, I can't publish our pricing, but with the updated core based model you no longer license Flash/TiB's. It's significantly less than what you posted. Let's estimate that 1 node would run about $75k with 5yr terms. When we were faced with refreshing our UCS, MDS, and PURE three-tier - then you add in the new VMware licensing and NX was a cost savings, even with excellent pricing from our other partners.

Our team spends less time managing the individual components and patching/troubleshooting our large Hyper-V environment - enough that it reclaimed 1.5FTE. This was after significant automation efforts.

Let me know if you have any specific questions.

Thanks so much. I appreciate the time you took to post this.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer

the spyder posted:

Current bugs:
1) Windows machines will reboot fast enough on our NVME nodes that ARP does something funky between the vswitch and our nexus TOR switches, resulting in the VM grabbing a 169 IP. Does not happen on our ACI network in the DC, only at our campus sites. Happens 1:200, so it's not a big deal. But annoying during patching reboots. Networking does not want to change their switch side config, so we just wrote a script to check for the IP state and reboot if necessary. NX has an engineering bug filed.
We had a similar issue years ago (on VMware) and pushed out a GPO disabling gratuitous ARP - it sets a REG_DWORD ArpRetryCount at HKLM/System/CurrentControlSet/Services/Tcpip/Parameters to 0

Clark Nova
Jul 18, 2004

repiv posted:

vmware has made workstation pro and fusion pro free for non-commercial use

https://blogs.vmware.com/teamfusion/2024/05/fusion-pro-now-available-free-for-personal-use.html

maybe a partial reversal on their killing off the entry level products? wonder if they'll bring back esxi free

lol https://matduggan.com/the-worst-website-in-the-entire-world/

The link to sign up (support.broadcom.com, a bad sign in itself) is currently showing the cloudflare "this site has poo poo its doo doo rear end" page

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer
Lmao I tried to download VMRC today and it’s nowhere to be found

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast

Aunt Beth posted:

Lmao I tried to download VMRC today and it’s nowhere to be found

Have fun when you try to search online for some VMware issue, find a relevant result, click on it, then end up on some godforsaken Broadcom landing page because now the support pages are all dead and buried.

They're really loving the dog

Kaddish
Feb 7, 2002

HalloKitty posted:

Have fun when you try to search online for some VMware issue, find a relevant result, click on it, then end up on some godforsaken Broadcom landing page because now the support pages are all dead and buried.

They're really loving the dog

Same thing happened when they swallowed up Brocade. Not great for a fabric switch administrator with no up-to-date support contract.

They keep FOS updates under lock and key now like their lives depend on it.

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast

Kaddish posted:

Same thing happened when they swallowed up Brocade. Not great for a fabric switch administrator with no up-to-date support contract.

They keep FOS updates under lock and key now like their lives depend on it.

Yeah, it's extreme, the newest fw versions require installation of a certificate on the switch before an update can be done. You probably know that anyway but yeah

Kaddish
Feb 7, 2002

HalloKitty posted:

Yeah, it's extreme, the newest fw versions require installation of a certificate on the switch before an update can be done. You probably know that anyway but yeah

I did not, I'm running 8.x still.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer

Aunt Beth posted:

Lmao I tried to download VMRC today and it’s nowhere to be found

Update: found an old version squirreled away in our misc tools network share, installed it, then SCCM used whatever magic it has to update that as soon as it detected the out of date version, and one of our desktop support guys grabbed the MSI out of cache for me.

Potato Salad
Oct 23, 2014

nobody cares


Broadcum is finally trickling out new quotes again.

Big client has enough of a VMware licensing cost increase to just buy Scale hardware outright :confuoot:

Potato Salad
Oct 23, 2014

nobody cares


FYI vmrc is on winget and chocolatey

afflictionwisp
Aug 26, 2003

HalloKitty posted:

They're really loving the dog

Was on a call just a few hours ago with our Supermicro and VMware technical reps troubleshooting a VCF deployment issue, and they were both ripping Broadcom like they belonged in this thread. Im enough of a cynic to tale pleasure in the fact that even the VMware people are pisses off and hopeless.

Motronic
Nov 6, 2009

afflictionwisp posted:

Im enough of a cynic to tale pleasure in the fact that even the VMware people are pisses off and hopeless.

It's also got to be the only sane way to handle customer interactions when you're front line support at a time like this. You do not want to be sounding like you tow the company line because your clients will take that as license to treat you like the company. You want to align with your clients on this one.

Adbot
ADBOT LOVES YOU

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast

Kaddish posted:

I did not, I'm running 8.x still.

Well, no support, no newest 9.x. The certs are generated by Broadcom from the portal and tied to the hardware serial.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply