Virtualization Megathread V2: VMs inside VMs

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Virtualization Megathread V2: VMs inside VMs

«‹›312 »

Wild Bill: Oct 4, 2001

Thanks for the quick response, I didn't see the Lab thread, probably should have dug deeper in the forums. I'll head over that way.

# ? Jan 30, 2014 20:55

Adbot: ADBOT LOVES YOU

# ? May 11, 2024 23:28

Dilbert As FUCK: Sep 8, 2007; by Cowcaster; Pillbug

Wild Bill posted:

Thanks for the quick response, I didn't see the Lab thread, probably should have dug deeper in the forums. I'll head over that way.

https://www.youtube.com/watch?v=Ntf0SBmWL1U

I did a video GUG on Home lab designs and how to set one up if you are interested.

I'd avoid things like Dell refurbs, I don't mind helping put a whitebox together

# ? Jan 30, 2014 21:15

Daylen Drazzi: Mar 10, 2007; Why do I root for Notre Dame? Because I like pain, and disappointment, and anguish. Notre Dame Football has destroyed more dreams than the Irish Potato Famine, and that is the kind of suffering I can get behind.

Dilbert As gently caress posted:

https://www.youtube.com/watch?v=Ntf0SBmWL1U

I did a video GUG on Home lab designs and how to set one up if you are interested.

I'd avoid things like Dell refurbs, I don't mind helping put a whitebox together

And watch out which loving RAID controller/HBA you buy - some of them are on the HCL, but are not really supported by the manufacturer and therefore do not interact with the hypervisor, defeating the whole purpose of buying one.

# ? Jan 30, 2014 23:34

Mausi: Apr 11, 2006

skipdogg posted:

Well I thought it was going to be a slow day at work, but a different department is having issues with their ESX 4 cluster. One of the hosts just shows up disconnected. Network connectivity and all the normal troubleshooting seems fine, storage is fine, but I can't even login to the local console on the physical host to try to restart the management service or anything. It's just completely hung.

On my old environment this was usually all-paths-down or similar issues, sometimes figuring out which HBA going into the host is hosed and unplugging it brought the management console back into service. But usually we just waited until after hours and gave it the 3finger salute.

Bob Morales posted:

Any reason for/against installing vCenter 5.0u2 on Server 2008 instead of 2012?

I'm running 5.5.0b on 2008r2 with no issues, so I can't see why you'd have issues with earlier releases. Going for the latest stable release which supports your hardware is 100% the best way to go.

# ? Jan 30, 2014 23:46

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Dilbert As gently caress posted:

https://www.youtube.com/watch?v=Ntf0SBmWL1U

I did a video GUG on Home lab designs and how to set one up if you are interested.

I'd avoid things like Dell refurbs, I don't mind helping put a whitebox together

This is so completely not what I expected your voice to sound like

# ? Jan 31, 2014 01:42

Dilbert As FUCK: Sep 8, 2007; by Cowcaster; Pillbug

Misogynist posted:

This is so completely not what I expected your voice to sound like

Not sure if that's good or bad but okay?

# ? Jan 31, 2014 02:27

Mierdaan: Sep 14, 2004; Pillbug

Dilbert As gently caress posted:

Not sure if that's good or bad but okay?

I thought the same thing, you sound... normal.

# ? Jan 31, 2014 02:57

Dr. Arbitrary: Mar 15, 2006; Bleak Gremlin

Dilbert As gently caress posted:

Not sure if that's good or bad but okay?

It's good. You have a good, normal voice. I understood you just fine and you don't sound like a dork.

Many people are self conscious about their voice, you shouldn't worry about it.

# ? Jan 31, 2014 03:14

three: Aug 9, 2007; i fantasize about ndamukong suh licking my doodoo hole

Dilbert is normal and pretty cool in person. V :shobon:

# ? Jan 31, 2014 03:28

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Dilbert As gently caress posted:

Not sure if that's good or bad but okay?

I have no idea why but I always had this picture in my head of you sounding like Milhouse from The Simpsons.

# ? Jan 31, 2014 03:50

evil_bunnY: Apr 2, 2003

Misogynist posted:

I have no idea why but I always had this picture in my head of you sounding like Milhouse from The Simpsons.

FYI you're a meanie.

# ? Jan 31, 2014 11:28

HalloKitty: Sep 30, 2005; Adjust the bass and let the Alpine blast

evol262 posted:

don't buy 1U gear for home.

I did, it's very space efficient since I run it sideways. Absolutely minuscule footprint!

Of course, it's noisy as gently caress.

# ? Jan 31, 2014 14:24

Count Thrashula: Jun 1, 2003; Death is nothing compared to vindication.; Buglord

Yeah, don't buy 1U gear for the home.

Only buy 2/4U gear for the home :getin:

no... it's true though, I have a couple lovely 1U servers that I bought for around $20 a pop and they sound like drat vacuum cleaners so I never want to run them. Since I managed to get my ACS 5.4 and Virtual WLC both working in VMware Workstation, I haven't even touched those two units.

# ? Jan 31, 2014 16:58

some kinda jackal: Feb 25, 2003; �
�

Troll craigslist for a desktop SuperMicro case that converts to rack format. You can stick silent or near silent fans in that thing and you'll be so much happier.

I picked up two super cheap and they're the best buys I ever made. I can keep swapping out newer motherboards in there, and it's got like a 16 drive hotswap bay. Of course they're all for full-height drives so I'm looking for adapters to put SSD drives in there. All the normal adapters I've seen put the SATA ports in the wrong place though :[

VM question:

I will likely need to do a P2V on two NT4 servers running SQL server for a client. I did a P2V conversion on NT4 before by just DD'ing the physical drives and running the raw file through a converter. I'll be damned if I remember how I fixed the INACCESSIBLE_BOOT_DEVICE error though. I seem to remember I had to boot with an alternate OS and replace some DLL drivers or something. Google has a wide range of options but I'd like to narrow down to what you guys think the safest way to do a P2V on an NT4 machine would be. I gather the target environment is ESXi 4.1 still, but I doubt that will make too much of a difference.

# ? Jan 31, 2014 17:13

Dilbert As FUCK: Sep 8, 2007; by Cowcaster; Pillbug

Martytoof posted:

VM question:

I will likely need to do a P2V on two NT4 servers running SQL server for a client. I did a P2V conversion on NT4 before by just DD'ing the physical drives and running the raw file through a converter. I'll be damned if I remember how I fixed the INACCESSIBLE_BOOT_DEVICE error though. I seem to remember I had to boot with an alternate OS and replace some DLL drivers or something. Google has a wide range of options but I'd like to narrow down to what you guys think the safest way to do a P2V on an NT4 machine would be. I gather the target environment is ESXi 4.1 still, but I doubt that will make too much of a difference.

Try changing the virtual SCSI adapter to the other options available.

# ? Jan 31, 2014 17:55

thebigcow: Jan 3, 2001; Bully!

Try changing it to an IDE disk.

The last time I had to do this for Win2k I used Clonezilla to copy the drive, it understands NTFS and won't copy blank space.

# ? Jan 31, 2014 18:55

Kachunkachunk: Jun 6, 2011

skipdogg posted:

Well I thought it was going to be a slow day at work, but a different department is having issues with their ESX 4 cluster. One of the hosts just shows up disconnected. Network connectivity and all the normal troubleshooting seems fine, storage is fine, but I can't even login to the local console on the physical host to try to restart the management service or anything. It's just completely hung.

It's not really my problem as it's not my cluster, so the guy that manages is calling support. 2 out of the 3 VM's on it are running without issue. I have a feeling support is just going to tell him to shut those VM's down and reboot the host, but they just happen to be running 2 very high visibility critical apps they use and there is going to be a poo poo ton of questions about why and how this happened and what he's going to do to prevent it in the future. Like Spanish Inquisition level poo poo is going to go down. I feel bad for the guy. Maybe support can dig through the logs and figure out whats up. OOB management of the physical host shows no hardware issues either.

Is this ESXi or ESX?
I presume local tech support mode (under Alt+F1) was disabled (per default)? If it was enabled, did it stop after attempting to input a username?
Have them try Alt+F12 - this will show them what the vmkernel is logging. And whatever they do, ensure this is done before rebooting. Take screenshots, etc. and supply that to Support.
Better: if the host was pre-configured to panic upon NMI reception, NMI the host when everyone is ready for downtime (you may shut down the VMs before-hand via the network (RDP, SSH)).

I just wrapped up an on-site with a customer about this very issue.
- The DCUI doesn't respond to F2 to log in. Alt+F1/F12 responds fine.
- Local tech support mode may or may not work (customer had it disabled, which is default).
- SSH is disabled by default as well, but even for one system that did have it enabled, it didn't proceed to authenticate/login.
- Storage/Networking looked fine.
- VMs run fine.

We don't have root-cause yet, but I figure it must have to do with a socket leak for vmware-hostd or some other non-critical component. Either way, it's difficult to instrument and this is going to take some time/effort.
It also has occurred for a bunch of systems at different times. Different configurations, etc. Fairly long uptimes of about 80+ days. And never twice on the same system.
I also suspect perhaps security scans or vulnerability testing could induce this behavior in some environments, but it'd probably have induced this behavior on way more systems at once.

Very elusive. And perhaps another reason get off of ESX/ESXi 4.x, really.

# ? Feb 2, 2014 01:16

whaam: Mar 18, 2008

Anyone see an issue in doing this to avoid buying a second vcenter license:

Two offices, one main, one DR site.

Netapp storage at both sites, replicated between.

vCenter with 2 clusters (6 hosts) at main office. 4 more hosts in their own cluster that are physically at the DR site.

Is it "Ok" to add those 4 hosts to the primary vCenter datacenter? They are connected over a 100Mb Lan extension. The vCenter guest lives on the primary hosts but in the event of primary loss, could be spun up on the remote site via direct connect to one of the dr hosts.

Not ideal but budget getting tight and the second license could be a problem.

Also we are near capacity on our vRAM licensing on our 6 primary hosts, I take it adding those other 4 hosts will give us their 8 cpus worth of additional vRAM allotments for the entire datacenter?

whaam fucked around with this message at 18:55 on Feb 4, 2014

# ? Feb 4, 2014 18:17

skipdogg: Nov 29, 2004; Resident SRT-4 Expert

Kachunkachunk posted:

Is this ESXi or ESX?
I presume local tech support mode (under Alt+F1) was disabled (per default)? If it was enabled, did it stop after attempting to input a username?
Have them try Alt+F12 - this will show them what the vmkernel is logging. And whatever they do, ensure this is done before rebooting. Take screenshots, etc. and supply that to Support.
Better: if the host was pre-configured to panic upon NMI reception, NMI the host when everyone is ready for downtime (you may shut down the VMs before-hand via the network (RDP, SSH)).

I just wrapped up an on-site with a customer about this very issue.
- The DCUI doesn't respond to F2 to log in. Alt+F1/F12 responds fine.
- Local tech support mode may or may not work (customer had it disabled, which is default).
- SSH is disabled by default as well, but even for one system that did have it enabled, it didn't proceed to authenticate/login.
- Storage/Networking looked fine.
- VMs run fine.

We don't have root-cause yet, but I figure it must have to do with a socket leak for vmware-hostd or some other non-critical component. Either way, it's difficult to instrument and this is going to take some time/effort.
It also has occurred for a bunch of systems at different times. Different configurations, etc. Fairly long uptimes of about 80+ days. And never twice on the same system.
I also suspect perhaps security scans or vulnerability testing could induce this behavior in some environments, but it'd probably have induced this behavior on way more systems at once.

Very elusive. And perhaps another reason get off of ESX/ESXi 4.x, really.

Interesting post. I haven't followed up with my co-worker on it. We do have weekly vuln scanning in place though. He's working on upgrading to the latest 5.1 release anyway

# ? Feb 4, 2014 18:40

Nukelear v.2: Jun 25, 2004; My optional title text

skipdogg posted:

Interesting post. I haven't followed up with my co-worker on it. We do have weekly vuln scanning in place though. He's working on upgrading to the latest 5.1 release anyway

You don't happen to be using TrendMicro DeepSecurity or some other host level av? Your issue sounds almost exactly like what we had and ended up root causing it to that.

# ? Feb 4, 2014 18:52

skipdogg: Nov 29, 2004; Resident SRT-4 Expert

Nukelear v.2 posted:

You don't happen to be using TrendMicro DeepSecurity or some other host level av? Your issue sounds almost exactly like what we had and ended up root causing it to that.

Nope.

edit: Talked to my co-worker. One of the RAID-1 drives failed in the array holding the ESXi software, so maybe that was it. He's also diagnosing some other possible hardware issue with the server.

skipdogg fucked around with this message at 19:41 on Feb 4, 2014

# ? Feb 4, 2014 19:28

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

gently caress. I go to create a new 64-bit Windows 2008 VM on our ESX server at the disaster recovery location (remotely), and I see this:

# ? Feb 4, 2014 20:42

Pile Of Garbage: May 28, 2007

Exactly what it says: make sure Intel VT-x and NX-bit is enabled in the BIOS of all ESXi hosts in the cluster.

# ? Feb 4, 2014 20:47

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

cheese-cube posted:

Exactly what it says: make sure Intel VT-x and NX-bit is enabled in the BIOS of all ESXi hosts in the cluster.

Why the heck would you not enable VT-x on a goddamn VMware server? :argh:

This thing has been in production 3 years!

iLO isn't configured and I asked if the KVM down there was an IP KVM, and my boss doesn't know what one of those is. Durr.

# ? Feb 4, 2014 20:54

Pile Of Garbage: May 28, 2007

Bob Morales posted:

Why the heck would you not enable VT-x on a goddamn VMware server? This thing has been in production 3 years!

iLO isn't configured and I asked if the KVM down there was an IP KVM, and my boss doesn't know what one of those is. Durr.

Welcome to my world: projects being implemented with configurations to meet the bare-minimum required functionality while ignoring important and sometimes crucial considerations

# ? Feb 4, 2014 21:05

evol262: Nov 30, 2010; #!/usr/bin/perl

Bob Morales posted:

Why the heck would you not enable VT-x on a goddamn VMware server? This thing has been in production 3 years!

iLO isn't configured and I asked if the KVM down there was an IP KVM, and my boss doesn't know what one of those is. Durr.

# ? Feb 4, 2014 21:07

Erwin: Feb 17, 2006

Enjoy your road trip.

# ? Feb 4, 2014 21:09

Count Thrashula: Jun 1, 2003; Death is nothing compared to vindication.; Buglord

That kind of project management is why I'm flying up the coast next week.

"We implemented some wireless access points and segmented the network, but things aren't working right. Trunking? No, what's that? Oh the switches? They're Dell PowerConnect something-or-others..."

SIGH.

edit-- whoops, just realized this isn't the bitching thread. They all start to blend together...

# ? Feb 4, 2014 21:14

Pile Of Garbage: May 28, 2007

QPZIL posted:

That kind of project management is why I'm flying up the coast next week.

"We implemented some wireless access points and segmented the network, but things aren't working right. Trunking? No, what's that? Oh the switches? They're Dell PowerConnect something-or-others..."

SIGH.

edit-- whoops, just realized this isn't the bitching thread. They all start to blend together...

Just merge the bitching into general discussion. For example: I'm currently on the rear end-end of a project where we are deploying two 3-host clusters in South Korea. So far things have been an absolute nightmare due to the over-engineered networking configuration but we are getting there.

Anyway, these clusters are managed by our vCenter server back at our DC in Australia and the best-case round-trip-time between SK and AU is ~100ms so I was wondering are there any specific considerations for vCenter in a situation like this?

# ? Feb 4, 2014 21:23

Mausi: Apr 11, 2006

Bob Morales posted:

Why the heck would you not enable VT-x on a goddamn VMware server?

Depending on the server Vendor and Bios version, they'll ship with it disabled. Sometimes a Bios upgrade will reset the configuration. Sometimes the chucklehead who swapped out the faulty mainboard didn't put the standard bios settings back. Can't think of any others at the moment.

# ? Feb 4, 2014 23:59

Bhodi: Dec 9, 2007; Oh, it's just a cat.; Pillbug

You guys laugh but we got that exact same error recently. I think it has to do with the default install of older versions of not enabling it by default, and later versions requiring it to migrate hosts. You'd never see that error unless you install more blades or perform an upgrade.

# ? Feb 5, 2014 23:33

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Bhodi posted:

You guys laugh but we got that exact same error recently. I think it has to do with the default install of older versions of not enabling it by default, and later versions requiring it to migrate hosts. You'd never see that error unless you install more blades or perform an upgrade.

I know when I installed 5.5 on a few test boxes a couple weeks back, it brought up the error during installation.

And of course, my test boxes were our old servers that are the same as what we have in production.

# ? Feb 5, 2014 23:40

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Bob Morales posted:

I have a 3-node cluster running 3.5 or 4.0 or something and it needs to be upgraded to 5.5 - Is there some kind of guide as to how about going through with this? Our consultant wants 25 hours of time to do it, and I've only upgraded single hosts and never a whole cluster.

I ended up doing a migration to vCenter 5.0u3 and fixing all the HA errors that came up. Now I have to actually upgrade the hosts when I can schedule some downtime, maybe either one night this week, or tomorrow. It's been fairly painless so far, just some trial and error with the backup/restore configuration progress and installing new vCenter VM's (32 to 64-bit). Saving $3,500 and learning quite a bit as I go along.

# ? Feb 5, 2014 23:42

evil_bunnY: Apr 2, 2003

Bob Morales posted:

Why the heck would you not enable VT-x on a goddamn VMware server? This thing has been in production 3 years!

iLO isn't configured and I asked if the KVM down there was an IP KVM, and my boss doesn't know what one of those is. Durr.

Sounds like my place before I showed up *D

# ? Feb 6, 2014 00:16

Daylen Drazzi: Mar 10, 2007; Why do I root for Notre Dame? Because I like pain, and disappointment, and anguish. Notre Dame Football has destroyed more dreams than the Irish Potato Famine, and that is the kind of suffering I can get behind.

I'm very curious if my unit is going to upgrade their ESXi and vCenter installs. I think the Virtualization guys are of the opinion "if it ain't broke yet, don't try and fix (or upgrade) it". I have a feeling I'll be getting some experience in updating ESXi hosts sometime soon (we have a couple hundred).

# ? Feb 6, 2014 01:37

Dr. Arbitrary: Mar 15, 2006; Bleak Gremlin

Anyone have any tips on studying for the VCP-DCV?

I've got the Scott Lowe book and I'm in a class so I've got the basics, but I can't fail this test.

# ? Feb 6, 2014 03:42

Dilbert As FUCK: Sep 8, 2007; by Cowcaster; Pillbug

Dr. Arbitrary posted:

Anyone have any tips on studying for the VCP-DCV?

I've got the Scott Lowe book and I'm in a class so I've got the basics, but I can't fail this test.

read the questions and answers twice.
Take your time
Chances are you're first guess is probably right
http://mylearn.vmware.com/mgrReg/plan.cfm?plan=44097&ui=www_cert nice practice test
Go over the blueprint http://mylearn.vmware.com/mgrReg/plan.cfm?plan=45082&ui=www_cert
Hersey and I kinda put some resources together http://www.vhersey.com/vcp5-study/

Are you taking the 5.5 or 5.1?

If a few more people want I can do a gug on Saturday, along with my storage one tomorrow, of VCP 5.1 and just do exam blueprint review.

# ? Feb 6, 2014 04:37