Virtualization Megathread V2: VMs inside VMs

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Virtualization Megathread V2: VMs inside VMs

SamDabbers: May 26, 2003

Tatsuta Age posted:

Yeah, I'm struggling to find a reliable list that goes into more detail than "this motherboard vendor with this chipset type SHOULD work, but really who knows!"

The IOMMU stuff is more to pass through the graphics card to make the server run my VM as close to a native desktop would as possible. Then, in my mind, I can run a management tool from that desktop to handle the VMs as necessary.

One option would be to run Hyper-V on Windows 10 Pro. It now supports RemoteFX GPU virtualization (DirectX 11 only), so you can game and such as usual on the "Host" OS while running (potentially) 3D accelerated guests. Then you don't have to worry about your motherboard fully supporting VT-d for passing through the GPU, while still getting to game at native speed and tinker with VMs.

# ¿ Feb 18, 2016 02:59

Adbot: ADBOT LOVES YOU

# ¿ May 5, 2024 10:50

SamDabbers: May 26, 2003

priznat posted:

Just more reason to ditch uart reliance on our boards and go with ethernet socket interfaces uart can be the last ditch everything is broken and we are debugging in the lab fallback only!

If the only USB devices you want to pass through are UART adapters, maybe a console server would suit your requirements. Then your users can SSH to it from within their Hyper-V VMs and you won't have to buy VMware licenses.

# ¿ Jul 7, 2016 13:16

SamDabbers: May 26, 2003

Is your case big enough to hold an SSI-EEB board? Transcode all the things like it's going out of style, and plenty of RAM for ZFS...

# ¿ Sep 9, 2016 16:44

SamDabbers: May 26, 2003

That config idles at about 90W, and it's way more than twice the machine that single socket E3 is, so you'd really be coming out ahead on energy efficiency vs running two E3 boxes. That's an extra 1.2KWh per day (50W * 24H / 1000) and at $0.10/KWh that amounts to an extra $44/year to run it. It's insignificant if you aren't running a datacenter full of them.

# ¿ Sep 9, 2016 17:52

SamDabbers: May 26, 2003

Quoting myself:

SamDabbers posted:

Is your case big enough to hold an SSI-EEB board? Transcode all the things like it's going out of style, and plenty of RAM for ZFS...

Intel S2600CP2J motherboard, dual E5-2670s, and 128GB ECC RAM for $500.

Edit: I recently built this exact machine, and it's a beast for how little it cost.

Intel case w/power supply designed for this board (the seller took a best offer of $100)
Passive heatsinks x2
Air duct kit required for this board/case/heatsink combination
I/O Shield
8x3.5" Hot Swap backplane including drive trays (the seller took a best offer of $40)
Case bezel with a door for the hot swap bays (the seller took a best offer of $15)

SamDabbers fucked around with this message at 21:46 on Sep 11, 2016

# ¿ Sep 11, 2016 21:13

SamDabbers: May 26, 2003

The extent of said trickery for libvirt/KVM with a Windows guest:

code:

  <features>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='0123456789ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>

# ¿ Sep 29, 2016 16:12

SamDabbers: May 26, 2003

The GeForce driver doesn't start if it detects a known hypervisor. Hiding KVM is necessary, but you can use the Hyper-V paravirtualizations if you change the vendor ID.

# ¿ Sep 29, 2016 17:05

SamDabbers: May 26, 2003

Paul MaudDib posted:

Is there a container system with a bit more of a reasonable security model? FreeBSD doesn't seem like they'd do that bullshit, do jails work reasonably well? How about LXC?

FreeBSD Jails were explicitly designed as a security boundary to confine root. They are mature and work well, and with FreeBSD 11 the VIMAGE network stack virtualization has most of its bugs (memory leaks/instability at teardown) ironed out. Admin tools like iocage and warden (both in Ports) are the way to go for provisioning jails with ZFS.

Linux namespaces were not designed as a security boundary, so your namespace-using toolset (Docker, LXC, systemd-nspawn) is responsible for locking down your container with SELinux/AppArmor et al. LXD purports to do this, building on top of LXC, and it includes libvirt and OpenStack bindings for automation.

Illumos Zones are another secure container tech to check out in SmartOS. Like Jails, they were designed as a security boundary. SmartOS has first-rate Linux binary emulation, so you can use the familiar userland of your distro of choice with the Illumos kernel and all its services underneath.

# ¿ Apr 26, 2017 03:44

SamDabbers: May 26, 2003

SEKCobra posted:

I'm giving up on virtualizing my old PC, there just isn't a solution that will let me use the shitton of HDDs in there as well as have windows VMs that can use USB devices exclusively.

Passing through HDDs and USB devices is straightforward and reasonably performant in pretty much every modern hypervisor. :confused:

# ¿ Jul 12, 2017 20:51

SamDabbers: May 26, 2003

It seems like bare-metal Linux with KVM can meet your NAS and hypervisor requirements. Pass through your GPU and a USB controller for your desktop VM and you'll be set. Right tool for the job and whatnot.

# ¿ Jul 13, 2017 20:32

SamDabbers: May 26, 2003

evol262 posted:

I think my facetiousness was missed, but kvm works fine on Solaris if you really hate systemd for whatever reason.

KVM works great on Illumos, but it doesn't support PCI passthrough yet. The Joyent folks don't need it, so they didn't implement it.

# ¿ Jul 14, 2017 04:20

SamDabbers: May 26, 2003

I also do something similar, only with vanilla FreeBSD + Jails + bhyve. Just about everything I run has a Port and works fine in a Jail, and I use bhyve VMs for the few services that don't.

OS containers > hardware virtualization

SamDabbers fucked around with this message at 18:35 on Feb 2, 2018

# ¿ Feb 2, 2018 18:33

SamDabbers: May 26, 2003

I have a Lenovo TS440 with an E3-1225 v3 (Haswell) that I've been using as an all-in-one NAS and hypervisor for a few years, and it's been great. It's very quiet and idles at less than 100W with 8 spinners in it, and it has sufficient horsepower for transcoding or compilation. You can probably pick them up for a decent price on eBay these days, and fill it with cheap(er) DDR3 UDIMMs.

# ¿ Feb 5, 2018 18:06

SamDabbers: May 26, 2003

Mr Shiny Pants posted:

I have the 1245 version of this machine and it really is awesome, the only downside is the 32GB RAM.

It depends what you're doing, of course. I've not run into memory pressure with 32GB RAM, but most of the stuff I do runs in bare-metal containers on the host OS and I only have a couple full-fat VMs for e.g. Windows stuff.

The best part is that the case can hold up to an EATX/SSI-EEB board and a regular ATX power supply, so you have an upgrade path that retains the nice hotswap chassis when you need more RAM, CPU, or PCIe lanes.

# ¿ Feb 6, 2018 17:07

SamDabbers: May 26, 2003

Boris Galerkin posted:

Thanks guys. Looks like I'm sticking to a Linux host and Windows guest, but might look into VMWare Workstation. Does the free version work better out of the box? Again, all I need Windows for is the Office suite. Right now I have Windows 10 in Window mode in VirtualBox, with the Windows task bar hidden and Excel/PowerPoint running as full screen apps inside the VirtualBox window, so it's kinda got the a native app feel going for it.

In a similar vein, I don't need to run Windows 10. Would Windows 8.1 or even 7 be better if all I need it for is just Excel and PowerPoint?

I use KVM with virt-manager as the UI and find it needs-suiting for desktop virtualization. It gives a VirtualBox/VMware-like experience, works well, and is extremely feature rich.

There are accelerated graphics drivers for Windows 7 on QEMU/KVM so that's probably the way to go on that stack if you don't want to use RDP. Microsoft removed the display driver model the accelerated drivers were using in Windows 8 and up, so RDP ends up being a better experience with those versions as the guest.

Semi-serious edit: Depending on which version of MS Office you're using, WINE might be an actual option for you. It's one of the better supported apps.

SamDabbers fucked around with this message at 15:30 on Feb 15, 2018

# ¿ Feb 15, 2018 15:21

SamDabbers: May 26, 2003

Yeah not enough ROI on the cost of the lawyers and such.

# ¿ Apr 1, 2018 20:28

SamDabbers: May 26, 2003

Mr Shiny Pants posted:

Regular KVM with Virtual Machine manager also works pretty well.

Yes, this is pretty much ideal for a single node.

# ¿ Jun 19, 2018 14:36

SamDabbers: May 26, 2003

Paul MaudDib posted:

I realize this is a completely nuts setup but I have a loaded dual-processor Xserve G4 that I got for $10 like 5 years ago, is there any way to virtualize MacOS 7/8/9 environments on it with a (native/passthrough) G4 processor?

SheepShaver is probably your best bet for virtualizing classic MacOS if the Classic environment in OS X prior to 10.5 isn't doing it for you.

# ¿ Sep 6, 2019 12:00

SamDabbers: May 26, 2003

What about FreeNAS jails makes you feel less safe than Proxmox?

# ¿ Jul 9, 2020 17:14

SamDabbers: May 26, 2003

Is it safe to assume you need strict VT100 emulation, not "close enough" VT100-ish? Also, how does SmarTerm connect to this database system? Telnet?

# ¿ Feb 16, 2021 23:11

SamDabbers: May 26, 2003

CommieGIR posted:

SecureCRT maybe? Putty supports setting vt100 emulation for Telnet: term=xterm or set term=vt100

That just sets what PuTTY reports itself as, but doesn't actually change the way it emulates a terminal. If you need strict VT100 behavior then you need software that advertises that.

# ¿ Feb 17, 2021 00:55

SamDabbers: May 26, 2003

BlankSystemDaemon posted:

Gaming in Windows on ESXi was a fun adventure, but it doesn't really work out.

Try KVM with a relatively recent kernel and qemu. I have had good results passing through an Nvidia card and USB3 controller to a Windows VM for this use case under Fedora.

# ¿ Apr 4, 2021 04:06

SamDabbers: May 26, 2003

Less Fat Luke posted:

LOL you're no help. I guess I'll make a bootable ESXi drive and just boot that host, and see if the PCI card shows up as available to passthrough.

Old school PCI doesn't support IOMMU IIRC so you'll probably have to pass through everything that's on the same PCI bus, and it may only work if the PCIe-to-PCI bridge is wired into a PCIe port that supports ACS.

# ¿ Apr 30, 2021 16:34

SamDabbers: May 26, 2003

I've never used it before but maybe something like Aster would do what you're looking for:

https://www.ibik.ru/

# ¿ Jul 16, 2021 17:55

SamDabbers: May 26, 2003

Seems fine if you like being findom'd by Oracle

# ¿ Jun 27, 2022 13:42

SamDabbers: May 26, 2003

Semi-related: is there any benefit for east-west traffic between VMs on the same box to use SR-IOV and let the NIC switch the packets instead of the CPU? Seems like there'd be less CPU load at the expense of a round trip over PCIe, and would also be dependent on the internal switching capacity of the particular NIC.

Has anybody tried this?

# ¿ Apr 17, 2023 20:19

SamDabbers: May 26, 2003

BlankSystemDaemon posted:

It'd be nice if folks learned the lesson that having any one solution is the wrong way, and instead some focus on working on kvm, while others improve bhyve (found in FreeBSD and Illumos distributions, among others) while still others work on Xen.

With enough work, and someone working on interoperability, it'd be possible to have a fleet of three (or more?) hypervisor solutions, all being able to work together.

KVM, bhyve, and Xen are all actively developed though? The closest thing we have for a unified management plane is libvirt. There's also virtio for PV devices but only KVM and bhyve implement it AFAIK. Xen and VMware have little incentive to adopt it unfortunately.

# ¿ Dec 17, 2023 16:09

SamDabbers: May 26, 2003

BlankSystemDaemon posted:

Is there any virtualization on x86 or its derivatives, other than that of VMware, that doesn't use hardware-accelerated virtualization with SLAT (aka AMD Vi/Intel VT-x)?
I know VMware can also use it, but their original software did it all in software and somehow managed very low overhead.

EDIT: Oh, right - XenServer is a thing. I forgot
orz

VirtualBox runs 32 bit guests without hardware extensions

# ¿ Jan 29, 2024 19:35

Adbot: ADBOT LOVES YOU

# ¿ May 5, 2024 10:50

SamDabbers: May 26, 2003

Anybody considering Red Hat OpenShift Virtualization (or kubevirt on some other flavor of k8s) as a VMware alternative? Seems worth kicking the tires especially if your org is already running clusters.

# ¿ Apr 14, 2024 14:45

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Virtualization Megathread V2: VMs inside VMs