|
Internet Explorer posted:Sometimes you just read posts and cringe. That was one of those posts. I worked staff-aug for a company that ran Arcsight SIEM. They deployed it to a beefy server but somehow managed to skimp on storage. To make matters worse, they used like 90% of the datastore for the SIEM VM but thin provisioned it. Anyway, so Veeam -- You know what, I can't even finish this with a straight face. Let's just say that the story ends with me getting a frantic call at 2am because the SIEM VM won't power on because the datastore is out of space. Thank christ I don't do IT for a living anymore.
|
#
?
Apr 7, 2016 14:45
|
|
|
|
| # ? May 8, 2024 04:50 |
|
|
devmd01 posted:It's not, but the slowness of our current san makes that process more painful with the additional I/O. I've always been much happier with fiber channel. Brand new SAN with SSD tier in July, wooo!
|
|
#
?
Apr 7, 2016 15:11
|
|
|
VMware really ought to prompt you to set a max life for snapshots when they are created with a default value like 24 hours so they also-prune if you forget.Vulture Culture posted:you know that iSCSI supports MPIO too right And NFSv4 if you're feeling adventurous.
|
|
#
?
Apr 7, 2016 16:08
|
|
|
pNFS is nowhere near the caliber of MPIO last time I played with it, but it's been about a year. I'm sure the new SAN will help when the storage admins don't bother to configure multiple pathing, because bonds are better somehow.
|
|
#
?
Apr 7, 2016 16:41
|
|
|
devmd01 posted:god dammit boss. I don't mean to be a topper but 
|
|
#
?
Apr 7, 2016 16:56
|
|
|
evol262 posted:I'm sure the new SAN will help when the storage admins don't bother to configure multiple pathing, because bonds are better somehow. Stop talking about the inadequacies of my storage infrastructure  evol262 posted:pNFS I'm a grown adult. Why am I giggling uncontrollably when I read this aloud.
|
|
#
?
Apr 7, 2016 16:57
|
|
|
devmd01 posted:It's not, but the slowness of our current san makes that process more painful with the additional I/O. I've always been much happier with fiber channel. Brand new SAN with SSD tier in July, wooo!
|
|
#
?
Apr 7, 2016 17:08
|
|
|
Martytoof posted:They deployed it to a beefy server but somehow managed to skimp on storage. Storage always seems to be what gets skimped on. "Our business-critical data and apps are sitting on a bunch of 5400 RPM consumer SATA drives in a whitebox tower. But hey, we were able to afford another 0.1 Ghz on each VM host's CPUs! That'll make up for any performance issues"  Potato Salad posted:I'm a grown adult. Why am I giggling uncontrollably when I read this aloud. God dammit, I am never going to be able to read/say this as anything but "peen FS" from now on. Thanks a lot, buddy 
|
|
#
?
Apr 7, 2016 17:21
|
|
|
adorai posted:Somehow I doubt FC would have made this any better. It absolutely wouldn't. Your storage sucks, doesn't matter what you use to connect to it.
|
|
#
?
Apr 7, 2016 17:22
|
|
|
Docjowles posted:God dammit, I am never going to be able to read/say this as anything but "peen FS" from now on. Thanks a lot, buddy I was doing "penifs" but now I can't even use the acronym. Thanks, right back at ya
|
|
#
?
Apr 7, 2016 17:32
|
|
|
BangersInMyKnickers posted:VMware really ought to prompt you to set a max life for snapshots when they are created with a default value like 24 hours so they also-prune if you forget. This might cause some unpleasant interactions with VADP based products like VEEAM though. There are alarm triggers snapshot size that can provide warning before they grow too big, which should be sufficient. Or they could just re-write VMFS to not use an incredibly lovely snapshot mechanism.
|
|
#
?
Apr 7, 2016 17:40
|
|
|
Anyone get system lockups when shutting down their Windows guest with VGA passthrough? I've been running into it sporadically, but I don't see any lines in the kernel logs which would point to the problem, so I'm a little stumped as to the problem. Kernel is 4.4.3-300.fc23.x86_64, qemu is 2.5.0.
|
|
#
?
Apr 7, 2016 18:35
|
|
|
What a coincidence. Guess who decided to clean up vCenter and found that our critical company DB is running on a snapshot from May 2015? 
|
|
#
?
Apr 7, 2016 18:46
|
|
|
cliffy posted:Anyone get system lockups when shutting down their Windows guest with VGA passthrough? I've been running into it sporadically, but I don't see any lines in the kernel logs which would point to the problem, so I'm a little stumped as to the problem. For some reason, this changes something in the shutdown procedure and gets KVM all up in a tizzy. Took me a while to track it down. May also apply to Windows 8.x, IDK.
|
|
#
?
Apr 7, 2016 19:06
|
|
|
Our team doesn't admin the majority of the guest OSes running in our environment, but we do give snapshot ability out to dozens of teams that do. One of the biggest quality of life improvements we have made is a powershell script that finds VMs with snaps older than X days or Y GB in size and emails a nag to the owner to deal with it with a link to the best practices kb where VMware recommends to keep snaps no longer than 1-3 days. Once those hit a certain threshold, it will instead send off a ticket to us with wording about possible VM downtime if they don't address it. We then follow up to make sure it gets taken care of. Most of the time it was honestly just forgotten about and the nag alone is enough... It is fairly rare we see a ticket these days and when we do it is almost always immediately responded to, even before we get a chance to inquire about it.
|
|
#
?
Apr 7, 2016 19:07
|
|
|
Yeah, I just set up alerts in vSphere so at least now I'll be aware of what machines are on snaps and for how long. Oof I shudder to think how long a consolidation would take. Current plan is to clone it.
|
|
#
?
Apr 7, 2016 19:46
|
|
|
Combat Pretzel posted:If it's Windows 10, go into Power Options -> Choose what power buttons do, and then disable Fast Startup. Fast startup does some hibernation stuff with drivers so yeah it can get confused because of that.
|
|
#
?
Apr 7, 2016 19:58
|
|
|
Is there a good way to identify old orphaned VMDK's in a datastore? I've had a lot of people working in my environment over the years and I'm guessing there are a couple old disk files hanging around that were dismounted but not deleted because someone forgot.
|
|
#
?
Apr 7, 2016 21:39
|
|
|
BangersInMyKnickers posted:Is there a good way to identify old orphaned VMDK's in a datastore? I've had a lot of people working in my environment over the years and I'm guessing there are a couple old disk files hanging around that were dismounted but not deleted because someone forgot. RVtools does this.
|
|
#
?
Apr 7, 2016 22:53
|
|
|
NippleFloss posted:RVtools does this. This has been my goto tool for random stuff like this for years.
|
|
#
?
Apr 8, 2016 17:20
|
|
|
NippleFloss posted:RVtools does this. Thank you for introducing me to my new favorite thing
|
|
#
?
Apr 8, 2016 18:18
|
|
|
I've having trouble upgrading a vcenter appliance from 5.5 to 6.0. Everything goes fine at the start, but it eventually bombs out with "fatal error during network configuration" when the status bar is not quite to halfway. I found some posts in the vmware forums saying to make sure your DNS records and NTP settings are good for the old appliance, and mine appear to be fine. Anyone else experience this and have any suggestions on how to get past it?
|
|
#
?
Apr 14, 2016 20:16
|
|
|
Use Hyper-V
|
|
#
?
Apr 14, 2016 20:33
|
|
|
stubblyhead posted:I've having trouble upgrading a vcenter appliance from 5.5 to 6.0. Everything goes fine at the start, but it eventually bombs out with "fatal error during network configuration" when the status bar is not quite to halfway. I found some posts in the vmware forums saying to make sure your DNS records and NTP settings are good for the old appliance, and mine appear to be fine. Anyone else experience this and have any suggestions on how to get past it? Are the PTR records correct?
|
|
#
?
Apr 14, 2016 22:41
|
|
|
mayodreams posted:Are the PTR records correct? Yep, A and PTR both in place and accurate. I've been loving around with it most of today, and it appears that it didn't like the fact that the old server was using a DHCP address. I changed the network config to static, and it worked perfectly the first time. I don't have a good explanation for it, but it's working now.
|
|
#
?
Apr 14, 2016 23:41
|
|
|
Our team got to demo a VxRail. Three of these could replace 2+ racks of ours, it's bananas. Only thing is that they don't support GPU on VDI yet (SolidWorks at 30fps through a web browser really is something). http://www.vce.com/products/hyper-converged/vxrail
|
|
#
?
Apr 21, 2016 16:00
|
|
|
So I have a problem that I'm trying to solve and I am sort of running my head into a wall. I have a client running a ESXi 4.1 environment, and I need to set the VMs in it to save their snapshots to a new NAS. I've attached the NAS as a datastore, but I cannot for the life of me figure out how VMware wants me to point to it in the vmx file. I know I need to use the workingDir option, but I'm sorta lost on how to tell it to find this datastore. The NAS is at, let's say 10.0.0.2, and the folder on the NAS is /mnt/array1/backup. The datastore is identified in vSphere as "NAS", and contains only a single folder inside, called "snapshots". I've "netfs://10.0.0.2//mnt/array1/backup", "[NAS]", "[NAS] snapshots/", etc, and can't seem to figure out what the actual correct way of pointing at this thing is. I'm sure it's a simple issue of me being awful at *nix systems, but I am totally at a loss here. Any thoughts? edit: For a little more info, if I right click on the VM in vSphere and choose Edit Settings and go to the options tab, it looks like it thinks the working directory is being pointed inside the original datastore. ie, "[ds2-raid5] systemname/netfs:/10.0.0.2/mnt/array1/backup" or "[ds2-raid5] systemname/[NAS] snapshots/" Lights fucked around with this message at 02:06 on Apr 22, 2016 |
|
#
?
Apr 22, 2016 01:53
|
|
|
Kaninrail posted:So I have a problem that I'm trying to solve and I am sort of running my head into a wall. You need the absolute path on the ESX host that represents that datastore. It will look like /vmfs/volumes/696f6be3-c7edb046-0d3c-0898665f415c/snapshots/ where the long UUID is a randomly generated string. One way to get it is to do an "ls -lah /vmfs/volumes" and look for the symlink of the datastore name to the UUID. That requires you to SSH into the box. It's exposed somewhere in the client too under storage or storage devices, but I don't have a 4.1 server/client to look at and tell you where. Should be visible under the datastore properties though. This is also a good time to ask *WHY* you want to do this. VMware snapshots are bad, and putting them on their own datastore makes me think that the intent is to keep them for long period of time so that they take up lots of space and then cause outages when you try to delete them.
|
|
#
?
Apr 22, 2016 02:15
|
|
|
NippleFloss posted:This is also a good time to ask *WHY* you want to do this. VMware snapshots are bad, and putting them on their own datastore makes me think that the intent is to keep them for long period of time so that they take up lots of space and then cause outages when you try to delete them. Wondering this as well. Snapshots only cause issues for most people (my coworkers included).
|
|
#
?
Apr 22, 2016 02:26
|
|
|
NippleFloss posted:You need the absolute path on the ESX host that represents that datastore. It will look like /vmfs/volumes/696f6be3-c7edb046-0d3c-0898665f415c/snapshots/ where the long UUID is a randomly generated string. One way to get it is to do an "ls -lah /vmfs/volumes" and look for the symlink of the datastore name to the UUID. That requires you to SSH into the box. It's exposed somewhere in the client too under storage or storage devices, but I don't have a 4.1 server/client to look at and tell you where. Should be visible under the datastore properties though. Long story short, they want to start using Veeam for their backups. Veeam requires that it can make a snapshot of the largest drive. The SAN they have is at capacity, with no budget for replacing it. The datastores on the SAN are both close to capacity, and set to a 2MB block size. Since you can't create a large enough snapshot at that block size, Veeam won't work. So now there's this NAS. Unfortunately, I can't SSH into the box, so that's not workable. The UI doesn't expose an absolute path for the NAS datastore, only for the vmfs datastores. Super frustrating.
|
|
#
?
Apr 22, 2016 02:33
|
|
|
Friendly reminder that putting your deltas onto another disk means that your running VMs all blow up and die when that storage goes away, so if you're planning on snapping onto discount storage, think again. If you have that disk already, Storage vMotion some of your lower-impact VMs onto it and call it a day.
|
|
#
?
Apr 22, 2016 02:35
|
|
|
Anyone have any strong opinions on which is “best” out of these options? Option 1 (58k) – HP DL360 G9 paired with a HP MSA 2040 connected directly w/ 12Gbps SAS cables. 384Gb, 48 physical cores, 2x 10gb nics and 2x SAS HBA’s per server. 14Tb usable capacity, 2x 400gb ssd’s for read cache. I guess you can pay for a license to get auto data tiering and enables write caching to SSD’s. Option 2 (66k) - probably more like 70-80k) – 3 node Nutanix solution with 24Tb raw capacity, 384Gb memory, 36 physical cores. They claim 50% usable capacity which puts us at 12Tb usable, which is cutting it too close. This was quoted at $65k and sounded like they were going to drop it further. Sounds like if I wanted to get one for Portland, I might be able to get two for around 100k. Waiting on a quote for the next bundle up w/ 32Tb raw, 8 core procs and more memory and expect it to be about 70-80k hopefully. Option 3 (64k) – HP DL360’s mentioned above but paired with a Nimble CS235 with 24Tb raw. They claim 24TB usable due to compression. I'm going to dedicate some time into reading into all of these options (was also quoted a NetApp san for around the cost of a nimble) but as of right now, it seems like both the Nutanix and HP/Nimble options are pretty compelling. Nutanix's pricing is apparently the lowest it's ever been due to their quarter ending at the end of the month, and it's the last quarter before they go public. Both sound great from a support standpoint, although Nutanix has a leg up there considering it's just one vendor. That's a huge plus for me since it's currently just me here and and getting buried alive. If you want to continue reading for more info on the environment feel free; Commence rambling: I've been hoping to replace my company's primary servers and storage for years. But due to budget constraints, a poo poo ton of major issues that needed to be ironed out as well as long overdue big projects, it’s been continually pushed back. I’m also completely overloaded and actively trying to find a helpdesk guy to shield me from being interrupted every goddamn 5 minutes. Looks like it's finally going to happen here pretty soon. The company has a little over 200 total employees and about users with 110 workstations. The Seattle office has 3 IBM x3650 servers (original, M2 and M3 versions) in production, paired with an IBM DS3300 iSCSI SAN with 7.1Tb usable capacity. 28 physical cores with 172Gb of memory. ~30VM’s and Veeam backups to a Quantum dedupe appliance (DXi4601). The Portland office has a single, ancient IBM x3550 M2 and veeam backups to another DXI. CPU isn’t a concern. We’re typically at about 25-35% utilization. Memory is 90-95% all day every day. I’m using more storage than I have on our SAN, in production, only because I’ve shifted some low-importance production servers and test machines, etc to FreeNAS storage. I have about 9tb of crap storage I can use in a pinch in each office. For the sake of background info, we recently got a 1Gbps Comcast layer 2 fiber connection to connect the two offices, along with a 50Mb fiber internet connection in Seattle and a Microwave connection that should be upgraded within the month to be capable of bursting to 1Gbps. I also got a Comcast business coax connection installed recently in Portland for web traffic and as an additional backup connection that I could turn a "RED tunnel" or IPSec tunnel up on if the fiber were to go down. Network hardware consists of Sophos UTM's at each site and HP Procurve switches. Seattle has a 5412R and Portland, a 2920 iirc. Unifi AP’s. Seattle • Web app for accounting, cost projections, AP, HR, etc. SQL based, Tomcat front-end. Max 21 concurrent users. • Another server that runs reports off of said SQL db. • Exchange 2010 w/ mailbox database that just surpassed 1Tb. That number should drop massively once I have the storage to create new mailbox db’s, to migrate mailboxes to. No quotas were ever implemented and now the database is too big to offline defrag without taking forever. • Another SQL based application server – max 10 users. • Handful of other small application servers with 5-10 users each. • DC’s, DHCP, Print, a couple file servers, WDS/MDT server, WSUS.. • RDS gateway and broker server and a separate session host server. I’m currently piloting this and holding out until we have the new hardware to open it up to all staff. If usage takes off, it could be a resource hog. Portland is a babby. They have a couple of dinky application servers, but I fully plan on consolidating those into servers in Seattle as I migrate the Seattle applications to 2012 R2. No real need to keep them segregated when we have a 1Gbps connection to Portland. Also plan on combining the Portland file server with the Seattle one which adds about 1.5Tb to the capacity needs when not taking into account any sort of compression or deduping. Basically, in the long run, there isnt’ going to be much in Portland other than supporting servers like… DHCP, a DC, DHCP, DFS copy of the Seattle file server, downstream WDS, WSUS and (maybe) Kaspersky servers and a veeam proxy server if it makes sense. Basically looking for any sort of different perspectives. The potential for implementing a similar solution (such as two nutanix setups) is there and I need to at least pitch the idea. Didn't really think it was a possibility until I had an informal 30 min chat with the CFO about where I am looking into this poo poo. Haven't really thought too much about what exactly that would get us. DR site, what else? Original plan was to move our existing IBM garbage to Portland after implementing in Seattle. ...Jesus christ. I really need to go home now. goobernoodles fucked around with this message at 02:45 on Apr 22, 2016 |
|
#
?
Apr 22, 2016 02:38
|
|
|
Buy Nutanix if you like the incremental scale out growth proposition and integrated management. Buy Nimble if you want more usable capacity before reduction so you aren't counting on dedupe and compression to get you there. Nutanix will be easier to manage. Nimble is rock solid and performs well. Either is worlds better than a direct attached HP modular array.
|
|
#
?
Apr 22, 2016 03:09
|
|
|
Nimble isn't blowing smoke up your rear end, you will get the space they promise. We get WAY better compression than we or they estimated. San to san replication is really great too, it's super easy to reverse your replication if you want to go that route. Do you already have 10Gbe for FC switches? If not, don't forget their price in your quote (though you might be ok with 1Gbe). edit: shooting from the hip here, if I were building a new environment for a 200 person company today I would build two sites with the exact same specs. Each site would have a Nimble array, 3 of the smallest and cheapest HP servers that would give me n+1, and two small 10Gbe switches for it. I think you could do both sites for less than $120k with the VMware licensing. Some people will disagree, but you can probably dump your backup solution with this implementation, just go with san to san replication with longer SAN volume snapshot retention schedules. The DR benefits might justify this to your executive management. adorai fucked around with this message at 03:18 on Apr 22, 2016 |
|
#
?
Apr 22, 2016 03:13
|
|
|
One of my sites has a 13tb Nimble volume for production servers, very mixed load. VMware shows like 800gb free. Nimble showing like 3.6tb used. There is a good chunk of empty space on those VMs, but they are not too over allocated storage wise.
|
|
#
?
Apr 22, 2016 03:25
|
|
|
Pure has been flat out magical for us. One of my LUNs has 2 2tb file servers (DFS replication so they both have the same data). The VMs themselves are nearly at capacity, the LUN is only reporting 120gb used. 4 tb in VMware down to 120gb. Witchcraft.
|
|
#
?
Apr 22, 2016 03:31
|
|
|
bull3964 posted:Pure has been flat out magical for us. Their native encryption is great.
|
|
#
?
Apr 22, 2016 04:37
|
|
|
Kaninrail posted:Long story short, they want to start using Veeam for their backups. Veeam requires that it can make a snapshot of the largest drive. The SAN they have is at capacity, with no budget for replacing it. The datastores on the SAN are both close to capacity, and set to a 2MB block size. Since you can't create a large enough snapshot at that block size, Veeam won't work. So now there's this NAS. This sounds... Awful in every way.
|
|
#
?
Apr 22, 2016 05:17
|
|
|
A lot of those savings are from zero blocks being elimated or simply not being allocated at all on the storage side, and just reserved in VMFS. Compression is nice on Nimble, but comparing datastore use to storage use is really misleading unless you're using thin provisioned VMDKs everywhere. I've also seen some customers that don't get nearly the advertised compression ratios, though many do. I do think that the lack of deduplication is a problem for Nimble in competitive situations though. It's coming on their flash systems at least though, so that's a good sign. Pure is awesome if you can afford it.
|
|
#
?
Apr 22, 2016 07:05
|
|
|
|
| # ? May 8, 2024 04:50 |
|
|
Vulture Culture posted:Friendly reminder that putting your deltas onto another disk means that your running VMs all blow up and die when that storage goes away, so if you're planning on snapping onto discount storage, think again. If you have that disk already, Storage vMotion some of your lower-impact VMs onto it and call it a day.
|
|
#
?
Apr 22, 2016 08:31
|
|
