Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin
Would I be able to prove this with esxtop when I go into work tomorrow?

Edit:

I think I picked the right career field because I can't wait to go in tomorrow and find out!

Dr. Arbitrary fucked around with this message at 03:46 on Nov 22, 2014

Adbot
ADBOT LOVES YOU

Wicaeed
Feb 8, 2005
So this is going to sound like a really dumb question:

What kinds of companies are heavily invested in VMware?

I'm sort of the head VMware guy at our company (in that I have the most experience & knowledge) but don't actually possess any certs. We have a pretty limited deployment (few ESXi boxes for a dev/test environment, and a few for a limited production environment).

My employer is refusing to reimburse the cost of a VCP for me since:

A: They don't have an initiative for VMware on the table (no plans to expand our VMware footprint)
B: Even if it did happen, they historically don't reimburse certification costs

I want to know what companies to search for if I wanted a job that was pretty much dedicated to VMware & virtualization.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

DevNull posted:

Someone took it out because they were worried that a customer might somehow see it, even though it was not possible.
Like when that developer build expiration date made it into a released ESXi version and took out half the world's VMware clusters in a matter of hours?

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Wicaeed posted:

I want to know what companies to search for if I wanted a job that was pretty much dedicated to VMware & virtualization.
Consultants or managed service providers, pretty much.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer

Wicaeed posted:

I want to know what companies to search for if I wanted a job that was pretty much dedicated to VMware & virtualization.
since a good virtualization tech needs to know about storage, networking, guest operating systems, server hardware, etc.. There are probably only jobs where you only touch VMware at lovely IT shops.

Kachunkachunk
Jun 6, 2011

Wicaeed posted:

What kinds of companies are heavily invested in VMware?
The dedicated VMware job thing does seem to be a bit of a unicorn. At least if you want to be effective, you're going to need a lot of other experience or knowledge, to boot.
Most shops need a lot of broad experience. And well, even at VMware you need to have, or will eventually accumulate, a lot of third-party knowledge just so you or the customers can accomplish what they need; people don't run virtualization just for the sake of it, it's to run everything else on top of it. So I suppose knowing how to run those things well/properly/reliably is entirely the reason this secondary knowledge keeps proving necessary.

And it may be the kool-aid, but last I have heard (so... anecdotal by two degrees as of this comment here), all of the fortune 500 companies use VMware products somewhere. I wouldn't assume that any one of them will ever be 100% virtualized either, however.
Even shops like Facebook and Google have some sort of investment into VMware products; it depends on what the business unit within needs.

Your company really should look at it more as reimbursing you for the training, not just looking at that cert. It's pretty important that folks get at least some training on the vSphere products. That knowledge can save money (effective virtualization and better decisions on resource configuration/allocation), and save the company from a lot of pain later (poor design decisions, human error/mistakes, fault management). The training doesn't proof you for all of that, but it sets a really good ground and helps you start to ask the right questions later.

Wicaeed
Feb 8, 2005

Kachunkachunk posted:

The dedicated VMware job thing does seem to be a bit of a unicorn. At least if you want to be effective, you're going to need a lot of other experience or knowledge, to boot.
Most shops need a lot of broad experience. And well, even at VMware you need to have, or will eventually accumulate, a lot of third-party knowledge just so you or the customers can accomplish what they need; people don't run virtualization just for the sake of it, it's to run everything else on top of it. So I suppose knowing how to run those things well/properly/reliably is entirely the reason this secondary knowledge keeps proving necessary.

And it may be the kool-aid, but last I have heard (so... anecdotal by two degrees as of this comment here), all of the fortune 500 companies use VMware products somewhere. I wouldn't assume that any one of them will ever be 100% virtualized either, however.
Even shops like Facebook and Google have some sort of investment into VMware products; it depends on what the business unit within needs.

Your company really should look at it more as reimbursing you for the training, not just looking at that cert. It's pretty important that folks get at least some training on the vSphere products. That knowledge can save money (effective virtualization and better decisions on resource configuration/allocation), and save the company from a lot of pain later (poor design decisions, human error/mistakes, fault management). The training doesn't proof you for all of that, but it sets a really good ground and helps you start to ask the right questions later.

It's really an issue at the top. I know my boss (Director) wants to do it, but every time he asks the CEO for the money he gets shot down.

Personnel expenses are kind frowned upon here, which is lame as gently caress and part of the reason I'm starting to look around for a new employer.

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

Richard Noggin posted:

I wonder why the default isn't the VMXNET.

native drivers

evol262
Nov 30, 2010
#!/usr/bin/perl

This is reasonable for Solaris or BSD, but vmxnet has been mainline for half a decade in Linux. Defaulting to it if it's a Linux guest is very safe unless it's rhel5

Vaporware
May 22, 2004

Still not here yet.

Wicaeed posted:

What kinds of companies are heavily invested in VMware?

Anyone using computers in manufacturing has already decided on their highly available solution.

My parent company is already invested in the Microsoft Ecosystem, so even though we want to use VMware we have to use Hyper-V.

We still end up using VMware workstation and fusion for portables, because of the whole "no local USB" thing until your HyperV host is W8.1 and running a V2 guest.

DevNull
Apr 4, 2007

And sometimes is seen a strange spot in the sky
A human being that was given to fly

Misogynist posted:

Like when that developer build expiration date made it into a released ESXi version and took out half the world's VMware clusters in a matter of hours?

No, not at all like that. They put that time bomb into the release build intending to remove it before shipping the product and forgot. The message I mentioned is stripped out as part of the build type. If we sent out the wrong build type to customers, it would make it trivial to disassemble and reverse engineer our code. We would have far bigger problems than a hard to hit message.

Wicaeed
Feb 8, 2005
Do vCenter events/tasks track more detailed information, such as if someone mounted a CD-ROM drive or disconnected a VMs NIC?

Best I can see right now is that someone reconfigured a virtual machine, but that can mean anything from adding a new HDD or changing a network adapters VLAN assignment.

Kachunkachunk
Jun 6, 2011
A lot of VM configuration changes are tracked host-side, rather. You can refer to some content in the right /var/run/log/hostd*.gz file uzing "zcat <file>.gz | less" (or hopefully just plain old /var/run/log/hostd.log if it's recent enough), or for many changes you can also look right at the VM's respective vmware.log file.
The former is only valid for the server that had the VM registered to it at the time of the change.
The latter is only possible for the server that's running the VM right this moment.

Fake Edit:
Network related changes (virtual port assignment) is visible via /var/run/log/vmkernel.log. Like hostd, do this from the appropriate box that handled the request.

Anyway it's probably not as... autidy as you might hope. Maybe the web client tracks this stuff better! I don't really know.

1000101
May 14, 2003

BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY BIRTHDAY FRUITCAKE!
It doesn't but I believe the detail you're looking for is stuffed in the vCenter database. I just don't recall which table.

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

evol262 posted:

This is reasonable for Solaris or BSD, but vmxnet has been mainline for half a decade in Linux. Defaulting to it if it's a Linux guest is very safe unless it's rhel5

Windows does not have it native.


Also rebuilding a Citrix VDI + vSphere environment is before lunch is TOTALLY going on my resume.

Dilbert As FUCK fucked around with this message at 02:47 on Nov 27, 2014

Syano
Jul 13, 2005

Dilbert As gently caress posted:

Windows does not have it native.


Also rebuilding a Citrix VDI + vSphere environment is before lunch is TOTALLY going on my resume.

Right after the part where you upgraded a 25k customer SAN in 8 hours and then went to hooters? Did you do both in the same day and also fix the CSS on your business' home page while banging a hot chick?

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

What lovely SAN takes more than 30 minutes to upgrade?

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

Syano posted:

Right after the part where you upgraded a 25k customer SAN in 8 hours and then went to hooters? Did you do both in the same day and also fix the CSS on your business' home page while banging a hot chick?

Basically, drove from Charleston SC, found issue with VNX in Raleigh, fixed that! Rebuilt a vSphere environment from 5.1 to 5.5 and fixed citrix 7.5 got drunk and passed out. Woke up at 8am to PS600VX+8 enclosures issue for 25K user environment, fixed that and some sql poo poo + moodle; then some multipathing issues.

NippleFloss posted:

What lovely SAN takes more than 30 minutes to upgrade?

Equallogic with 8 enclosures

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

Dilbert As gently caress posted:

Equallogic with 8 enclosures

You guys should invest in something not horrible so that upgrading your storage isn't worthy of note.

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

NippleFloss posted:

You guys should invest in something not horrible so that upgrading your storage isn't worthy of note.

Wow that is great, I love it, man I wish I worked at a place that gave IT as much budget per the storage capacity it needed!

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

Dilbert As gently caress posted:

Wow that is great, I love it, man I wish I worked at a place that gave IT as much budget per the storage capacity it needed!

There are some good options out there now that are as cheap as EQL. It was fine for its time, but it's basically dead as a storage platform moving forward and it's really showing it's age. But even if it wasn't old it's still inexcusable that upgrades are that onerous. Literally everything I've ever worked with takes maybe a half hour and doesn't involve sweaty palms wondering what will break.

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

NippleFloss posted:

There are some good options out there now that are as cheap as EQL. It was fine for its time, but it's basically dead as a storage platform moving forward and it's really showing it's age. But even if it wasn't old it's still inexcusable that upgrades are that onerous. Literally everything I've ever worked with takes maybe a half hour and doesn't involve sweaty palms wondering what will break.

Yes I am well aware of those, but when you are told "move poo poo to AWS", and "no budget till 2015 Q2" you make do. Honeslty I don'y understand why SA has such a hard time understanding budget constraints; yes everything could be fixed by money but engineer a solution around those who don't for that project.

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

Dilbert As gently caress posted:

Yes I am well aware of those, but when you are told "move poo poo to AWS", and "no budget till 2015 Q2" you make do. Honeslty I don'y understand why SA has such a hard time understanding budget constraints; yes everything could be fixed by money but engineer a solution around those who don't for that project.

I understand budgets, I just don't understand why a storage firmware upgrade is worth bragging about. I'd put that somewhere around patching an ESX cluster on the difficulty scale.

If it's hard enough to merit bragging about then it's bad enough that you should be able to make a case for replacement.

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

NippleFloss posted:

I understand budgets, I just don't understand why a storage firmware upgrade is worth bragging about. I'd put that somewhere around patching an ESX cluster on the difficulty scale.

If it's hard enough to merit bragging about then it's bad enough that you should be able to make a case for replacement.

I wasn't bragging though, I was just stated what I did and how it 'sucked'. Are making this whole sub-forum where every post is made as a complaint? I orginally made the Working in IT threads to be some kind of non bitch poo poo; but drat..... It appears most any statements (either from experiences, tales, etc) are taken negatively...

Frimware upgrades for 25k users I think is worth a post; or should I pick a helpdesk teir 1 ticket and complain about how some printer won't work for a site of <75 people?

Please correct me if I am wrong.

some kinda jackal
Feb 25, 2003

 
 
Is there any way to let 5.5U2 intelligently determine what VMs need vFlash Read Cache most, or do I HAVE to create reservations manually for each of my VMs? I'm running an 80GB SSD VFRC for a 300GB datastore in one of my lab vhosts and I guess I'd rather not micromanage it but it's not a huge deal or anything.

some kinda jackal fucked around with this message at 04:18 on Nov 27, 2014

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

Martytoof posted:

Is there any way to let the 5.5U2 intelligently determine what VMs need vFlash Read Cache most, or do I HAVE to create reservations manually for each of my VMs? I'm running an 80GB SSD VFRC for a 300GB datastore in one of my lab vhosts and I guess I'd rather not micromanage it but it's not a huge deal or anything.

Manually recreate without a wild card and nice naming convention....

Just wait 4 months if you want a global structure....

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

NippleFloss posted:

What lovely SAN takes more than 30 minutes to upgrade?
Depends whether you're also doing disk firmware, no?

EoRaptor
Sep 13, 2003

by Fluffdaddy

NippleFloss posted:

You guys should invest in something not horrible so that upgrading your storage isn't worthy of note.

Having dealt with Equallogic before, DaF is probably not doing anything wrong here. EQL boxes that are grouped together form a storage grid, with each box acting independently to serve its own data, and sharing management info with other boxes to direct requests for data that it doesn't have to the correct box. It means you can add more i/o by adding more boxes in a linear fashion, but the downside is you need to upgrade each box one at a time and in the proper firmware sequence, or the inter-box communication will break and everything will poo poo the bed.

The upgrade process is mostly reliably boring, but it can a take a long, long time to get it done without missing any steps and avoiding rushing any of it.

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

EoRaptor posted:

Having dealt with Equallogic before, DaF is probably not doing anything wrong here. EQL boxes that are grouped together form a storage grid, with each box acting independently to serve its own data, and sharing management info with other boxes to direct requests for data that it doesn't have to the correct box. It means you can add more i/o by adding more boxes in a linear fashion, but the downside is you need to upgrade each box one at a time and in the proper firmware sequence, or the inter-box communication will break and everything will poo poo the bed.

The upgrade process is mostly reliably boring, but it can a take a long, long time to get it done without missing any steps and avoiding rushing any of it.

I understand how EQL works and don't think DAF is doing anything wrong. I just think EQL is bad. You can do scale out, even tightly coupled scale out, without making upgrades long and painful.

Internet Explorer
Jun 1, 2005





As annoying as I find DaF's posting and as much as I appreciate NippleFloss's insight, I wouldn't call updating SAN firmware mundane and boring. Yeah, it mostly works, but lets be honest here who hasn't had a firmware upgrade blow up in their faces?

I have had EQL, EMC VNX, and HP LeftHand firmware upgrades all go south and cause unexpected downtime.

Hell, quite a while ago in this thread I was saying I update firmware on SANs I touch fairly often, and almost everyone else came out to say don't touch it if it isn't broken. If it is so easy / fail-proof, why was everyone voting for "don't touch it"?

I actually just got done installing the Dell OpenManage VIB on an ESXi 5.5 Update 1 host. Done it a million times. This server rebooted and couldn't find its hard drives. Had nothing to do with the VIB, but made my day longer than it should have been. Sometimes poo poo just breaks when you reboot it.

Gucci Loafers
May 20, 2006

Ask yourself, do you really want to talk to pair of really nice gaudy shoes?


Are SAN firmware updates typically like other hardware?

You restart the device and boot into some kind of flash firmware mode from USB/CD and pray.

Internet Explorer
Jun 1, 2005





They're generally a little more advanced than that, but the basic gist is the same.

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

Tab8715 posted:

Are SAN firmware updates typically like other hardware?

You restart the device and boot into some kind of flash firmware mode from USB/CD and pray.

update a frimware that holds 25Mil worth of data.


Is that odd?

Internet Explorer posted:

As annoying as I find DaF's posting and as much as I appreciate NippleFloss's insight, I wouldn't call updating SAN firmware mundane and boring. Yeah, it mostly works, but lets be honest here who hasn't had a firmware upgrade blow up in their faces?

I have had EQL, EMC VNX, and HP LeftHand firmware upgrades all go south and cause unexpected downtime.

Hell, quite a while ago in this thread I was saying I update firmware on SANs I touch fairly often, and almost everyone else came out to say don't touch it if it isn't broken. If it is so easy / fail-proof, why was everyone voting for "don't touch it"?

I actually just got done installing the Dell OpenManage VIB on an ESXi 5.5 Update 1 host. Done it a million times. This server rebooted and couldn't find its hard drives. Had nothing to do with the VIB, but made my day longer than it should have been. Sometimes poo poo just breaks when you reboot it.

Walked in Monday to upgrade us ti 5.5 U2; kinda funny when dipshits are like "wow this is so hard and 'hur dur can't do it'. Just learn how to understand storage, network, and compute infrastructure...

It pisses me off when people say they have more than 5 people and run a 25K+ user environment and complain about poo poo...

Dilbert As FUCK fucked around with this message at 05:57 on Nov 29, 2014

jaegerx
Sep 10, 2012

Maybe this post will get me on your ignore list!


Jesus Christ stop bragging. I just setup 200 servers in 9 datacenters with 800 vms that monitor 4 million datapoints of all of the networking hardware for a multi billion dollar company. Yes that's with a B. It's backed on 5tb of San data and using luster for its rrd data.

I don't loving post every minuscule loving thing I do. You're not special. You're not some unique snowflake.

E: removed bullshit money thing cause I feel like an rear end in a top hat posting it even as a comeback to daf

Shut the gently caress up DaF

jaegerx fucked around with this message at 06:18 on Nov 29, 2014

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

jaegerx posted:

Jesus Christ stop bragging. I just setup 200 servers in 9 datacenters with 800 vms that monitor 4 million datapoints of all of the networking hardware for a multi billion dollar company. Yes that's with a B. It's backed on 5tb of San data and using luster for its rrd data.

I don't loving post every minuscule loving thing I do. You're not special. You're not some unique snowflake.

Also when we went public I tripped down on my stock and made more money than you can possibly hope to even retire on.

Shut the gently caress up DaF

Cool! sounds fun, what testing and other environmental challenges did you face?

jaegerx
Sep 10, 2012

Maybe this post will get me on your ignore list!


Dilbert As gently caress posted:

Cool! sounds fun, what testing and other environmental challenges did you face?

There a normal post. Rrd files are just terrible in nature due to the small size and constant writing nature. Running them on lustre was a challenge. We tried gluster first but it wasn't up to it. We eventually found the sweet spot with 6 management nodes. We could've used fiber but the expense was too high so instead we use the fiber to support our vms over clvm so we can pass them around instantly via kvm.

This was with zenoss commercial edition. We needed some of the commercial packages to support our larger network hardware.

Gucci Loafers
May 20, 2006

Ask yourself, do you really want to talk to pair of really nice gaudy shoes?


Dilbert As gently caress posted:

update a frimware that holds 25Mil worth of data.


Is that odd?

I don't know, I'm storage illiterate. You tell me.

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

Tab8715 posted:

I don't know, I'm storage illiterate. You tell me.

No, it's not odd. Storage upgrades don't get harder when the data they hold is worth more, and they don't generally get harder as the amount of data held increases, other than taking more time as you add controllers. It's about as "difficult" as upgrading an ESX cluster or something similar. You load the new software/firmware, fail over in some capacity, and reboot, then repeat on all controllers. It can be pucker inducing, especially the first time or two, but it's not really tricky.

Internet Explorer posted:

I have had EQL, EMC VNX, and HP LeftHand firmware upgrades all go south and cause unexpected downtime.

Sure, things can go wrong. I'd understand a post about that. When things break it's interesting, but when something that works right most of the time works right again it's not really that interesting or instructive. If it's your first time doing it then I can understand wanting to make a little happy, self-congratulatory post, but DAF is past that point in his career.

As far as why you'd be cautioned not to upgrade if you don't have a reason to...well, because you don't have a reason to. Upgrades themselves are often trivial, but bugs and regressions are common. If it ain't broke, don't fix it is generally a good rule for any critical infrastructure. Since storage isn't generally as exposed to security flaws and new features aren't added at the rate you get with something like ESX, the reasons to upgrade are usually limited to mitigating bugs or performance issues.

YOLOsubmarine fucked around with this message at 06:59 on Nov 29, 2014

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

jaegerx posted:

There a normal post. Rrd files are just terrible in nature due to the small size and constant writing nature. Running them on lustre was a challenge. We tried gluster first but it wasn't up to it. We eventually found the sweet spot with 6 management nodes. We could've used fiber but the expense was too high so instead we use the fiber to support our vms over clvm so we can pass them around instantly via kvm.

This was with zenoss commercial edition. We needed some of the commercial packages to support our larger network hardware.
Did you try any distributed databases like OpenTSDB that are actually built to handle time series data at that kind of scale?

Adbot
ADBOT LOVES YOU

jaegerx
Sep 10, 2012

Maybe this post will get me on your ignore list!


Misogynist posted:

Did you try any distributed databases like OpenTSDB that are actually built to handle time series data at that kind of scale?

Unfortunately not. Zenoss 4.x series doesn't support it. However 5.0 is supposed to be going towards using Hadoop and I wouldn't be surprised if opentsdb is their solution. Rrd files are just nasty. We had to turn off data for things that our noc would like to have but wasn't necessary. Fortunately the new environment is built with a staging environment in mind to allow testing of 5.x without taking down the existing.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply