Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
ephphatha
Dec 18, 2009




notwithoutmyanus posted:

Why is this response to a ticket that was a result of an outage "open more tickets"? When do people get to analyzing what actually happened in the first place? :negative:

Broadcast message to all users posted:

<webapp> is experiencing high load and may be slow or non-responsive. If you need to access <webapp> try refreshing the page.

Broadcast message to all users, 10 seconds later posted:

<webapp> is experiencing high load and may be slow or non-responsive. If you need to access <webapp> try waiting five minutes, then refreshing the page.

Adbot
ADBOT LOVES YOU

hihifellow
Jun 17, 2005

seriously where the fuck did this genre come from

tjl posted:

Welp, so much for hoping I would dodge any KB2859537 issues. Sounds like we got lucky though, no BSOD's or anything too horrific; just broken apps.

Aww hell. I don't think this one has caused us any issues but it was approved in WSUS :negative:

I've already approved it for uninstall with a deadline of yesterday, but are there any advisories or anything that I can watch that will note when Microsoft releases a bum update like this?

FlapYoJacks
Feb 12, 2009

hihifellow posted:

Aww hell. I don't think this one has caused us any issues but it was approved in WSUS :negative:

I've already approved it for uninstall with a deadline of yesterday, but are there any advisories or anything that I can watch that will note when Microsoft releases a bum update like this?

The one thing I have learned from the embedded world, is to not update. EVER.

And if you must update, wait a few weeks before you update after the update has come out.

Inspector_666
Oct 7, 2003

benny with the good hair

Varkk posted:

You forget the immediate follow up "When is it coming back?"

Last time this happened I was IT support for a small newspaper and the editor was asking me, never mind the fact at the same moment he had one reporter on the phone with the power company preparing a story about the outage and another reporter talking to a police spokesperson about the car which crashed in to a power transformer, nope better ask the IT guy keeping an eye on the servers/UPS to make sure it was all behaving as it should during a blackout, he is the one who will know all about the power situation in the wider area..

Oh man. The place I'm working now is pod paradise compared to where I used to be. Customers (except for one or two) actually understand that I'm not the ISP, or Microsoft, or their building management company, or their printer leasing company.

Having other people able to understand the concept of "scope of service," much less act on it, is truly amazing.


(gently caress printers though.)

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug
I can't loving believe the VMware class I am co-instructor for did not have computers ready for students to log into, at 4:30 I walk in and see two people working under desks somewhat frantically trying to put computers hooked up.

"Heh well the ole interns were supose't do this but you know interns!"

No I don't, I do know there are two kinds of people though people who want to work and people who don't.


Oh well the class was mainly a success got some new 2x16(possibly 4x16) core HP, 512GB ram, w/ 4 128GB ssd's, and 8 network interfaces. So far way overkill but happy for it. Sadly can't touch them till winter, anyone got a guess how fast you can do a full vCloud+view+ICM+VCAP deploy? Let's find out

Dilbert As FUCK fucked around with this message at 02:47 on Aug 23, 2013

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

poo poo pissing me off.... my dumbass didn't plug in 1 of my OOB management ports on my servers, and of course that server doesn't want to reboot properly after I installed some patches. Guess my dumb rear end is driving 75 miles each way tonight to fix it. No one to blame buy myself. Do always plug in your iLO cards people.

Yaos
Feb 22, 2003

She is a cat of significant gravy.

skipdogg posted:

poo poo pissing me off.... my dumbass didn't plug in 1 of my OOB management ports on my servers, and of course that server doesn't want to reboot properly after I installed some patches. Guess my dumb rear end is driving 75 miles each way tonight to fix it. No one to blame buy myself. Do always plug in your iLO cards people.
You're lucky, we will never setup the OOB ports. We actually had one setup but then the motherboard died and was replaced under warranty but the iDrac was not working so that was the end of that. What, call Dell about it, nope, can't do that!

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

skipdogg posted:

Do always plug in your iLO cards people.

Judging by past experience, if there is one server without remote access in your data center, that server will always be the next to crash, without fail, leaving you with the choice of paying $20 out of your own pocket for parking or walking twelve blocks through South Murderville to the data center at 2AM to reboot it.

skipdogg
Nov 29, 2004
Resident SRT-4 Expert

Drive 75 miles, get up here, reboot server, Drive Array Battery is dead. My timetable to decommission this fucker just got pushed to tomorrow. iLO card is plugged in now though. One more reboot and it's time to drive home.

Crowley
Mar 13, 2003
What a wonderful way to start the day.

One of the nodes on the X9720 went down overnight. As luck would have it it didn't just go down down, it just stopped doing anything.. except react to heartbeat requests and pings. That meant that when I came in this morning the entire storage array seemed offline, and would time out for anything that looked like actual work. Couldn't mount the shares, couldn't load the management interface, couldn't SSH into the Fusion Manager, but you could ping it.
Since the heartbeat was A-OK the other nodes didn't think anything was wrong so they didn't do anything except serve up files (that couldn't be requested) and wait for the heartbeat to stop so they could swap the FM to a different node. :argh:

It turned out that the OS had borked itself somehow, and after I tried rebooting the node it crashed during boot and refused to come up again in anything but single-user mode. At least the other nodes finally figured out something was wrong and took over the FM, so now we have a working storage again - minus one node.

To top it all off I got a snarky reply from HP that they couldn't open a case without information about serial numbers and service contract number - when I told them in the initial mail that I had attached the entire service contract (with everything they could possibly wish to know about the hardware) in the initial support request. At least they shut up and opened the case when I made a "look at the attachment mentioned in the original mail" reply.

Oh well, at least we're going to the summer cabin tonight, and then off to car-shopping with my parents tomorrow. They need a new car, and I could use a replacement for my aging 1996 Fiesta. We've shopped cars together before, and it's amazing how big a discount dealers will offer you on a small car when you're buying a luxury car too. :v:

GargleBlaster
Mar 17, 2008

Stupid Narutard
Seemed relevant in some way

JohnnyCanuck
May 28, 2004

Strong And/Or Free

GargleBlaster posted:

I love power outages. Everyone comes in, with the lights off and everything else off, with the question "is the power off?"

No it's just the bloody weeping angels. Yes the power's off. Sod off, we're trying to safely shut down the servers before the UPS runs out.

Alright, apparently we work close enough to each other to grab lunch sometime.

I had to chase my replacement's replacement out of the server room last night. "You've already shut everything off, nothing else is running on battery, and they're saying power won't be back until at least midnight. GO. HOME."

Crowley
Mar 13, 2003
Anyone remember the name of that goon-made suite of tools that could be used to fix the most common problems in windows?

Lum
Aug 13, 2003

^^^^ Dial-a-Fix?

Potato Alley posted:

Your boxers must be pretty big for an entire server to fit into them :v:

This is him IRL: http://i.imgur.com/JbjAMyF.jpg

I really hope I'm not the only UK goon that immediately thought of this.

Lum fucked around with this message at 13:30 on Aug 23, 2013

Crowley
Mar 13, 2003

Lum posted:

^^^^ Dial-a-Fix?

That's the one. Thanks. :)




:(

Demonachizer
Aug 7, 2004

Crowley posted:

That's the one. Thanks. :)



:(

Copy the link and paste it in a new window.

Lum
Aug 13, 2003


Fixed: http://i.imgur.com/JbjAMyF.jpg

BabyFur Denny
Mar 18, 2003

Wtf did I just read?

Lum
Aug 13, 2003

BabyFur Denny posted:

Wtf did I just read?

Probably won't get it if you're not British and/or never read Viz. But first thing that sprang to mind when someone said they'd deployed a server in their boxers.

Wasn't intended to go on for six posts for a cheap joke that only some would get. Sorry!

Lum fucked around with this message at 14:29 on Aug 23, 2013

GargleBlaster
Mar 17, 2008

Stupid Narutard

JohnnyCanuck posted:

Alright, apparently we work close enough to each other to grab lunch sometime.

I had to chase my replacement's replacement out of the server room last night. "You've already shut everything off, nothing else is running on battery, and they're saying power won't be back until at least midnight. GO. HOME."

Sadly England isn't Canada :c

Go home? But but.. workaholism fix!


Crowley posted:

Anyone remember the name of that goon-made suite of tools that could be used to fix the most common problems in windows?

Goonbuntu?

Gumball Gumption
Jan 7, 2012

Anyone else using mobileiron as their MDM and have user registration fail with the new update?

antisodachrist
Jul 24, 2007

jim truds posted:

Anyone else using mobileiron as their MDM and have user registration fail with the new update?

Yes, they sent an announcement that pin based registrations won't work with the 5.7.2 client. They suggest using username/password ones in the meantime. They are hoping to have a new client up by tomorrow.

Orcs and Ostriches
Aug 26, 2010


The Great Twist

Lum posted:

^^^^ Dial-a-Fix?

That's goon-made? Awesome. That was a godsend working with XP.

antisodachrist
Jul 24, 2007
Regarding Mobileiron, I think it's all registrations with the new client version. Tried in my two test environments, VSP versions 5.6.2 and 5.7, and it works with the 5.7.1 client on the 5.6.2 version but not the 5.7 one. The 5.7.2 client just does not work for either.

JohnnyCanuck
May 28, 2004

Strong And/Or Free

GargleBlaster posted:

Sadly England isn't Canada :c
Ah, I confised you for EAT THE EGGS RICOLA

Also I'm starting to think there was no puppy and we've been lied to. :mad:

ookiimarukochan
Apr 4, 2011

jim truds posted:

using mobileiron as their MDM

There's your issue (at least it's not AirWatch!)
Yes, they're the magic quadrant leader at Gartner but all that tells you is that they have the deepest pockets.
(Perhaps if I shipped broken updates the company I work for would be in the magic quadrant rather than the niche quadrant?)

ookiimarukochan fucked around with this message at 16:18 on Aug 23, 2013

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug
"No one in their right mind should virtualize their monitoring for their environment"

Holy poo poo I never thought I could nerd rage this hard about something. This comes from one of the people ranking above me....

feld
Feb 11, 2008

Out of nowhere its.....

Feldman

Recently a colo customer of ours was purchased by Microsoft. They just finished their integration, took over their network equipment locking us out, and sent this ticket in

:downs: Hi, can you send us passwords to all the network equipment? Thanks!
:colbert: You locked us out and changed all the passwords during the takeover, remember? You were applying your Microsoft default network policies...
:downs: Do you think the employees we fired know those passwords?
:colbert: I don't know
:downs: Can you ask them? We can't....

Dilbert As gently caress posted:

"No one in their right mind should virtualize their monitoring for their environment"

Holy poo poo I never thought I could nerd rage this hard about something. This comes from one of the people ranking above me....

I'm confused. Are for or against monitoring virtualization? Because we're definitely against -- when your VM cluster goes down it'd be nice if, you know, your monitoring could tell you.

feld fucked around with this message at 16:46 on Aug 23, 2013

Kyrosiris
May 24, 2006

You try to be happy when everyone is summoning you everywhere to "be their friend".



A login came in for our main abuse@ email.

Jesus christ this is a firehose. :stare:

DragonReach Ghost
Sep 16, 2002

My Avatar is a Red Avatar

Dilbert As gently caress posted:

"No one in their right mind should virtualize their monitoring for their environment"

Holy poo poo I never thought I could nerd rage this hard about something. This comes from one of the people ranking above me....

Holy schnikies. That is funny, I have customer's that have virtualized almost 100% of their infrastructure, and we have deliberately made it so all the system center stuff can run in VMs. The only things that shouldn't be virtualized these days are generally time sensitive stuff.

feld
Feb 11, 2008

Out of nowhere its.....

Feldman

DragonReach posted:

Holy schnikies. That is funny, I have customer's that have virtualized almost 100% of their infrastructure, and we have deliberately made it so all the system center stuff can run in VMs. The only things that shouldn't be virtualized these days are generally time sensitive stuff.

and databases.

and any sensitive information you don't want stolen through a side-channel attack from another compromised VM on the same host.

Dilbert As FUCK
Sep 8, 2007

by Cowcaster
Pillbug

feld posted:

I'm confused. Are for or against monitoring virtualization? Because we're definitely against -- when your VM cluster goes down it'd be nice if, you know, your monitoring could tell you.

Totally for it, HA will roll it over shortly and give you a bunch of events, or use FT. Or have the monitoring system in your PROD cluster and vCenter alerting in your management cluster.

Unless your whole cluster melts down IE every host goes, and you don't have a management cluster you are boned either way

feld posted:

and databases.

and any sensitive information you don't want stolen through a side-channel attack from another compromised VM on the same host.

Uhh what? You should still be segmenting networks and what not to ensure data security on the network; 1 VM can't 'attack' another without compromising the underlying Hypervisor. But all modern databases support Virtual environments.

Dilbert As FUCK fucked around with this message at 17:18 on Aug 23, 2013

DragonReach Ghost
Sep 16, 2002

My Avatar is a Red Avatar

feld posted:

and databases.

and any sensitive information you don't want stolen through a side-channel attack from another compromised VM on the same host.

specify databases... some shouldn't be virtualized, but most can be.

You should still be following proper security and not just throwing everything virtualized onto a single environment.

your argument on side channel attacks is really not useful as I could accomplish the same thing with a physical host that is compromised if I have connectivity to anything else in your environment.

feld
Feb 11, 2008

Out of nowhere its.....

Feldman

Dilbert As gently caress posted:

Uhh what? You should still be segmenting networks and what not to ensure data security on the network; 1 VM can't 'attack' another without compromising the underlying Hypervisor. But all modern databases support Virtual environments.

You can steal private keys (and certainly other data) from another running VM on the same physical host. This has been known for several years. I believe University of Wisconsin Madison was the first to discuss this attack[1]. It was thought too difficult to do in the wild. And now it's been proven it can be done in the wild.

http://eprint.iacr.org/2013/448


edit: It's also not "attacking" another VM or compromising the hypervisor -- it's reading the leftover contents of the CPU caches. You know, since the CPU is constantly being timesliced between the different VMs and the contents of a request from another VM on the host can still be in the cache long enough for a different VM to retrieve it. The only fix for this is for dedicated CPU caches for each VM. This doesn't even exist as a thing.

edit2:
article and paper published about this
[1] http://www.darkreading.com/attacks-breaches/researchers-develop-cross-vm-side-channe/240012743

feld fucked around with this message at 17:42 on Aug 23, 2013

Dick Trauma
Nov 30, 2007

God damn it, you've got to be kind.
I decided to finally get an SSD for my home machine but encountered some weirdness after cloning to it. I posted this in the SSD thread but it doesn't get much traffic and there's plenty of smart people here:

I have a weird problem. I bought a Samsung 840 Pro 256 gig drive. I'm using the free Easeus 6.0 backup to clone my old drive to the new one. The old drive has three partitions: System Recovery (100 megs), Main Drive (900 gigs) and manufacturer restore (10 gigs.)

After cloning disk to disk from old to new the manufacture restore is still 10 gigs, but the System Recovery partition ballooned to 5 gigs and the main drive takes what's left. System boots fast but it's annoying to see so much wasted space.

Booted off the old drive, cleared off the SSD partitions and manually cloned the partitions one at a time. Sizes were then correct, but it was skipped in the boot sequence in favor of the old drive. I noticed that the SSD main partition wasn't active so I activated it and now my system attempts to boot from it but I get a message that the boot manager is missing.

Should I just be using a Windows repair CD to fixmbr and fixbot this thing, or is there some other issue with how I cloned via partitions?

feld
Feb 11, 2008

Out of nowhere its.....

Feldman

DragonReach posted:

specify databases... some shouldn't be virtualized, but most can be.

If you care about the integrity of this data you shouldn't virtualize it. It's extremely difficult to prove that a database's fsync request was honored all the way down to the platters.

DragonReach posted:

You should still be following proper security and not just throwing everything virtualized onto a single environment.

your argument on side channel attacks is really not useful as I could accomplish the same thing with a physical host that is compromised if I have connectivity to anything else in your environment.

What if it's not your private VM environment? Or what if you have VMs that can talk to the internet on the same physical host as ones that cannot? Your super_secret_stuff.txt is now vulnerable if one of the internet-facing machines gets compromised -- even if every ACL and firewall in the world is between them. If they were on different physical servers or different VM clusters it would be much better, but the number of deployments that look like this are rare.

FlapYoJacks
Feb 12, 2009

Dick Trauma posted:

I decided to finally get an SSD for my home machine but encountered some weirdness after cloning to it. I posted this in the SSD thread but it doesn't get much traffic and there's plenty of smart people here:

I have a weird problem. I bought a Samsung 840 Pro 256 gig drive. I'm using the free Easeus 6.0 backup to clone my old drive to the new one. The old drive has three partitions: System Recovery (100 megs), Main Drive (900 gigs) and manufacturer restore (10 gigs.)

After cloning disk to disk from old to new the manufacture restore is still 10 gigs, but the System Recovery partition ballooned to 5 gigs and the main drive takes what's left. System boots fast but it's annoying to see so much wasted space.

Booted off the old drive, cleared off the SSD partitions and manually cloned the partitions one at a time. Sizes were then correct, but it was skipped in the boot sequence in favor of the old drive. I noticed that the SSD main partition wasn't active so I activated it and now my system attempts to boot from it but I get a message that the boot manager is missing.

Should I just be using a Windows repair CD to fixmbr and fixbot this thing, or is there some other issue with how I cloned via partitions?

Use a partimage livecd and try that. Perhaps the ssd is misaligned also considering it's block sizes are (*possibly*) different than the hdd.

SEKCobra
Feb 28, 2011

Hi
:saddowns: Don't look at my site :saddowns:

Dick Trauma posted:

I decided to finally get an SSD for my home machine but encountered some weirdness after cloning to it. I posted this in the SSD thread but it doesn't get much traffic and there's plenty of smart people here:

I have a weird problem. I bought a Samsung 840 Pro 256 gig drive. I'm using the free Easeus 6.0 backup to clone my old drive to the new one. The old drive has three partitions: System Recovery (100 megs), Main Drive (900 gigs) and manufacturer restore (10 gigs.)

After cloning disk to disk from old to new the manufacture restore is still 10 gigs, but the System Recovery partition ballooned to 5 gigs and the main drive takes what's left. System boots fast but it's annoying to see so much wasted space.

Booted off the old drive, cleared off the SSD partitions and manually cloned the partitions one at a time. Sizes were then correct, but it was skipped in the boot sequence in favor of the old drive. I noticed that the SSD main partition wasn't active so I activated it and now my system attempts to boot from it but I get a message that the boot manager is missing.

Should I just be using a Windows repair CD to fixmbr and fixbot this thing, or is there some other issue with how I cloned via partitions?

EaseUS Partition Master is what I used and it worked perfectly.

DragonReach Ghost
Sep 16, 2002

My Avatar is a Red Avatar

feld posted:

If you care about the integrity of this data you shouldn't virtualize it. It's extremely difficult to prove that a database's fsync request was honored all the way down to the platters.

don't use VMDK or VHD for the database drives, vHBA to SAN or RDM should get you what you need. Again, not all DBs should be virtualized, but a number of them can be. I'd still say most can be, but you have to design your environment for the requirements of the data.

feld posted:

What if it's not your private VM environment? Or what if you have VMs that can talk to the internet on the same physical host as ones that cannot? Your super_secret_stuff.txt is now vulnerable if one of the internet-facing machines gets compromised -- even if every ACL and firewall in the world is between them. If they were on different physical servers or different VM clusters it would be much better, but the number of deployments that look like this are rare.

So we basically agree then. I did say proper security.

Adbot
ADBOT LOVES YOU

feld
Feb 11, 2008

Out of nowhere its.....

Feldman

DragonReach posted:

don't use VMDK or VHD for the database drives, vHBA to SAN or RDM should get you what you need. Again, not all DBs should be virtualized, but a number of them can be. I'd still say most can be, but you have to design your environment for the requirements of the data.

Hypervisors can (and do!) still lie about the fsync flush. vHBA or RDM definitely help remove an entire layer, but don't solve the problem entirely.

edit: I think the best you could do is PCI passthrough of an iSCSI/fiber HBA or an entire RAID controller, but then what's the point of virtualizing when that VM is tied to the hardware anyway?

feld fucked around with this message at 17:56 on Aug 23, 2013

  • Locked thread