Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
spankmeister
Jun 15, 2008






Z170 iirc.

Adbot
ADBOT LOVES YOU

SamDabbers
May 26, 2003



Re: booting from NVMe without BIOS/UEFI support, one workaround would be to put your bootloader on another disk (or USB stick) and boot from that while having your C drive on the NVMe. When installing Windows, create an EFI (System) partition on a disk the system can boot from, and manually create your C partition on the NVMe.

Light reading:
https://superuser.com/questions/665923/move-efi-system-partition-to-another-drive

stevewm
May 10, 2005
I can tell you for a fact that my old Asrock Z97 mobo supported NVMe, but was initially unable to boot from it when I got my first NVMe drive. However I discovered Asrock had made a BIOS update that added the ability to boot from NVMe.

I ended up upgrading mobos anyways.. Z97 is PCI 2.0 for all but the 16x slot.. so my OCZ RD400 SSD was being limited to 1500MB/sec. Which it is fully capable of 2800MB/sec.

ilkhan
Oct 7, 2004

I LOVE Musk and his pro-first-amendment ways. X is the future.

blackswordca posted:

Have you done a bios update? Older versions of the bios dont have proper support for the m2 drive. They will see it as a storage device, not a boot device.

Also the Z97 board only has a gen 3.0 x2 m2 slot so there may be some odd performance issues.
Bingo

Updated bios (and ordeal in and of itself) and good to go.

Zamboni Apocalypse
Dec 29, 2009

Arquinsiel posted:

No, MI-6 is when your overseas datacenter link drops. You're thinking of MI-5.

Six and Five are pleasure to deal with, compared to the hassle of.... "incidents of exotic cause", shall we say, that draw the attention of MI-13. :cthulhu:

The Muffinlord
Mar 3, 2007

newbid stupie?
My first call today was from a doctor who called to ask me how, "ever since you guys stopped letting us use webmail", he was supposed to print forms from his Gmail that he absolutely needed now now now. I told him that he could forward them along to me and print them down here, and come down to the IT dungeons to get them. He sends me the documents and I send him an email back that I have them at my desk, and then I never hear back from him.

Until like 1:40 today when the spam filter on our O365 gives me a digest message with his response on it from five minutes after my reply, asking me to run them up to him. I guess it wasn't really worth him following up on, despite the urgency.

I never really got why my high school girlfriend had such a distrust of doctors until I started working for them.

Arquinsiel
Jun 1, 2006

"There is no such thing as society. There are individual men and women, and there are families. And no government can do anything except through people, and people must look to themselves first."

God Bless Margaret Thatcher
God Bless England
RIP My Iron Lady

Zamboni Apocalypse posted:

Six and Five are pleasure to deal with, compared to the hassle of.... "incidents of exotic cause", shall we say, that draw the attention of MI-13. :cthulhu:
Sadly MI-13 was just a recon unit. Learning ruins fun.

potato of destiny
Aug 21, 2005

Yeah, welcome to the club, pal.

The Muffinlord posted:

My first call today was from a doctor who called to ask me how, "ever since you guys stopped letting us use webmail", he was supposed to print forms from his Gmail that he absolutely needed now now now. I told him that he could forward them along to me and print them down here, and come down to the IT dungeons to get them. He sends me the documents and I send him an email back that I have them at my desk, and then I never hear back from him.

Until like 1:40 today when the spam filter on our O365 gives me a digest message with his response on it from five minutes after my reply, asking me to run them up to him. I guess it wasn't really worth him following up on, despite the urgency.

I never really got why my high school girlfriend had such a distrust of doctors until I started working for them.

THIS IS AFFECTING PATIENT CARE

literally everything we do or don't do affects patient care, it's a god damned hospital

The Muffinlord
Mar 3, 2007

newbid stupie?

potato of destiny posted:

THIS IS AFFECTING PATIENT CARE

literally everything we do or don't do affects patient care, it's a god damned hospital

One of these days I'm gonna get a talking to because I'm gonna snap and ask a doctor why, if they're so drat smart, can't they figure out the simple poo poo they're trying to do.

Myrridinos
Jan 7, 2010
Backup software caused virtual machine running in Hyper-V to be running from a snapshot which used up the rest of the volume's free space resulting in an inability to merge the snapshot and the virtual machine going critical due to there being no more space to expand the snapshot image that it was running on. Said VM is running terminal services for a site that is all thin clients.

Didn't get my morning joe till it weren't morning no more.

CrazyLittle
Sep 11, 2001





Clapping Larry
paging larchesdanrew:




Jaded Burnout posted:

a minor incident is MiI.

excuse me, it's base-2 so it should be MibibiI

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else

Myrridinos posted:

Backup software caused virtual machine running in Hyper-V to be running from a snapshot which used up the rest of the volume's free space resulting in an inability to merge the snapshot and the virtual machine going critical due to there being no more space to expand the snapshot image that it was running on. Said VM is running terminal services for a site that is all thin clients.

Didn't get my morning joe till it weren't morning no more.

Did the backup stall and just not remove its own snapshot? Even then, how old was this snapshot? I feel like there must have been a couple points where this could have been noticed before almost rendering an entire host offline.

Renegret
May 26, 2007

THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*


Cat Army Sworn Enemy

potato of destiny posted:

literally everything we do or don't do affects patient care, it's a god damned hospital

lmao same here, except not a hospital

A few years back we had to go and define what an outage was. The old way was saying "Well, does it affect customers?"

That ended around the time I had to open a Severity 1 urgent ticket all hands on deck let's wake up the on call ticket on loving Christmas day because we lost a single mail store. I mean yeah we had 29 other servers but you see it could be customer affecting because we have less capacity :siren: :siren: :siren: :siren: :siren: ALERT ALERT ALERT

Ghostlight
Sep 25, 2009

maybe for one second you can pause; try to step into another person's perspective, and understand that a watermelon is cursing me



My coworker has just discovered that our backup system runs a daily incremental Monday-Thursday to disk, Friday it runs an incremental to disk with duplicate to tape, and every month we do a full to disk then tape. :madmax:

Aragaith
Jul 26, 2006

Kirisame Marisa.
In space.
Help.

potato of destiny posted:

THIS IS AFFECTING PATIENT CARE

literally everything we do or don't do affects patient care, it's a god damned hospital

Someone or something just locked out every ones accounts at 4:30pm today. For a hospital that has ambulatory sites in a nearly 100 mile radius. Over 4000 user accounts disabled in AD and as far as I'm hearing right now there is currently no way to unlock them. No computer login means no EMR. The downtime solution doesn't really take this into account. I'm only slowly getting the play by play of the 3 hour long conference call going on right now. The word of the day is shitshow.

hihifellow
Jun 17, 2005

seriously where the fuck did this genre come from

Aragaith posted:

Someone or something just locked out every ones accounts at 4:30pm today. For a hospital that has ambulatory sites in a nearly 100 mile radius. Over 4000 user accounts disabled in AD and as far as I'm hearing right now there is currently no way to unlock them. No computer login means no EMR. The downtime solution doesn't really take this into account. I'm only slowly getting the play by play of the 3 hour long conference call going on right now. The word of the day is shitshow.

There's a virus out there that will try to brute force every AD account it can find. Last time I heard of it infecting a network the only way they could do anything was being lucky enough to have someone with domain admin still logged in with ADUC already up so they could unlock their own account every time they wanted to try something.

Myrridinos
Jan 7, 2010

ChubbyThePhat posted:

Did the backup stall and just not remove its own snapshot? Even then, how old was this snapshot? I feel like there must have been a couple points where this could have been noticed before almost rendering an entire host offline.

Snapshot was made last night. Backup went stupid and somehow did two 900GB incremental backups of a virtual machine that has less than 800GB of data stored. This filled the repository and for some reason old restore points were unable to be purged causing the backup to fail. I don't know if that strangeness is related to this issue though.

From what I can gather/guess each time it backed up successfully and merged the snapshot back into the main vhdx the dynamic vhdx grew by 1-2 gb. Over a course of a month this caused the free space to get gobbled up enough that it no longer had enough room to merge the snapshot.

I moved the vhdx and corresponding snapshot to another volume, merged, and got them back up. Tonight we clean up data inside the vhdx and convert it to a fixed disk image before moving it back.

Normally I get an alert long before disk space goes critical like this, but N-central monitoring was never enabled for the volume. I was told our automation should have added it automagically but I guess that's not the case.

Myrridinos fucked around with this message at 02:58 on May 4, 2018

Inspector_666
Oct 7, 2003

benny with the good hair
I had to expand the OS drive on the Ubuntu VM that runs our Zabbix server today, which was harrowing, but I snapshotted the thing before doing the work (which ended up coming in handy) and then immediately deleted the snapshot when it came back up and showed the right info thanks to the amount of reading this thread I do.

Thanks Goons!

Jaded Burnout
Jul 10, 2004


Ghostlight posted:

My coworker has just discovered that our backup system runs a daily incremental Monday-Thursday to disk, Friday it runs an incremental to disk with duplicate to tape, and every month we do a full to disk then tape. :madmax:

Hang on, the incremental that goes to tape on Friday is an incremental based off the disk-only copy from Thursday?

Ghostlight
Sep 25, 2009

maybe for one second you can pause; try to step into another person's perspective, and understand that a watermelon is cursing me



Jaded Burnout posted:

Hang on, the incremental that goes to tape on Friday is an incremental based off the disk-only copy from Thursday?
yes

dogstile
May 1, 2012

fucking clocks
how do they work?
I remember before swapping to itil (which i'm still convinced is like 90% a waste of time) we used MI at my first job.

Everything else was ticket number and severity.

"Hey Dogstile we've got an MI with BIG CUSTOMER so can you drop 23453 sev 3 and 23455 sev 4 to Jim to have a look now?"

Jaded Burnout
Jul 10, 2004



Oops.

ElehemEare
May 20, 2001
I am an omnipotent penguin.

Write your fulls to an on-prem backup device, diffs to a public cloud, and your log backups to the operational SAN.

NO SINGLE POINTS OF FAILURE except for you, the engineer who designed this.

Jaded Burnout
Jul 10, 2004


ElehemEare posted:

NO SINGLE POINTS OF FAILURE except for you, the engineer who designed this.

Who told you I was single? Bitch I'll cut them.

The Macaroni
Dec 20, 2002
...it does nothing.

Aragaith posted:

Someone or something just locked out every ones accounts at 4:30pm today. For a hospital that has ambulatory sites in a nearly 100 mile radius. Over 4000 user accounts disabled in AD and as far as I'm hearing right now there is currently no way to unlock them. No computer login means no EMR. The downtime solution doesn't really take this into account. I'm only slowly getting the play by play of the 3 hour long conference call going on right now. The word of the day is shitshow.
Reading this makes me more than a little ill. Yikes. I've been through a couple of EMR downtime events (fortunately I don't support the EMR system at all) but never anything like a giant AD lockout. I would not want to be on that conference call.

Neddy Seagoon
Oct 12, 2012

"Hi Everybody!"

The Macaroni posted:

Reading this makes me more than a little ill. Yikes. I've been through a couple of EMR downtime events (fortunately I don't support the EMR system at all) but never anything like a giant AD lockout. I would not want to be on that conference call.

I can only assume whoever caused that is going to experience what looks to be all the hallmarks of a perfectly-natural heart attack. Why there was an ambulance already at their home when the 911 call came through is pure coincidence.

GWBBQ
Jan 2, 2005


mllaneza posted:

Working as intended, will not do not fix.

Boss is good, you guys hiring ?
Sadly, we're not. Maybe in a couple of years when we double our student population.

spankmeister posted:

Good boss, that's his job.
He's the easiest guy in the world to work for because he's competent, nice, and has realistic expectations.

The Macaroni posted:

The weird thing about it is that customer confidence would actually be boosted by the vendor saying, "You're totally right, this is not working and we're going to make it right within two business days. Oh look, we did it in 2 hours! Have a great day!" It's not like they openly report their incident metrics.
I have a contact at a vendor for a product I had repeated issues with, and one time they tried a couple of unreleased hotfixes, then asked me to do a full memory dump of the device with an undocumented command. They said they would contact me in a day or two, but that afternoon I got a call back and was told that they had written me a new firmware version. It seems to have fixed the problem completely.

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else

Myrridinos posted:

Snapshot was made last night. Backup went stupid and somehow did two 900GB incremental backups of a virtual machine that has less than 800GB of data stored. This filled the repository and for some reason old restore points were unable to be purged causing the backup to fail. I don't know if that strangeness is related to this issue though.

From what I can gather/guess each time it backed up successfully and merged the snapshot back into the main vhdx the dynamic vhdx grew by 1-2 gb. Over a course of a month this caused the free space to get gobbled up enough that it no longer had enough room to merge the snapshot.

I moved the vhdx and corresponding snapshot to another volume, merged, and got them back up. Tonight we clean up data inside the vhdx and convert it to a fixed disk image before moving it back.

Normally I get an alert long before disk space goes critical like this, but N-central monitoring was never enabled for the volume. I was told our automation should have added it automagically but I guess that's not the case.

Alright that's acceptable I suppose; though I've never heard of a vhdx ballooning out of control like that. What software do you use for backups? Was the disk filling with 0 space that could be hole punched back down? You've got me curious now.

Bunni-kat
May 25, 2010

Service Desk B-b-bunny...
How can-ca-caaaaan I
help-p-p-p you?
*sigh* They federated a student-facing service last night. Here at the staff support desk we've gotten about 30 students contacting us because they can't log in. Phones in actual student facing areas are ringing off the hook, and apparently they've gotten over 150 emails.

Agrikk
Oct 17, 2003

Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

Arquinsiel posted:

No, MI-6 is when your overseas datacenter link drops. You're thinking of MI-5.

Bravo

Myrridinos
Jan 7, 2010

ChubbyThePhat posted:

Alright that's acceptable I suppose; though I've never heard of a vhdx ballooning out of control like that. What software do you use for backups? Was the disk filling with 0 space that could be hole punched back down? You've got me curious now.

Unitrends, I've never seen it do that before either.

I am going to shrink the filesystem and compact the VHDX then convert it to a fixed disk. I have the Unitrends off right now to prevent further issues and I am relying on manual vhdx copies and windows backup to at least have some sort of recovery.

Once I have the vhdx wrangled back to normal I am going to have Unitrends support re-deploy their backup software just so I can say that I didn't screw anything up.

The Muffinlord
Mar 3, 2007

newbid stupie?
I had a dude straight up put the phone down like four times while I was trying to help him with a fairly simple issue this morning. Then I asked him to send us an email so there'd be a paper trail and he hosed up that as well. Too bad another guy already called in sick or I'd be coming down with something right now.

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else

Myrridinos posted:

Unitrends, I've never seen it do that before either.

I am going to shrink the filesystem and compact the VHDX then convert it to a fixed disk. I have the Unitrends off right now to prevent further issues and I am relying on manual vhdx copies and windows backup to at least have some sort of recovery.

Once I have the vhdx wrangled back to normal I am going to have Unitrends support re-deploy their backup software just so I can say that I didn't screw anything up.

Keep me posted on if they give you any answers as to what went wrong here.

AlexDeGruven
Jun 29, 2007

Watch me pull my dongle out of this tiny box


New product being POC'd (not by me)in our environment.

VM requirements: 24c, 96gb RAM, 4TB thick-provisioned disk.

Ok. Seriously. Just gently caress off and say it needs a dedicated server.

Methanar
Sep 26, 2013

by the sex ghost

AlexDeGruven posted:

New product being POC'd (not by me)in our environment.

VM requirements: 24c, 96gb RAM, 4TB thick-provisioned disk.

Ok. Seriously. Just gently caress off and say it needs a dedicated server.

Tell them how vCPU co-scheduling works.

Don't try to give a VM more than like 8 cores at a maximum ever. If it needs more than 8, it needs to be a dedicated piece of hardware.
Also unless there is a licensing concern, give things 8 vCPUs each with 1 core and not 8 vCores on one vCPU.

Zorak of Michigan
Jun 10, 2006


It's nice to hear other people saying that stuff. Every time we push back at someone's demand for a 16 vcpu 128gb RAM VM around here, I get a lot of whining that boils down to, "but Mom would give us a VM that size!" Then we put them on a 2x16, evaluate usage, add 8 more gigs of RAM, and find that it runs fine and the requirements were entirely fictional anyway.

Squatch Ambassador
Nov 12, 2008

What? Never seen a shaved Squatch before?
It would help if vendors stopped listing ridiculous minimum or recommended requirements. Just look at these for SCCM:

Kurieg
Jul 19, 2012

RIP Lutri: 5/19/20-4/2/20
:blizz::gamefreak:
Yes our software needs 102GB of ram and 20 cores.

Why are you laughing?
Stop laughing!

The Iron Rose
May 12, 2012

:minnie: Cat Army :minnie:

Hungry Computer posted:

It would help if vendors stopped listing ridiculous minimum or recommended requirements. Just look at these for SCCM:



:laffo:

SCCM is a beast that will happily use as much RAM as you give it, but <10000 endpoints and you can make do just fine with 32 or 64 gb of RAM on your CAS with the sql database installed locally.

That being said sccm is also the weird beast where things will just fail in weird and unexpected ways and giving it more RAM will definitely ease the load on your poor SCCM admins.

Adbot
ADBOT LOVES YOU

Bunni-kat
May 25, 2010

Service Desk B-b-bunny...
How can-ca-caaaaan I
help-p-p-p you?
An email came in:

quote:

From: I cannot access my accounts. It will not allow me to reset my passwords. Thank you [mailto:user@contoso.com]
Sent: Friday, May 4, 2018 2:08 PM
To: Service Desk <us@contoso.com>
Subject: account access

HOW DID A USER MANAGE TO CHANGE THE FROM FIELD?! DEVILRY!

Also, they sent that FROM their account. So, uh.

  • 1
  • 2
  • 3
  • 4
  • 5