Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
mllaneza
Apr 28, 2007

Veteran, Bermuda Triangle Expeditionary Force, 1993-1952




darthbob88 posted:

In defense of all those, it's significantly easier to restore a computer to a known good state than to repair it from an unknown bad state. I can either check which policies have gotten corrupted or how the IP configuration is broken or which of a thousand different files is a trojan, or I can just wipe it all and go back to something I know works and isn't corrupt.

That being said, if they just apply those methods without thinking about it at all, yeah it's a cargo cult.


Gpupdate is an essential tool in a managed environment.
Rebooting really does cure a multitude of ills.

Flatten and re-image is the only way to go in a very large environment, or anywhere there's a shortage of engineering talent to really dig into the cause of the one weird issue on one laptop. Root cause investigations on one-off issues are luxuries that you pay for with the time of your most skilled desktop engineers. You want them spending their time making sure the image is stable, not fixing isolated issues. The key issue is, is a re-image faster than spending a lot of time researching an issue ? Your priority should be getting the user back to work as fast as possible.

Slow day ? Interesting problem ? Already got the user back to work somehow ? Go for it, dig deep. But any issue that survives a re-image to a known good state either has some weird interaction of non-standard apps, or is a hardware issue. And if it looks like hardware, swap drives to a good system to rule out software.

Adbot
ADBOT LOVES YOU

peak debt
Mar 11, 2001
b& :(
Nap Ghost
The first time a weird issue happens that I can't fix in 15 minutes, I have the PC reimaged. It's just not worth the time delving deep into event logs, procmon, and similar poo poo to find out why it happened simply to satisfy my curiosity. If the same issue shows up a few times then that's justification for spending a half day on trying to find out which arcane configuration item exactly causes it, if only to save time on future incidents because I can forward the solution to 2nd level.

Proteus Jones
Feb 28, 2013



You both are missing the point.

It's not that those aren't useful or that they don't solve problems. It's that for some people that is the ONLY way they know to solve problems. They have absolutely no understanding other than a rote recitation of commands or mouse clicks. And when the time comes where those tools are the wrong thing for a particular problem, these people are ineffectual at best or actively exacerbating the issue at worst.

RFC2324
Jun 7, 2012

http 418

flosofl posted:

You both are missing the point.

It's not that those aren't useful or that they don't solve problems. It's that for some people that is the ONLY way they know to solve problems. They have absolutely no understanding other than a rote recitation of commands or mouse clicks. And when the time comes where those tools are the wrong thing for a particular problem, these people are ineffectual at best or actively exacerbating the issue at worst.

To test them, ask them to explain what the switches on the commands do. If they can answer, they are just going easy way, if they can't they are just cargo-culting

mllaneza
Apr 28, 2007

Veteran, Bermuda Triangle Expeditionary Force, 1993-1952




flosofl posted:

You both are missing the point.

It's not that those aren't useful or that they don't solve problems. It's that for some people that is the ONLY way they know to solve problems. They have absolutely no understanding other than a rote recitation of commands or mouse clicks. And when the time comes where those tools are the wrong thing for a particular problem, these people are ineffectual at best or actively exacerbating the issue at worst.

Well that's certainly true. We had an outbreak of "reset it to factory default" in our desk side group a while back. Factory default on Elitebooks is UEFI boot, and we use Legacy. A reset makes it unbeatable, and also triggers Bitlocker. Every one of those tickets had to go to the hardware group who knew the correct BIOS settings and could retrieve Bitlocker keys. They did a printer too, goodbye name and DNS entry ! Also, the third desk side visit to that printer ended up with the transfer roller removed. Why yes ! It is printing very faintly !

RFC2324 posted:

To test them, ask them to explain what the switches on the commands do. If they can answer, they are just going easy way, if they can't they are just cargo-culting


Good system !

RFC2324
Jun 7, 2012

http 418

mllaneza posted:

Good system !

Full disclosure, I always double check the man page to double check what switches do.

Agrikk
Oct 17, 2003

Take care with that! We have not fully ascertained its function, and the ticking is accelerating.
Ever get a case/ticket that you know is the leading edge of a very massive shitstorm, but the case creator just doesn't know it yet?

I got a support case escalated to me, followed by a very nervous phone call by <customer> VP of Operations saying basically the same thing:

"Hey Agrikk, we are running low on IP addresses in our Production VPC. Can you please increase the number of available IP addresses?"

So I take a look at the VPC in question and it has a /20 mask specified. and almost all of the four thousand IP addresses are in use. Welp!


Edit:

For those who might not know, it is currently impossible to resize a CIDR block on an existing VPC. Choosing the right-sized CIDR is literally The One Thing You'd Better Not gently caress Up when examining your VPC design for just this very reason.

AWS has a helpful doc on the subject:

quote:

You can't change the size of a VPC after you create it. If your VPC is too small to meet your needs, create a new, larger VPC, and then migrate your instances to the new VPC. To do this, create AMIs from your running instances, and then launch replacement instances in your new, larger VPC. You can then terminate your old instances, and delete your smaller VPC. For more information, see Deleting Your VPC.

That's it, just snapshot all of the production instances of a $XX billion company and flip them over to a new VPC! What could possibly go wrong with migrating and re-IP-ing several thousand EC2 instances, RDS databases and service endpoints?

Yeah, this is going to be a fun conversation to have. But at least I know what I get to work on for the next few months.

Agrikk fucked around with this message at 20:14 on Apr 20, 2017

pr0digal
Sep 12, 2008

Alan Rickman Overdrive

Agrikk posted:

Ever get a case/ticket that you know is the leading edge of a very massive shitstorm, but the case creator just doesn't know it yet?

I got a support case escalated to me, followed by a very nervous phone call by <customer> VP of Operations saying basically the same thing:

"Hey Agrikk, we are running low on IP addresses in our Production VPC. Can you please increase the number of available IP addresses?"

So I take a look at the VPC in question and it has a /20 mask specified. and almost all of the four thousand IP addresses are in use. Welp!


Edit:

For those who might not know, it is currently impossible to resize a CIDR block on an existing VPC. Choosing the right-sized CIDR is literally The One Thing You'd Better Not gently caress Up when examining your VPC design for just this very reason.

AWS has a helpful doc on the subject:


That's it, just snapshot all of the production instances of a $XX billion company and flip them over to a new VPC! What could possibly go wrong with migrating and re-IP-ing several thousand EC2 instances, RDS databases and service endpoints?

Yeah, this is going to be a fun conversation to have. But at least I know what I get to work on for the next few months.

Holy poo poo.

We're getting pretty close to becoming an AWS partner so this is going to be a very good thing to remember.

MF_James
May 8, 2008
I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

Agrikk posted:

Ever get a case/ticket that you know is the leading edge of a very massive shitstorm, but the case creator just doesn't know it yet?

I got a support case escalated to me, followed by a very nervous phone call by <customer> VP of Operations saying basically the same thing:

"Hey Agrikk, we are running low on IP addresses in our Production VPC. Can you please increase the number of available IP addresses?"

So I take a look at the VPC in question and it has a /20 mask specified. and almost all of the four thousand IP addresses are in use. Welp!


Edit:

For those who might not know, it is currently impossible to resize a CIDR block on an existing VPC. Choosing the right-sized CIDR is literally The One Thing You'd Better Not gently caress Up when examining your VPC design for just this very reason.

AWS has a helpful doc on the subject:


That's it, just snapshot all of the production instances of a $XX billion company and flip them over to a new VPC! What could possibly go wrong with migrating and re-IP-ing several thousand EC2 instances, RDS databases and service endpoints?

Yeah, this is going to be a fun conversation to have. But at least I know what I get to work on for the next few months.

I would say this does not only apply to your instance, but just in general, it's a pain in the rear end to deal with something like this when you could have just planned bigger from the start. You current needs are a /26, so you think a /24 is good enough? gently caress it, make it a /16 or whatever.

hihifellow
Jun 17, 2005

seriously where the fuck did this genre come from
Previous job had a bunch of small satellite sites with like their own /30 carved out, almost all sequentially, so you'd have to re-ip everything to a different subnet instead of just expanding it.

Guess what happened when they went from analog to voip?

Bunni-kat
May 25, 2010

Service Desk B-b-bunny...
How can-ca-caaaaan I
help-p-p-p you?
I'm ornery today, and the users aren't improving my mood.

Special Needs User posted:

Subject: Printer connection down
Body:
Thanks. Very helpful in troubleshooting. This is one of our special snowflake Mac users. Strongly dislike the "just fix it" attitude she has.

Next on the hit parade, a user who had to be told twice that access to an application wasn't handled by my Service Desk, but by filling out a form that was then automatically sent to a security review team for that application, since it deals with a lot of PII. Then they forgot their password to a different application, I reset their password with a flag to expire it on next log-in. It took 3 emails to direct her on how to change her password after that when the screens are:
1. Standard log in screen
2. Screen saying password is expired and must be changed, with big "Click here to change password" link.
3. Screen with fields for current password, new password, repeat new password, submit.
That's it. That's all that's on all 3 of those screens.

Then we got a user who emailed wondering what the criteria to use on-site printers, and when to use the 3rd party printing service we have are. That's not so bad, except for directing it to Facilities instead of any IT department. Head of Service Desk answered her "There are no criteria, use your judgment." About 20 minutes later we get another email asking what the criteria are. As a reply to my boss's email.

People are extra stupid today.

ETA: so, the second user managed to lock herself out of her computer account. It was reset, and then she managed to fail to set a new password on that.

Bunni-kat fucked around with this message at 21:31 on Apr 20, 2017

Super Slash
Feb 20, 2006

You rang ?

Avenging_Mikon posted:

People are extra stupid today.
This was a good thing today as our hosted VOIP dropped out and people were happy for a change rather than going nuclear, even better as it was fixed within 10 minutes.

Aunt Beth
Feb 24, 2006

Baby, you're ready!
Grimey Drawer

Avenging_Mikon posted:

People are extra stupid today.
It is 420 today... :350:

Bunni-kat
May 25, 2010

Service Desk B-b-bunny...
How can-ca-caaaaan I
help-p-p-p you?

Aunt Beth posted:

It is 420 today... :350:

I just thought people would wait until after lunch! :catdrugs:

jre
Sep 2, 2011

To the cloud ?



Agrikk posted:

Ever get a case/ticket that you know is the leading edge of a very massive shitstorm, but the case creator just doesn't know it yet?

I got a support case escalated to me, followed by a very nervous phone call by <customer> VP of Operations saying basically the same thing:

"Hey Agrikk, we are running low on IP addresses in our Production VPC. Can you please increase the number of available IP addresses?"

So I take a look at the VPC in question and it has a /20 mask specified. and almost all of the four thousand IP addresses are in use. Welp!


Edit:

For those who might not know, it is currently impossible to resize a CIDR block on an existing VPC. Choosing the right-sized CIDR is literally The One Thing You'd Better Not gently caress Up when examining your VPC design for just this very reason.

AWS has a helpful doc on the subject:


That's it, just snapshot all of the production instances of a $XX billion company and flip them over to a new VPC! What could possibly go wrong with migrating and re-IP-ing several thousand EC2 instances, RDS databases and service endpoints?

Yeah, this is going to be a fun conversation to have. But at least I know what I get to work on for the next few months.

Why would you migrate them rather than just use cloudformation to bring up new machines in the new vpc ? If you have several thousand pets you're doing the cloud thing wrong

Agrikk
Oct 17, 2003

Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

jre posted:

Why would you migrate them rather than just use cloudformation to bring up new machines in the new vpc ? If you have several thousand pets you're doing the cloud thing wrong

Who said anything about pets? It has nothing to do with the servers themselves. Those get shot in the head over the course of several days. but setting up routing into and out of the VPC from your global operations and your network of partners and vendors is non-trivial. Also migrating terabytes and petabytes of data from RDS instances into your new VPCs RDS instances. Also vetting and testing your new infrastructure, etc.

Yes, I built my data center in five minutes. But moving existing data there, hooking up connections and testing all the interoperability will take weeks at the minimum.

madsushi
Apr 19, 2009

Baller.
#essereFerrari

MF_James posted:

I would say this does not only apply to your instance, but just in general, it's a pain in the rear end to deal with something like this when you could have just planned bigger from the start. You current needs are a /26, so you think a /24 is good enough? gently caress it, make it a /16 or whatever.

I mean, there's only so many private /16's to go around. I've seen places assign /16's to every network, and eventually you're taking up a good chunk of RFC 1918 addresses and connecting to another company's network probably involves bidirectional NAT or something.

BallerBallerDillz
Jun 11, 2009

Cock, Rules, Everything, Around, Me
Scratchmo

RFC2324 posted:

Full disclosure, I always double check the man page trial-and-error to double check what switches do.

Cumslut1895
Feb 18, 2015

by FactsAreUseless
poo poo, when do you start using IPV6

Thanks Ants
May 21, 2004

#essereFerrari


Cumslut1895 posted:

poo poo, when do you start using IPV6

The answer is always "it's about a year or two away"

Monolith.
Jan 28, 2011

To save the world from the expanding Zone.
It finally happened: user has 10k emails in his deleted items box.

"Yeah, maybe I shouldn't use this as a filing place."

Yeah, maybe you shouldn't.

Judge Schnoopy
Nov 2, 2005

dont even TRY it, pal

Monolith. posted:

It finally happened: user has 10k emails in his deleted items box.

"Yeah, maybe I shouldn't use this as a filing place."

Yeah, maybe you shouldn't.

export to PST and bomb it from the exchange side. Let her panic for two hours and then rescue her with a "miracle" fix that shouldn't be relied upon because it won't be available next time.

Monolith.
Jan 28, 2011

To save the world from the expanding Zone.

Judge Schnoopy posted:

export to PST and bomb it from the exchange side. Let him panic for two hours and then rescue him with a "miracle" fix that shouldn't be relied upon because it won't be available next time.

Nah the user is way too uptight; it would be amusing to watch though. Now they have a pst (9.2 gigs!)

fishmech
Jul 16, 2006

by VideoGames
Salad Prong

Cumslut1895 posted:

poo poo, when do you start using IPV6




(note that if you zoom in on the graph, you find that IPv6 usage is always lowest on Mondays and highest on Saturdays, presumably because IPv6 is more commonly deployed on residential connections than existing corporate infrastructure. So last Saturday we had Native: 17.44% 6to4/Teredo: 0.04% Total IPv6: 17.48%)

Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin
Aren't most phones on ipv6?

fishmech
Jul 16, 2006

by VideoGames
Salad Prong

Dr. Arbitrary posted:

Aren't most phones on ipv6?

Depends on carrier and how old the phone is. For instance on Verizon Wireless these days, you will still get IPv4 only service in some areas because the local equipment was not actually able to handle IPv6 routing at all, or had such problems that it's temporarily disabled. Most other carriers have some level of problem like that from place to place. And then of course you have things like some older phones not supporting IPv6 at all, or only supporting them with updates the users haven't applied, though that's becoming rarer each day.

spankmeister
Jun 15, 2008






Dr. Arbitrary posted:

Aren't most phones on ipv6?

They don't have to be, carriers can just NAT

The Fool
Oct 16, 2003


fishmech posted:

Depends on carrier and how old the phone is. For instance on Verizon Wireless these days, you will still get IPv4 only service in some areas because the local equipment was not actually able to handle IPv6 routing at all, or had such problems that it's temporarily disabled. Most other carriers have some level of problem like that from place to place. And then of course you have things like some older phones not supporting IPv6 at all, or only supporting them with updates the users haven't applied, though that's becoming rarer each day.

I know that for AT&T in my region, most cellular data devices are behind an IPv4 NAT.

Truga
May 4, 2014
Lipstick Apathy

flosofl posted:

You both are missing the point.

It's not that those aren't useful or that they don't solve problems. It's that for some people that is the ONLY way they know to solve problems. They have absolutely no understanding other than a rote recitation of commands or mouse clicks. And when the time comes where those tools are the wrong thing for a particular problem, these people are ineffectual at best or actively exacerbating the issue at worst.

Also, because microsoft support will just tell you to do a reboot, and then flatten and restore from backup when you have any problem you couldn't have solved yourself by looking at the event log and fixing the simple error.

Boogalo
Jul 8, 2012

Meep Meep




Monolith. posted:

It finally happened: user has 10k emails in his deleted items box.

"Yeah, maybe I shouldn't use this as a filing place."

Yeah, maybe you shouldn't.

We just migrated from onprem to O365 and a user is complaining that all of the old mails she kept in her junk folder are gone. We hold her 2 weeks ago that poo poo is gone and she's asking again. poo poo's still gone.

I've seen filing trees in deleted items but never in junk before.

pixaal
Jan 8, 2004

All ice cream is now for all beings, no matter how many legs.


Boogalo posted:

We just migrated from onprem to O365 and a user is complaining that all of the old mails she kept in her junk folder are gone. We hold her 2 weeks ago that poo poo is gone and she's asking again. poo poo's still gone.

I've seen filing trees in deleted items but never in junk before.

Well junk, you know, it means stuff. It's where the stuff goes

stevewm
May 10, 2005

Monolith. posted:

It finally happened: user has 10k emails in his deleted items box.

"Yeah, maybe I shouldn't use this as a filing place."

Yeah, maybe you shouldn't.

I had a user that was notorious for doing this... One day while trying to fix her mail client yet again because she had too many items that Thunderbird started crashing (like 20k I think) I took a stack of important looking papers on her desk and asked if they were important, when she said yes, I threw them in the trash can under her desk. I asked if that is the correct place to put important papers, to which she answered no.. So I said well neither is the Trash folder in your email.

The Fool
Oct 16, 2003


stevewm posted:

I had a user that was notorious for doing this... One day while trying to fix her mail client yet again because she had too many items that Thunderbird started crashing (like 20k I think) I took a stack of important looking papers on her desk and asked if they were important, when she said yes, I threw them in the trash can under her desk. I asked if that is the correct place to put important papers, to which she answered no.. So I said well neither is the Trash folder in your email.

I've heard the story a few times from different people. I don't believe anyone has actually done it.

nielsm
Jun 1, 2009



stevewm posted:

I had a user that was notorious for doing this... One day while trying to fix her mail client yet again because she had too many items that Thunderbird started crashing (like 20k I think) I took a stack of important looking papers on her desk and asked if they were important, when she said yes, I threw them in the trash can under her desk. I asked if that is the correct place to put important papers, to which she answered no.. So I said well neither is the Trash folder in your email.

Well, did it work?

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

We have policy so the deleted items is emptied every 7 days. Hard to store poo poo in there with that going on.

Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin
Deleted items go away when you close outlook for me.

stevewm
May 10, 2005

The Fool posted:

I've heard the story a few times from different people. I don't believe anyone has actually done it.

I can assure you I did... but we have a very relaxed workplace here. We all frequently call each other names, screw with others desks, pranks, etc.. So me doing that wasn't taken seriously. She laughed about it.

And no, it didn't work :( She still keeps poo poo there.

I need to just convert the last few stragglers over to Gmail. Almost entirely rid of local mail clients!

stevewm fucked around with this message at 21:46 on Apr 21, 2017

ConfusedUs
Feb 24, 2004

Bees?
You want fucking bees?
Here you go!
ROLL INITIATIVE!!





The Fool posted:

I've heard the story a few times from different people. I don't believe anyone has actually done it.

One day I was sitting at my desk in my home office, and I noticed the can was full. So I threw it out.

Later that day my wife asked me where all the trash went, and I said "I threw it out."

Turns out she had put all the receipts for her recent clothing purchases in my desk trash and needed to return something. She was a little angry about it, and my laughter didn't help any.

So while I can't claim I've ever done the deleted items folder comparison to trash can thing, I can unequivocally say that my wife would be able to say "yes" when I asked if she felt the trash can was the appropriate place for something she might need later.

baquerd
Jul 2, 2007

by FactsAreUseless

Dr. Arbitrary posted:

Deleted items go away when you close outlook for me.

Who closes outlook outside of mandated or voluntary restarts though?

Adbot
ADBOT LOVES YOU

Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin
Me all the time because I'm clumsy and accidentally hit the x instead of the -

But we also have mandatory reboots.

  • Locked thread