Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
potato of destiny
Aug 21, 2005

Yeah, welcome to the club, pal.
A ticket came in!

"Microsoft called <user> directly to tell her that pc has been notifying them of 8300 events in the event viewer & all 3 pc's need to be looked per Microsoft".

Helpdesk tech apparently uncritically took the call and escalated to the onsite desktop tech, who escalated to my team (tier 3 enterprise management) with the sole note being "Any suggestions".

:cripes:

I mean, I have some suggestions, but I somehow doubt they'd be taken that well...

Adbot
ADBOT LOVES YOU

skooma512
Feb 8, 2012

You couldn't grok my race car, but you dug the roadside blur.

Caged posted:

THIS IS AFFECTING PRODUCTION!

PATIENT CARE AFFECTED :frogsiren:

peak debt
Mar 11, 2001
b& :(
Nap Ghost

skooma512 posted:

PATIENT CARE AFFECTED :frogsiren:

Have you tried to turn the artificial lung off and on again?

Casull
Aug 13, 2005

:catstare: :catstare: :catstare:

potato of destiny posted:

A ticket came in!

"Microsoft called <user> directly to tell her that pc has been notifying them of 8300 events in the event viewer & all 3 pc's need to be looked per Microsoft".

Helpdesk tech apparently uncritically took the call and escalated to the onsite desktop tech, who escalated to my team (tier 3 enterprise management) with the sole note being "Any suggestions".

:cripes:

I mean, I have some suggestions, but I somehow doubt they'd be taken that well...

Set up a VM and gently caress with them? :buddy:

A c E
Jun 18, 2007

Is this weird? Is this too weird? Do you need to sit down?

Casull posted:

Set up a VM and gently caress with them? :buddy:

This. I have one waiting for it, not that they will call because they'd have to go through my PBX (and it's not like my numbers are listed anywhere) but if they ever do I will have so much fun playing dumb and toying with them as long as I can.

xpander
Sep 2, 2004
A ticket is never going to come in again.

Just had a meeting with a local IT shop that's gonna handle both day-to-day support as well as a bunch of our system administration stuff. Management is fully onboard with the cost. I am never going to have to touch a PST again. I've been waiting two and a half years to get out of support here and move into something more interesting.

Feels good man.

skooma512
Feb 8, 2012

You couldn't grok my race car, but you dug the roadside blur.

peak debt posted:

Have you tried to turn the artificial lung off and on again?

Biomed issue. Out of scope for IT. Issue resolved.

myron cope
Apr 21, 2009

I don't get tickets that say "this is affecting production" but basically every email or ticket our voicemail says I NEED THIS NOW TO DO <something> so it's the same thing. It's amazing how everything is URGENT. Have a problem that needs fixed but not IMMEDIATELY? Just wait until a day/hour before you actually need it and then tell us...and mention it's been broken for a few weeks and now you've told your district manager about it because WE NEED A SOLUTION

I feel like this happens everywhere though. Why? Is it just a lack of basic human decency?

Knormal posted:

Do these people call up book and newspaper publishers to complan that their print is too small?

Yeah, almost certainly.

arnbiguous
Feb 2, 2014
Gary’s Answer


This thing I stole from the DnD pics thread sums up my philosophy on helping people at work who don't know how to use computers pretty well.

evol262
Nov 30, 2010
#!/usr/bin/perl

Theresa Frontpage posted:



This thing I stole from the DnD pics thread sums up my philosophy on helping people at work who don't know how to use computers pretty well.

Why is Jewish philosophy in the DnD thread?

arnbiguous
Feb 2, 2014
Gary’s Answer
Because you can't post in there without posting a picture, and people run out of topical pictures pretty fast when they feel like they absolutely must have an argument in the pics thread.

Galler
Jan 28, 2008


That chart seems to be missing 0: gently caress Off. I'm not giving poo poo.

guppy
Sep 21, 2004

sting like a byob

myron cope posted:

I feel like this happens everywhere though. Why? Is it just a lack of basic human decency?

Simply put, because they do not respect your time.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Working for a MSP has put a lot into perspective for me. Allow me to explain.

Do you think it's infuriating when users complain about things like policies, password expiry's, mailbox size restrictions, etc? Wait until you tell a client that such limitations are a necessity to keep their assets secure and working. They reject that and make you give everyone local admin, remove mailbox sizes, etc, then their SBS hits the total exchange mailstore limit mail stops and it's YOUR fault their server stopped. Someone gets a virus(because they have local admin), which results in their IP getting blacklisted, and it's YOUR fault they can't email anyone. Also, how would you like it if a client got a cryptolocker infection because of lax security policies, and you had to restore their server from backups due to it nuking all their fileshares, THEN complain like hell about the bill to fix it even though you have it in writing that they refused to allow you to make steps to mitigate the damage in the first place.

The only reason I haven't gone completely insane yet is calls = money. Disasters = money. In MSP, the stupid rear end calls that result from people completely ignoring instructions are just easy billable hours. It took me almost a year of pure blind rage to figure that out. Corporate IT seems to be all about reducing issues. MSP seems to be about giving the client what they want, after informing them of the risks, then collecting the bill if/when it goes tits up.

I will soon transition back into the corporate IT world, provided a certain interview goes well, but I thought I had seen the worst of the "problem user" until I dealt with one that's paying for it, that refuses good advice, and then holds me accountable for the disaster when it happens. It's really a different world when the dumbasses are paying you.

arnbiguous
Feb 2, 2014
Gary’s Answer

Farking Bastage posted:

Working for a MSP has put a lot into perspective for me. Allow me to explain.

When you say stuff like "holds you accountable" and "blames you" you mean that they get angry and yell at you, not actually get any money/free work out of you, right? The latter would be pretty infuriating.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!

Theresa Frontpage posted:

When you say stuff like "holds you accountable" and "blames you" you mean that they get angry and yell at you, not actually get any money/free work out of you, right? The latter would be pretty infuriating.

A good bit of it ends up being forgiven on the bill. However, the accounting people handle all that. Generally, they rail on whoever answers the phone. I'll post a pertinent story in a sec.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Story time:

A new client contacted us after the person who set them up/supported them dropped off the face of the Earth. There are several of these one man shops around, and we have picked up several of their clients when they either can't respond to or can't handle a problem. Our sales folks set them up with a block of hours ( like 20@100 a pop ) after we handled a minor emergency for them. They have a setup that's pretty awful. It's a Dell 2850 running ESX4 that hosts a single SBS 2003 VM. It's as if they guy who set it up wanted to be able to put vmware on his resume without totally bullshitting. No reason for it whatsoever. Total Piece of poo poo that we immediately recommended for replacement, of course they said no.

Some time later, they call us because the LTO3 tape drive is acting up. Amazingly, they maintained the warranty on that box. I had Dell ship a replacement and took it to their site. Gracefully shut down the VM, then gracefully shut down the hypervisor, popped in the new tape deck and brought it back up....Virtual disk 0 not found, FFFFFFUUUUUCCCCKKKK. ESX booted, but the datastore containing the VHD for the SBS2003 VM under this monstrosity is corrupted as all hell and unrepairable. Turns out, they had a failed disk in a RAID 5 that no one knew or told us about( both LED's turned out to be inoperable on the tray). The reboot to replace the tape deck resulted in a second disk being kicked out of the array. Uh-oh...

Understandably, the customer is freaking the gently caress out. Their backups had been failing silently ( 2003 NTbackup) for months, way before we took over. So, we have a dead server, hosed up backups, with an extra layer of VMware just to make poo poo more complicated. poo poo.

I initially tried to export the datastore via v-sphere client, but they had the free version of ESX, and it was throttled. ETA on 2 TB of datastore was about 2 weeks. Since it was an old ESX4 host, attaching a large USB HDD was an exercise in futility. I ended up upgrading it to 5.1 ( USB disk support), attaching the disk, mounting a backup exec image on the VM, backing it up to the USB disk, then pulling about 95% of the corrupted data from it intact.

When I finished the recovery, a couple of things became evident: 1: Their copy of SBS2003 was torrented and wasn't licensed(thanks old guy). 2: Their copy of Blackberry Enterprise was either pirated or not documented. 3: The server still has a hosed up hardware problem

Dell, after a couple more days worth of calls, troubleshooting, and arguments, replaces 3 disks, the backplane, and the PERC. All that time being billable on my part. The client declines any additional cost to upgrade the server or the OS. I am tasked with rebuilding this loving thing with a Microsoft partner-gifted copy of SBS 2003 we had laying around.

I rebuilt the box with SBS2003 running natively, merged all their email, they all had cached exchange enabled, so no email lost. When the server died, I moved their MX to GFI Maxmail to cache all incoming mail while it was dead. I restored 1.5 TB of critically important files, set up a new BES installation with a legit license, rebuilt the AD domain, and migrated all their desktop profiles to the new SID's.

All in all, they didn't lose so much as a mail message. I absolutely saved their collective asses. I'll continue tomorrow.. too late

The final toll was 65 hours

Farking Bastage fucked around with this message at 02:26 on Feb 8, 2014

incoherent
Apr 24, 2004

01010100011010000111001
00110100101101100011011
000110010101110010
So it cost them 6 grand to just restore what they had? The problem here is you didn't Bilk them enough.

incoherent fucked around with this message at 07:12 on Feb 7, 2014

Qtotonibudinibudet
Nov 7, 2011



Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

myron cope posted:

I don't get tickets that say "this is affecting production" but basically every email or ticket our voicemail says I NEED THIS NOW TO DO <something> so it's the same thing. It's amazing how everything is URGENT. Have a problem that needs fixed but not IMMEDIATELY? Just wait until a day/hour before you actually need it and then tell us...and mention it's been broken for a few weeks and now you've told your district manager about it because WE NEED A SOLUTION

I feel like this happens everywhere though. Why? Is it just a lack of basic human decency?


Yeah, almost certainly.

Weirdly, while most of the URGENT CALL YOUR SALES REP AND HAVE THEM RAISE HELL WITH SUPPORT tickets we tend to get are for inconsequential poo poo, when something does go horribly wrong (network border is crashing every reboot, no internet connectivity for the entire site), people are usually surprisingly nonchalant, to the point that I don't realize that the case probably should have been marked as such (which alerts management and such) until after it's fixed.

arnbiguous
Feb 2, 2014
Gary’s Answer
Whoa, holy hell. A ticket came in to my house: I tried to replace my two media drives with a single larger drive and windows wouldn't boot.

I have This Problem. Anybody who is planning on installing win7 any time soon should probably read this. Why the hell would win7 installer put the boot manager on a different drive? aaaghhh

peak debt
Mar 11, 2001
b& :(
Nap Ghost
Because it assumes you are trying to install a multi-boot system. General best practice is to physically disconnect all drives but the one you will install the OS on before doing so.

Collateral Damage
Jun 13, 2009

Or just go into disk manager after you've installed and make sure the boot manager is on your system drive. If it's not you'll need to use BCDboot to copy it.

http://technet.microsoft.com/en-us/library/dd744347(v=ws.10).aspx

anthonypants
May 6, 2007

by Nyc_Tattoo
Dinosaur Gum

Theresa Frontpage posted:

Anybody who is planning on installing win7 any time soon should probably
install Windows 8.1 instead?

arnbiguous
Feb 2, 2014
Gary’s Answer

Collateral Damage posted:

Or just go into disk manager after you've installed and make sure the boot manager is on your system drive. If it's not you'll need to use BCDboot to copy it.

http://technet.microsoft.com/en-us/library/dd744347(v=ws.10).aspx

Wow, thanks, the solution in the comments section I linked didn't actually work. Let's hope this does.

edit: It worked! Excellent. The additional comments on that technet page make me really glad I use G4L to image systems instead of whatever that guy is using that isn't picking up all the partitions on his discs. Sorry to clutter up the thread with personal support junk, I just figured somebody working in a low-end business somewhere who frequently has to install win7 themselves would benefit from my weird experience.

arnbiguous fucked around with this message at 09:13 on Feb 7, 2014

Emushka
Jul 5, 2007
Since the winter olympics are about to start, I thought about blocking some national tv streams of the games and see what kind of excuses/shitstorm I get.


Should be fun.

Collateral Damage
Jun 13, 2009

Personally I don't care if people waste their time watching sportsball when they should be working. I'm not their boss.

If the bandwidth use becomes a noticable problem I might do something but until then it's a management problem, not IT.

Emushka
Jul 5, 2007

Collateral Damage posted:

Personally I don't care if people waste their time watching sportsball when they should be working. I'm not their boss.

If the bandwidth use becomes a noticable problem I might do something but until then it's a management problem, not IT.


I agree, I don't really care what type of working ethic other have as long I can live with mine.


Would be just fun to see what would be the consequence. if any.

arnbiguous
Feb 2, 2014
Gary’s Answer
If I didn't block access to the streaming sites during March Madness the entire sales department literally does nothing. No outbound calls, people letting incoming calls go to voicemail, seriously. It's the only time I've ever been asked to block something by management.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
Continued..

After it was all said and done, the bill came out to around 8500 bucks ( we charge 130 an hour for server support) and they got a free OS out of it. A few days later, the beancounter calls me up to his office. This client disputed almost every time entry I made. They were refusing to pay for time like running the backup of the old system, OS installation time, or Array building time. Apparently, they had their old guy look at the bill and find things that were over his head in the first place to suggest as "fishy".

I think they ended up paying about 50% of what we billed them and every time they have called in since, the owner has always had some snarky comment towards me. He came by to get his new blackberry set up and I got an earful about how we "hosed him."

When this whole thing started, I pushed and pushed for us to just tell them it's dead and they need to get OnTrack to recover the data, however, the management insisted that we fix the problem by any means necessary. I KNEW what kind of a clusterfuck was in store, but they didn't get it.

Yeah gently caress this business.

Proud Christian Mom
Dec 20, 2006
READING COMPREHENSION IS HARD

Farking Bastage posted:

Working for a MSP has put a lot into perspective for me.

The golden time as a MSP is when you've carefully selected and cultivated your clients so they trust you 100% and are willing to do virtually anything you tell them so after its all set up and running smoothly you just sit back all Maytag repairman like and collect those nice monthly checks while occasionally installing updates.

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

The best is when the MSP you rely on has one SCCM resource and then the fucker up and quits. What do you mean I got to learn all this on my own :argh:

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
^^ I'm the only Cisco resource at this one :D


go3 posted:

The golden time as a MSP is when you've carefully selected and cultivated your clients so they trust you 100% and are willing to do virtually anything you tell them so after its all set up and running smoothly you just sit back all Maytag repairman like and collect those nice monthly checks while occasionally installing updates.

We have 4 or 5 of these and they are wonderful. The ones that listen are stable and the money's rolling in. 90% of our stress comes from 10% of our clients.

AlternateAccount
Apr 25, 2005
FYGM

Farking Bastage posted:

Continued..

After it was all said and done, the bill came out to around 8500 bucks ( we charge 130 an hour for server support) and they got a free OS out of it. A few days later, the beancounter calls me up to his office. This client disputed almost every time entry I made. They were refusing to pay for time like running the backup of the old system, OS installation time, or Array building time. Apparently, they had their old guy look at the bill and find things that were over his head in the first place to suggest as "fishy".

I think they ended up paying about 50% of what we billed them and every time they have called in since, the owner has always had some snarky comment towards me. He came by to get his new blackberry set up and I got an earful about how we "hosed him."

When this whole thing started, I pushed and pushed for us to just tell them it's dead and they need to get OnTrack to recover the data, however, the management insisted that we fix the problem by any means necessary. I KNEW what kind of a clusterfuck was in store, but they didn't get it.

Yeah gently caress this business.

Honestly, I would have just said welp, your poo poo is lost. Lesson learned I guess. Maybe go as far as recovering some e-mail, but that's it. There's a point where I will just call something dead on the table rather than take herculean steps to rescue some broken piece of poo poo back to a state where it's just as liable to explode again in less than a year.

ConfusedUs
Feb 24, 2004

Bees?
You want fucking bees?
Here you go!
ROLL INITIATIVE!!





The problem isn't that he did the work, it's that his superiors caved in to the customer's unreasonable and unwarranted demands to lower the bill.

That's a bad precedent to set. It proved the customer 'right' that the charges were inflated and undermines confidence in the quality of work and service.

Polio Vax Scene
Apr 5, 2009



go3 posted:

The golden time as a MSP is when you've carefully selected and cultivated your clients so they trust you 100% and are willing to do virtually anything you tell them so after its all set up and running smoothly you just sit back all Maytag repairman like and collect those nice monthly checks while occasionally installing updates.

Imagine this but being a developer. My pod smells like cinnamon and fresh dollar bills. We have one client that is a special snowflake but they're some loaded medical company that always pays their bills.

Nerdrock
Jan 31, 2006

An email came in from our "content filtering" person :

Good Afternoon All

We are noticing an increased number of lockouts when links within Olympic
sites are selected. This seems to be due to links getting automatically
redirected to Russian porn sites. If the lockouts become problematic, the
lockout duration can be reduced in the rule sets.



So : apparently the olympics website is currently , or was yesterday, hijacked and pointing people to lots of Russian porn sites. Good times.

you ate my cat
Jul 1, 2007

This is the best Olympics. I can't wait for Russian hackers to put malware on the displays for the opening ceremony.

Farking Bastage
Sep 22, 2007

Who dey think gonna beat dem Bengos!
They have been compromising laptops and phones left and right. Redirecting the hotel wireless registration pages to rootkits

QuiteEasilyDone
Jul 2, 2010

Won't you play with me?
And thus the Sochi Cyber-Olympics began with some banging.

Adbot
ADBOT LOVES YOU

MJP
Jun 17, 2007

Are you looking at me Senpai?

Grimey Drawer

Farking Bastage posted:

Working for a MSP has put a lot into perspective for me. Allow me to explain.

Do you think it's infuriating when users complain about things like policies, password expiry's, mailbox size restrictions, etc? Wait until you tell a client that such limitations are a necessity to keep their assets secure and working. They reject that and make you give everyone local admin, remove mailbox sizes, etc, then their SBS hits the total exchange mailstore limit mail stops and it's YOUR fault their server stopped. Someone gets a virus(because they have local admin), which results in their IP getting blacklisted, and it's YOUR fault they can't email anyone. Also, how would you like it if a client got a cryptolocker infection because of lax security policies, and you had to restore their server from backups due to it nuking all their fileshares, THEN complain like hell about the bill to fix it even though you have it in writing that they refused to allow you to make steps to mitigate the damage in the first place.

The only reason I haven't gone completely insane yet is calls = money. Disasters = money. In MSP, the stupid rear end calls that result from people completely ignoring instructions are just easy billable hours. It took me almost a year of pure blind rage to figure that out. Corporate IT seems to be all about reducing issues. MSP seems to be about giving the client what they want, after informing them of the risks, then collecting the bill if/when it goes tits up.

I will soon transition back into the corporate IT world, provided a certain interview goes well, but I thought I had seen the worst of the "problem user" until I dealt with one that's paying for it, that refuses good advice, and then holds me accountable for the disaster when it happens. It's really a different world when the dumbasses are paying you.

This is the literal and complete truth about the MSP sphere. I worked in an NYC MSP for almost two years. Not only is it everything and the above where clients do not care, blame you, and you are forbidden from saving them money by good policies, but the fixes put into place are cowboy as hell. Not enough documenting happened and clients ended up only really trusting senior techs. Who got bonuses for revenue retention, so it wasn't even in their interest to spread knowledge.

Also, mine was so much all-helpdesk that there was no room for advancement. Got my 2k3 and then 2k8 MCSAs and got a "promotion" where instead of being remote all the time and having a reliable schedule, I was doing deskside support in addition. Kind of a step down.

I got out when the getting was good.

ConfusedUs posted:

The problem isn't that he did the work, it's that his superiors caved in to the customer's unreasonable and unwarranted demands to lower the bill.

That's a bad precedent to set. It proved the customer 'right' that the charges were inflated and undermines confidence in the quality of work and service.

This is not a precedent, this is standard MSP practice. I had an hourly client at the MSP just after I started. It was a husband-and-wife capital investment firm. The daughter's ancient XPS laptop was having issues and it ended up that the power jack on the mobo was dead. They wanted us to repair it. Out of warranty, I looked up the cost of the mobo and did a quick calculation of time to do it at $165/hr (the hourly rate we did). I presented the cost to the guy of the time I'd already spent verifying what I could and how much it was worth, and how much it'd be to back up the stuff onto an external disk. At least they listened and bought a new laptop instead of the recovery, but I later found out we'd knocked it down by about 3/4ths. Personal poo poo for non-hourly clients was done for free.

The checks never bounced but we had stupid poo poo/personal poo poo all the time. If nothing else, that job motivated me to get my certs and skill up to make me too valuable to waste time asking a user if they'd REALLY rebooted their computer, and the guys who worked the floor and the phones were great guys. But it was one giant pyramid scheme of knowledge and bonuses.

MJP fucked around with this message at 17:08 on Feb 7, 2014

  • Locked thread