Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
AndyElusive
Jan 7, 2007

Yay, I passed my 901 exam for CompTIA A+ certification.

Kind of insane how easy it would have been for me to cheat this test because the examiner put me in a room by myself, with a camera she told me she didn't watch, and a cup of coffee with a sleeve that I could have used to stash some answers behind for bullshit questions I didn't want to remember.

Adbot
ADBOT LOVES YOU

Corsair Pool Boy
Dec 17, 2004
College Slice

Vargatron posted:

Yeah, but the ideal situation would have been to free up space by shrinking VMDKs or selectively disabling other non essential VMs. That way we could have restored business critical services and then formed a plan to clean up the snapshots at a less critical time. At the end of the day the commits would happen, but it looks a bit better if you can go to management with "hey, we're going to have to do a 12 hour maintenance window on Sunday" as opposed to "yeah we lost 3rd shift production and most of 1st due to a systems issue".

Makes sense, ty

Thanks Ants
May 21, 2004

#essereFerrari


It’s also entirely not your fault if the self-declared expert on the system didn’t set it up properly, didn’t monitor anything, and then didn’t respond to you trying to escalate. Meanwhile you’re getting people breathing down your neck and suggesting they’re going to hold you responsible for lost production if you don’t do *something*.

By all means learn from the experience but don’t feel like you reacted in the wrong way at the time.

Vargatron
Apr 19, 2008

MRAZZLE DAZZLE


Yeah I kind of look back and laugh at the situation now. It was just one of many "OH GOD THE BUILDING IS BURNING DOWN" scenarios I had the pleasure of being involved in.

Sheep
Jul 24, 2003

AndyElusive posted:

Yay, I passed my 901 exam for CompTIA A+ certification.

Good job!

AndyElusive
Jan 7, 2007

Sheep posted:

Good job!

Thanks. It's my first real step in this line of work and I know it must be small potatoes to everyone in this thread.

Now to pass 902.

Docjowles
Apr 9, 2009

Reminds me of the job I took at a tiny development shop as the sole sysadmin in ~2012. Within the first week I was there, the main file server fell over. It was due to the ZFS-on-Linux filesystem being allowed to grow to 99% full. This makes ZFS extremely sad, especially the poo poo-rear end 2012 port of ZFS, and there were no backups. My boss held the previous admin in the absolute highest regard due to his cool looking Tmux setup (I poo poo you not, this is what he cited to demonstrate to me what a towering genius my predecessor was), so obviously this was my fault. I don't think he ever really forgave me for this incident, and I left after about a year of him micromanaging my every move for a way better job.

There are a lot of very bad "experts" out there taking credit for the shitheaps they've built and made others take the fall for supporting. Glad you've likewise been able to move on so you can look back and laugh.

jaegerx
Sep 10, 2012

Maybe this post will get me on your ignore list!


You were running zfs in 2012? On Linux? Yeah that company deserved what it got.

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

YOLOsubmarine posted:

ZFS will silently correct things like bit rot by rebuilding from parity data when a checksum fails or a read error occurs, and scrubs will catch sectors that are going bad and re-allocate them, so unless you're actively looking for this information, for instance by looking at zpool status to see the scrub status, you wouldn't ever know about it. Modern disk controllers will also do a lot of work to retry writes or correct bad sectors before they ever get to the requesting host. And that 10^14 number is a worst case scenario and over the lifetime of the drive. So there's a lot of reasons why everyone's raid arrays aren't blowing up like they were supposed to.

RIP if you are using ZFS in prod and you don’t have something watching ur zpool status and usage.

Like even zol ships zed now.

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

Granny RAIDing, not erasure encoding like you should.

jaegerx
Sep 10, 2012

Maybe this post will get me on your ignore list!


Rip using zfs on Linux at all. Hasn’t been touched in years. Betterfs was supposed to be our savior but even redhat ditched that.


Everyone back to murderwifefs

Docjowles
Apr 9, 2009

I wouldn't say "we" were, since that implies I had any involvement in the choice, but yeah. Let me also tell you about the virtualization environment built on loving Ganeti (what, you've never heard of it?) using DRBD volumes for "HA storage". Or the old-even-then HP MSA SANs I was not allowed to ever, ever patch or reboot despite known bugs that would crash them after X days of uptime. Because "one time an update went bad".

...actually, do not let me tell you that, because I will get very mad and chug a fifth of bourbon.

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer

jaegerx posted:

Everyone back to murderwifefs
I have a char in eve named Hans Reiser and his bio talks about murdering his family. Good stuff.

edit: for real content, I still run openindiana at home as it IMO the epitome of an awesome open source storage device.

adorai fucked around with this message at 03:11 on Jun 15, 2018

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

jaegerx posted:

Rip using zfs on Linux at all. Hasn’t been touched in years. Betterfs was supposed to be our savior but even redhat ditched that.


Everyone back to murderwifefs

Eh, ZoL dev is active and stays in sync with openzfs upstream. Your best option if you want to run zfs on modern hardware.

But be sure you have backups; also reconsider if you haven’t had a beer with a zfs dev or had a beer with someone whose had a beer with a zfs dev.

XFS is great as long as you don’t need performant deletes.

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

adorai posted:

I have a char in eve named Hans Reiser and his bio talks about murdering his family. Good stuff.

edit: for real content, I still run openindiana at home as it IMO the epitome of an awesome open source storage device.

Is OI still one dude trying to rebase from Illumos every few years?

When you make the OmniOS community look lively...

Schadenboner
Aug 15, 2011

by Shine

jaegerx posted:

Rip using zfs on Linux at all. Hasn’t been touched in years. Betterfs was supposed to be our savior but even redhat ditched that.


Everyone back to murderwifefs

How did Hans Reiser dispose of his wife’s body?

Distributed it over several trees.

Wibla
Feb 16, 2011

PCjr sidecar posted:

Eh, ZoL dev is active and stays in sync with openzfs upstream. Your best option if you want to run zfs on modern hardware.

But be sure you have backups; also reconsider if you haven’t had a beer with a zfs dev or had a beer with someone whose had a beer with a zfs dev.

XFS is great as long as you don’t need performant deletes.

I'm not a fan of ZFS myself, due to not being able to grow an array with a drive at a time (though they were looking into changing that a while back).

I've been running my home poo poo on linux + raid + XFS since forever, starting out in 2007 with 3ware 9500S cards and XFS and then later on SAS HBAs, mdraid6 and XFS... it just works, haven't had any dataloss since I installed my first server.

XFS is plenty fast, but deleting small files is a drag.

DropsySufferer
Nov 9, 2008

Impractical practicality
Where do I escalate a ticket when there is no viable near term solution and I see no resolution in the next 3 months? Answer: Myself :sigh:

I used to dream the one getting the ticket escalated to the top had it easy.

New career plan
•get into management.
•Get an office.
•Always keep office door shut.
•Important client complains to me about x problem.
•Email employee in networking to resolve issue.
•Watch Netflix rest of day.

Management works exactly like this right? :v:

AndyElusive posted:

Yay, I passed my 901 exam for CompTIA A+

Good job, you'll be past comptia and working on Cisco or Microsoft Certs in no time.

Judge Schnoopy
Nov 2, 2005

dont even TRY it, pal

DropsySufferer posted:


New career plan
•get into management.
•Get an office.
•Always keep office door shut.
•Important client complains to me about x problem.
•Email employee in networking to resolve issue.
•Watch Netflix rest of day.

Management works exactly like this right? :v:


You forgot the part where you go home unsatisfied because you know you're siphoning the company's money without adding any value whatsoever, and decide to start drinking regularly to get some sort of enjoyment out of your lovely boring life. Then you come in to work increasingly angry as you wait for the clock to roll over to 5 PM, at which time you can crack open the flask in your glove compartment and space out as you drive to your dull overpriced house that gives you nothing but problems that you don't want to fix. One day a deer jumps in front of your car and you give a sigh of relief as your car crumples into your brain and you experience the sweet release from this mortal coil.

That's how management works.

MF_James
May 8, 2008
I CANNOT HANDLE BEING CALLED OUT ON MY DUMBASS OPINIONS ABOUT ANTI-VIRUS AND SECURITY. I REALLY LIKE TO THINK THAT I KNOW THINGS HERE

INSTEAD I AM GOING TO WHINE ABOUT IT IN OTHER THREADS SO MY OPINION CAN FEEL VALIDATED IN AN ECHO CHAMBER I LIKE

Judge Schnoopy posted:

You forgot the part where you go home unsatisfied because you know you're siphoning the company's money without adding any value whatsoever, and decide to start drinking regularly to get some sort of enjoyment out of your lovely boring life. Then you come in to work increasingly angry as you wait for the clock to roll over to 5 PM, at which time you can crack open the flask in your glove compartment and space out as you drive to your dull overpriced house that gives you nothing but problems that you don't want to fix. One day a deer jumps in front of your car and you give a sigh of relief as your car crumples into your brain and you experience the sweet release from this mortal coil.

That's how management works.

I am pretty sure this is my current manager, his sole job seems to be eating poo poo from his boss because they don't understand how to properly evaluate people that are working in a role higher than helpdesk and think number of notes added to tickets and dumb metrics like that are good indicators that we are doing our job.

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else

MF_James posted:

I am pretty sure this is my current manager, his sole job seems to be eating poo poo from his boss because they don't understand how to properly evaluate people that are working in a role higher than helpdesk and think number of notes added to tickets and dumb metrics like that are good indicators that we are doing our job.

In all seriousness, KPIs can be a really hard thing to decide what is "good". Sometimes that's just because the decision makers don't understand the decisions they are making (whole other problem). That being said, gently caress everything about that.

Vargatron
Apr 19, 2008

MRAZZLE DAZZLE


Dumbest KPI I've had to deal with is "tickets closed at first touch". Like what exactly does that mean? To my previous manager, it meant that you closed the ticked immediately upon receiving it. Asking the user for more information or clarification automatically meant that you couldn't close the ticket at first touch.

I tried explaining this paradox repeatedly and he would not budge.

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else
I can only assumed he figured you would always immediately call the user and they would absolutely pick up the phone and not be busy. That way you can gain all the info you need and fix it before... un-touching the ticket?

Vargatron
Apr 19, 2008

MRAZZLE DAZZLE


No, you see all communication had to be done within the ticket, so everybody could "remain informed".

I think I was literally gaslighted at one point in my career.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Vargatron posted:

Dumbest KPI I've had to deal with is "tickets closed at first touch". Like what exactly does that mean? To my previous manager, it meant that you closed the ticked immediately upon receiving it. Asking the user for more information or clarification automatically meant that you couldn't close the ticket at first touch.

I tried explaining this paradox repeatedly and he would not budge.
This is actually a pretty good metric for benchmarking a whole service desk organization, and one I wouldn't have thought of. Easy to game, and awful for measuring the effectiveness of an individual on the team, though!

Defenestrategy
Oct 24, 2010

How do you measure the effectiveness of an IT department though? If poo poo keeps breaking obviously youre loving up fixing it, and if poo poo doesnt break obviously we dont need you around here, please justify your paycheck.

CrazyLittle
Sep 11, 2001





Clapping Larry

Internet Explorer posted:

I don't believe you. And I'm not looking it up because my day is depressing enough already.

Internet Explorer posted:

I think I need to just go home and curl up into a ball. Today has been one of those days.


here i fixed it for you plz don't die


George H.W. Cunt
Oct 6, 2010





Defenestrategy posted:

How do you measure the effectiveness of an IT department though? If poo poo keeps breaking obviously youre loving up fixing it, and if poo poo doesnt break obviously we dont need you around here, please justify your paycheck.

The president at my old small business MSP would say he wants to get to the point where the client is wondering why they even pay us since nothing breaks. It’s a good problem to have really. Once things are settled you’re more of an insurance policy.

nielsm
Jun 1, 2009



How do you measure the efficiency of the HR department?

Grassy Knowles
Apr 4, 2003

"The original Terminator was a gritty fucking AMAZING piece of sci-fi. Gritty fucking rock-hard MURDER!"

Vargatron posted:

Dumbest KPI I've had to deal with is "tickets closed at first touch". Like what exactly does that mean? To my previous manager, it meant that you closed the ticked immediately upon receiving it. Asking the user for more information or clarification automatically meant that you couldn't close the ticket at first touch.

I tried explaining this paradox repeatedly and he would not budge.

I had a boss specifically track this metric by having us file a third form during case resolution, because they wanted to provide it to the next-up in the chain-of-command as justification for our jobs.

We called it "on first contact" and it only ever happened when the ticket was originally opened with all the relevant data and the customer was filing an enhancement request, which was always a child ticket. They ended up adding all invalid/duplicate tickets when providing the metric to others, so it meant literally nothing.

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

nielsm posted:

How do you measure the efficiency of the HR department?

By the # of lawsuits avoided

lampey
Mar 27, 2012

Measuring performance in a way that minimizes the impact on the orginization is a hard problem to solve. Customer satisfaction is harder to game than most, but it is not very useful on an individual level. What customers want and expect can be unintuitive, they often prefer timely communication with a bad solution or delayed fix than no communication with a timely fix. Setting expectations can completely change how happy the customer is without showing in other metrics. There are a lot of different ways to measure satisfaction too.

Metrics can be misleading and easily gamed, and focusing too much on certain metrics is detrimental. One of our competitors has an SLA for one hour to first response, but count the automated response. The close on first touch metric encourages multiple tickets to be entered for simple problems instead of requesting info and getting to the root of the problem.

Vargatron
Apr 19, 2008

MRAZZLE DAZZLE


I mean there's a ton of metrics that can be used to gauge relative performance for any IT gig. The problem is that a lot of them are easily gamed.

Budget/Project performance metrics are pretty solid I feel, but only if they address things like scope creep and non-IT principles not doing their fair share of the work. The other thing is that metrics such as "X tickets closed a month" can sometimes backfire and prevent actual root issues from being fixed, because that would lower the amount of tickets created and closed.

I'm pretty thankful that my direct superior has the mindset "if nobody is complaining, then you are doing your job". But I've totally been at jobs where we had a monthly group performance review and had to defend why our average ticket resolution was 10 days (I was the one getting hosed with doing system changes and process implementations).

Also, I absolutely hate working at businesses who expected admin level personnel to also do helpdesk duties. Like the work itself isn't actually beneath me or anything, but it's extremely hard to effectively manage an environment while also having to stop multiple times a day to address end user issues. Then you get grilled later because all of your projects are behind, or your helpdesk queue isn't clean enough.

Basically, full communism 2020!!!

Coredump
Dec 1, 2002

LOLOLOL OUR SERVER ROOM AC WENT OUT. AND WE FOUND OUT THERE'S A ROOF LEAK OVER OUR DATAROOM UPS. ITS LITERALLY DRIPPING WATER ONTO THE CABINET SIZED UPS. LOLOL

Vargatron
Apr 19, 2008

MRAZZLE DAZZLE


Coredump posted:

LOLOLOL OUR SERVER ROOM AC WENT OUT. AND WE FOUND OUT THERE'S A ROOF LEAK OVER OUR DATAROOM UPS. ITS LITERALLY DRIPPING WATER ONTO THE CABINET SIZED UPS. LOLOL

Happy Friday!

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

Coredump posted:

LOLOLOL OUR SERVER ROOM AC WENT OUT. AND WE FOUND OUT THERE'S A ROOF LEAK OVER OUR DATAROOM UPS. ITS LITERALLY DRIPPING WATER ONTO THE CABINET SIZED UPS. LOLOL

Nice.

At a previous job we had a drain on an ac unit get plugged in a server room.

We noticed when we saw an inch of water underfloor.

Next to the 100KVA feed.

CLAM DOWN
Feb 13, 2007




One time years ago at another job we got humidity alerts from the server room. Everything seemed ok so we were ready to chalk it up to a bad sensor, then we lifted one of the floor tiles up. Rushing water like a river lmao

in a well actually
Jan 26, 2011

dude, you gotta end it on the rhyme

Docjowles posted:

I wouldn't say "we" were, since that implies I had any involvement in the choice, but yeah. Let me also tell you about the virtualization environment built on loving Ganeti (what, you've never heard of it?) using DRBD volumes for "HA storage". Or the old-even-then HP MSA SANs I was not allowed to ever, ever patch or reboot despite known bugs that would crash them after X days of uptime. Because "one time an update went bad".

...actually, do not let me tell you that, because I will get very mad and chug a fifth of bourbon.

Also necro, but lol. I remember ganeti. Google uses it! Limoncelli works on it! LISA presentation! It’s gotta be good!

(drbd is a great red flag in any project)

Vargatron
Apr 19, 2008

MRAZZLE DAZZLE


God drat it, my boot files are hosed on my home PC and now I'm trying to fix it. Just because I do this poo poo for a living doesn't mean I want to do it at home!!!

Adbot
ADBOT LOVES YOU

Judge Schnoopy
Nov 2, 2005

dont even TRY it, pal
Best metric is change success. But it requires that you track change requests separately from tickets.

If the changes being proposed have a high success rate, your people know what they're doing and are effectively managing the system. If no changes are needed, the metric doesn't go down.

Also tracking emergency changes against standard changes is great to understand the 'hair on fire' status of the department. But again, that requires actual change review processes that small businesses (especially msps lol) won't have.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply