Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Cimber
Feb 3, 2014
I'm so loving bored with my job. Been doing unix SA work for 20 years,and I've loving had it.


I hate computers, I hate the users of the computers, I hate the people who design the systems at work.

Sorry, had to rant

Adbot
ADBOT LOVES YOU

Cimber
Feb 3, 2014
Can i mention that Service Now is the most gawdawful piece of poo poo software out there for incident and change management? Terrible UI. All their focus has gone into great reporting for middle management drones and awful tools for us slobs who have to use it.

Cimber
Feb 3, 2014

Cup Runneth Over posted:

Otherwise it's sparkling TCP

I could tell you a joke about UDP but i wouldn't care if you got it.

Cimber
Feb 3, 2014
I'm a unix systems administrator for a large financial, and last year was a major reshuffle of senior management. So big announcement just came down after a series of offsite meetings.

Going forward, I am no longer going to be a unix SA, I am going to be a SRE! Yay?

Sounds good, right? Exceeeept, we all have to interview with a 'SRE advisory panel' to see if we have the proper mindset for this position.


I get to interview for my job again, after 10 years with the company! I'm thrilled!

Cimber
Feb 3, 2014
https://www.youtube.com/watch?v=m4OvQIGDg4I


[edit] Beaten to the punch at this too! I'm doomed!

Cimber
Feb 3, 2014

tokin opposition posted:

Standard and not loving stupid are, sadly for this industry, not the same thing

End of year management KPI: Reduced staffing costs by 30 percent

(two years later when new management team comes in)
End of year management KPI: Reduced critical incident amount by 30 percent.

Cimber
Feb 3, 2014

Submarine Sandpaper posted:

A new interview means new comp negotiations

HAHAHAHA, ok, who let the comedian in?

Cimber
Feb 3, 2014

Wizard of the Deep posted:

Yea, finance is nice in that they generally understand the value of spending money, and are okay with long project timelines. If you can stand the red tape, documentation, and auditing it's not a bad place to have a gig.

Let me tell you about the horrors of MAS.(Money Authority of Singapore)

Cimber
Feb 3, 2014

Umbreon posted:

I've worked out of NOCs my entire career and the longest amount of training I ever got was 3 weeks. Getting dumped directly into a raging tire fire fresh out of a week of training or so has been my experience at every job for over 10 years now. It's literally giving me anxiety to be sitting here doing nothing while the team I'm supposed to be helping eventually is so busy that they can't train me.

As for what training I did get, well... I'm honestly terrified. The processes are so extremely convoluted and broken that while showing me how to work a simple cable replacement ticket, it took almost 2 hours and at the end I was told we were "almost halfway done with the ticket". At one of my previous jobs, a ticket like that would have taken less than 3 minutes tops. Working a ticket here is literally 95% extremely convoluted processes that are all done manually and require waiting on approval for anything ever, and none of it is written down anywhere because "The processes change so much that any guide we make will become outdated in a month or so anyway".

Oh, and that ticket I mentioned it before that took 2 hours? We wound up having to start over the next day because it got rejected by the field techs. Apparently, for one specific location in one state, if you try to dispatch without first identifying the color of the cable you need to be replaced, it will be rejected. And of course that color coding isn't written down anywhere (nor is the fact that we need to identify the cable colors in the first place for that location) so we had to do a separate dispatch to identify the cable first, wait for that dispatch to complete, and then make another one to go actually replace it on a different day, all of which requires multiple approval steps even though there is redundancy.

Nothing about this job so far feels attached to reality and I don't know how to feel anymore.

I mean, yeah it sounds like hell from your perspective that something as simple as a cable swap can be so annoying, but consider it from the business standpoint.

How well do you think this conversation would go "Well, the reason for the outage Mr. Executive Director is that the data center operations guys had a ticket to swap out a cable, they did it during business hours but accidentally touched the production cable for our host rather than the backup cable for the dev host. We were at a single point of failure at that moment as we had a down NIC and were waiting on HP to ship us the replacement. All touch activities should happen during off prod hours going forward."

Cimber
Feb 3, 2014

BGP or DNS. Gotta be

Cimber
Feb 3, 2014

Umbreon posted:

It's been two months since I've started this job at <BANK>. I still haven't worked a single ticket, my training still isn't even *defined* anywhere, much less complete, and I only actually receive any training from someone for an hour or two around once a week, twice if I'm lucky. The rest of the time I'm just being paid to sit there.

I get that they're too understaffed to spare anyone to train me, but this still feels so incredibly wrong and I've been desperately looking for any training material I have access to. I found a 40 page document labeled "Basic ticket handling steps" which I've been using to keep myself busy, but it's outdated as hell and missing a ton.

Dumb question:

Is it possible for a company to hire someone for an understaffed team but have to let that person go because they literally can't spare anyone to train the new guy?

I would also say that being proactive is better than being reactive. If you want to learn something put a meeting request on team members calenders and ask for stuff. Don't go to the same person rather tap everyone on the shoulder for even a 15 minute chat about some aspect of the job. Also talk to your manager and let him know that you are looking for guidance on the next step.

Cimber
Feb 3, 2014
So I'm reading up on Chaos Monkey as part of my 'Journey to SRE'.

For those that don't know, Chaos Monkey is a application developed by Netflix to test the resiliency of their AWS infrastructure. They do this by causing random outages in production during production hours and seeing how their systems route around that error. From small things like shutting down access to S3 buckets and killing EC2 nodes to large things like nuking entire availability zones and regions.

I would have looooooved to have been on a fly on the wall in the meeting when they tried to sell that to their management.

Cimber
Feb 3, 2014
Service now is absolutely horrible and a miserable user experience. While you are there please slap a dev for me.

Cimber
Feb 3, 2014

Nuclearmonkee posted:

This is the part that kills these deployments.

If you are putting in a platform like ServiceNow, you need to prep the groundwork before you ever pull that trigger. If you aren't going to resource it appropriately and integrate/automate everything, don't loving buy it. Go buy some lovely small/mid sized business ticketing system that does the basics.

I also work at a place where we wasted huge amounts of $ and didn't resource the team. I just don't hook my automation into the drat thing and it's basically a manual virtual paperwork engine for the unfortunates who live in that world. If it ever gets resourced, we can automate just about anything but jesus christ that's the entire value prop for buying something like that. Self service automated everything with an audit trail.

At my shop, there are a ton of features that are just loving locked down for permissions. Have ten tickets all with the same thing? Nope, can't bulk close tickets, ya gotta go into each one and spend 45 seconds filling out the forms to properly resolve it.I can SEE the 'bulk resolve' button there, its taunting me all greyed out and inaccessable.

Same thing for changes. Got ten changes? Gonna be going into each one and hitting that 'approve' button. Oh, and I have to approve EVERY change, because all the CIs hit my group since I'm a unix SA and the hosts are defined CIs. Doesn't matter if the business is rolling out improved java code, I gotta hit the 'approve' button.

Cimber
Feb 3, 2014

Internet Explorer posted:

I was dealing with that yesterday. Took me over an hour to get Logitech support to send me a cheap and lovely key cap for a $160 keyboard. It broke with normal use, literally never seen that before. Chat, email, phone, their helpful AI was going to walk me through diagnosing the problem.

Well first I need you to unplug your keyboard, wait five seconds and then plug it back in. Did that solve your problem?

No? Ok, lets try something else. Please try turning your keyboard over and tapping it three times. Did that solve your problem?

No? Ok, lets try something else. Please unplug your keyboard, wait ten seconds and then plug it back in. Did that solve your problem?

No, Ok. I'm going to transfer you to our customer support team. The average wait time is 188 minutes. Thank you for your continued support!

Cimber
Feb 3, 2014
No one appreciates NTP until you need NTP

Cimber
Feb 3, 2014

Antigravitas posted:

It's honestly awe inspiring how suddenly things go sideways if time is allowed to drift too far. Time offset relative to the monitoring system is something we check for because the consequences of going too far are completely unpredictable and chaotic.

I work in finance. If time drift can be measured more than 3ms alarms go off.

Cimber
Feb 3, 2014
Can I bitch about using Microsoft teams?

Actually, scratch that, I'm going to bitch about being forced to use Microsoft Teams if you like it or not.

MS teams sucks, I hate the fact that it has to 'ding' every time someone messages me, and that I can't simply turn off dings for existing chats and have it on only for newly created chats. I hate

when
people
message me
like this
and every
message
causes a ding.

loving millennials.

Cimber
Feb 3, 2014

Thanks Ants posted:

Teams is as bad as every other corporate chat app

Skype was great. It did two features: Chat and screen share. Thats all.
Teams tries to be useful and does everything and ends up getting in its way. Basic poo poo like sharing the screen while doing a chat is really hard. Either it doesn't work, it works for some people but not others, or the red bar doesn't disappear when I am done and I have to restart the whole app.

It's also really hard to see who's in the drat chat with me, while Skype would have a useful bar on the left side showing everyone's name and status.

Cimber
Feb 3, 2014
I really just
wish that
people would
not send
me messages
that look like this.

Especially if they pause for 10 seconds between each message, causing my teams to ping/flash dragging me away from what I am trying to do. Use complete sentences for Christ sakes, this is a professional setting not some discord jerkoff session.

Cimber
Feb 3, 2014

The Fool posted:

if you're not careful I'll send a message that is just "Hello"

Hello.


<ten minutes pass>

How are you?


LOL, someone made a website about this. https://nohello.net/en/

Cimber
Feb 3, 2014

RFC2324 posted:

This feels like a specific situation tho. You should be aware of when the people you are pinging work, and tailor your interactions appropriately.

If its someone in or near your TZ its more socially acceptable to be more conversational, while yeah, if someone is on the other side of the planet the message should effectively be an email.

The point is, don't waste my time and make me to stop thinking of what I was working on while I wait for you to say something useful.

Hi
<time passes>
How are you?
<time passes>
Do you know how to list contents of a file system?

or

Hey, how are you doing? Hey listen, I am trying to list contents of a file system, can you help?

(to which I'm going to say to both with some irritation "Did you bother to google first, fuckstick?")

Cimber
Feb 3, 2014
New Thread title: Working in IT 3.0: Don't say hi.

Cimber
Feb 3, 2014

LochNessMonster posted:

You’ve described the auditing department.

Oh, the group that makes a bunch of unrealistic requests, never checks up on them afterwords except once a year and goes to highest management if we push back on their stupidity? That auditing group?

Cimber
Feb 3, 2014
Team manager, who has been promoted from within from the offshore group two months ago has just dictated that no ticket in our queue can be older than 48 hours. If a ticket is going to be older than 48 hours we are to close it and open a new one with the same information.

Cimber
Feb 3, 2014

tokin opposition posted:

So if you're touching it at 4:59 and the ticket is 32 hours old you have to make a new one since you won't be back for ~16 hours? Lol wut

If when we review the queue we see that a ticket is about to 'go amber' we are to immediately copy the details out, close it and then open a new one that is 'green'.

Cimber
Feb 3, 2014
He's....very much the type of guy who would stab you in the back at the same time he's telling you how much of a good team member you are.

Cimber
Feb 3, 2014

Lobsterboy posted:

sr mgmt is instead going to reward them for their meticulous care and maintenance of the queue and ensuring that everything is handled within 48 hrs :patriot:

I see the ticket volume is up about 25 percent, which isn't great, but the closure rate is also up as well as the average incident time is down significantly compared to the TTM average.! Increased ticket volume probably correlates to increased efficiency you've been able to get out of your team. Impressive for your first month! Congratulations, I'm giving you a 15K raise since you are making me look good to my boss.



[edit] Even typing that out made my blood pressure skyrocket.

Cimber
Feb 3, 2014

Darchangel posted:

This is a valid approach. Depends on how your personality works, I guess. Personally, I’m on the same track.

I want to have a life and not work 60+ hours a week. I've seen too many folks think they are getting up the management ladder only to hit a management ceiling of team lead, but be expected to always be available.

gently caress. That.

Cimber
Feb 3, 2014
CI/CD is a great buzzword for managers, but to implement it properly where it gives you actual benefits requires a lot of buy in from a lot of teams, and if one team isn't enthusiastic about it the whole house of cards falls down.

Now, why does k8 spooling up for CI/CD cost so much money? The entire point of K8 is to be able to create and destroy infrastructure on demand to SAVE money.

Cimber
Feb 3, 2014

CommieGIR posted:

Pretty much.

K8s: "Check out this cool tech stack, its for specific use cases!"

Devs: "Let's use it for EVERYTHING."

I need webservers quickly because its Amazon Prime day! K8 to the rescue!
I need more auth servers quickly because its Amazon Prime day? K8 to the rescue!
I need more database space quickly because its Amazon Prime day? DON'T EVEN loving THINK ABOUT USING K8.

Cimber
Feb 3, 2014
Configuration as code and infrastructure as code are good things.

Cimber
Feb 3, 2014
So i had my SRE internal interview yesterday. I think it went pretty well, except for not being able to identify by name what a blue/green deployment was. Being a unix admin for my professional career I never had to do software deployments like application teams did, but I did mention about deploying to pod B first, testing with live traffic mirrored from pod A, validating the result on B vs A and if it matched then slowly move new connections to B and ween A off. Once traffic is bled out take A down and deploy new software there while B handles the load.

I'm off this week to burn carry over vacation days from last year, and I really am fighting the urge to log in every twenty minutes to check my mail and see if my results came back yet.

My teammate had the same interview two weeks ago, she failed it, but said she did hers in an hour. My interview was an hour and a half with a lot more questions asked of me than were asked of her.

I don't think I massively screwed anything up and I feel confident that I answered the questions properly. No huge self doubts in my head on my answers so thats a good sign?

Cimber
Feb 3, 2014

Vampire Panties posted:

MS announced a five-nines SLA for teams phone this week, going into effect April 1st

get ready for all teams, all the time, everywhere

99.999 SLA for teams phone means they can 25.9 seconds outage per month, or a total of 5 minutes per year. Not a great error budget.

Cimber
Feb 3, 2014

Vampire Panties posted:


The previous SLA was 8 hours a year, and people still migrated to it in droves. There was a holdout of cranky telecom dudes who screamed about downtime, but here's a secret - nobody uses desk phones anymore unless its fundamentally part of their role (i.e. call center agents, assistants, receptionists, etc). Everyone uses chat or video.


I can't remember I actually used my phone for anything at work. We use zoom for everything, even something that would take 30 seconds.

My company had just laid out a few hundred million to get Cisco video phones on all our desks in 2019 with the project ending March 31st 2020. Whoops.

That being said, on the rare occasions that zoom has had an outage at work we just cope. Doesn't need to be a call when it can be an email.

Cimber
Feb 3, 2014

Thanks Ants posted:

C-levels who don't see why they should have to learn anything new and anyway headsets are what secretaries and call centre workers wear.

I bet you they still have the POWAH BLUETOOT EARPIECE that flashes a little blue light when they are on a call walking down the street to show how important they are.

Cimber
Feb 3, 2014
Well, I failed my SRE interview. 20+ years as a Unix SA apparently doesn't qualify me to understand systems, monitoring and the rest.

My weakness? Understanding the software deployment life cycles and deployment models. Because as a unix SA I deal with that all the time.

This is such bullshit, I'm so upset right now. Hopefully I don't lose my job.

HR said that the interview interview has a 30 percent pass rate.

Cimber
Feb 3, 2014

Thanks Ants posted:

I kind of miss that VC stuff, you could dial an IP and the two boxes would negotiate codecs and that was it. Now you build a room for Teams and it's better than a standards-based room could ever be at that task, but going outside of the ecosystem depends on each service being supported. Feels like a step backwards.

I might have missed it, but why are they interviewing for internal roles? Surely people know more about your abilities by now that an interview is pointless.

They are reclassifying all us unix SAs as Site Reliability Engineers, and requiring us to interview for the job. The people interviewing me have never worked for me, they are the head of the company's new SRE management chain. I suspect they have sold SRE to upper management as an elite strikeforce of technology like we are navy seals or something.

Cimber
Feb 3, 2014

Darchangel posted:


So, hold on - you've done your job at least adequately for however long, but these jackholes come in and somehow prove you're incompetent for the job you've been doing?
That is so incredibly Office Space - you know the scene. gently caress that noise.


Very much so. Outside guy, never dealt with him before, he's convinced upper managers that SRE! is the way to go, only he is capable of leading this concept within the firm.

"So, what is it that you do here?"

I got word that others in my team have also failed.

Adbot
ADBOT LOVES YOU

Cimber
Feb 3, 2014

Gucci Loafers posted:

What kind of questions did they even ask you?

Stuff about CI/CD, Blue/Green deployment models. (I didn't know what that was specifically, but I did basically describe it in other terms). I also think that I didn't do myself any favors when I said that CD to production might not be a great idea for us (I work for a large bank) while it might be fine for an internet startup.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply