Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Internet Explorer
Jun 1, 2005





The Iron Rose posted:

The level of internal pushback I get on any security item is absolutely appalling. If I wasn't on this team, I'm pretty sure nobody would give a poo poo. We're failing on the basics - encryption, patching, offboarding - and every time I try and campaign for it I get told to sit down and shut up in nicer words.

It's incredibly frustrating.

yeah, sounds about right

Adbot
ADBOT LOVES YOU

xzzy
Mar 5, 2009

No one cared about patches/updates until a system got owned and some high level people got embarrassed by it. From that day forward there were no reservations at all about updating anything and we even got a monthly scheduled downtime window to do updates.

I'm not saying you should let a system get hacked, but it would probably be effective. :angel:

Internet Explorer
Jun 1, 2005





If that worked for you, that's awesome, but I would say it is not a given that a breach would lead to things changing. I certainly have seen it not. In fact, I'm not sure I've ever seen a breach or really any other catastrophic event lead to improvement. And I have seen my fair share of "company-ending" level of events.

Scaramouche
Mar 26, 2001

SPACE FACE! SPACE FACE!

The only time Ive seen action on security is when investors are sniffing around and they've asked for some kind of bare minimum action plan as part of the deal.

Jaded Burnout
Jul 10, 2004


xzzy posted:

No one cared about patches/updates until a system got owned and some high level people got embarrassed by it. From that day forward there were no reservations at all about updating anything and we even got a monthly scheduled downtime window to do updates.

I'm not saying you should let a system get hacked, but it would probably be effective. :angel:

A company I worked for finally listened to my requests for rotating passwords (and not sharing them, please) when a pissed off ex-employee logged into the webmail of one of the owners and forwarded an email chain to the entire company, wherein she and her son (another director, of course) were arguing about how she needs to give him more cash from the business (which he was literally spending on hookers and blow) or else he'd destroy the company.

CollegeCop
Jul 11, 2005

You're right. I'm not a real cop. Those are imaginary handcuffs. And in a minute, we'll be going to the make-believe jail.
What's pissing me off today?

"Known Issue" that 3rd level teams "know" about, but apparently do not plan to do anything about

When we connect to our VPN, there is apparently some back-end encryption going on. While this encryption is being negotiated, Win10 gives the yellow Alert! triangle on the network connection, which shows No Internet Connection when you hover over it. Sometimes it takes a little time to clear. And sometimes, the encryption never completes and the icon never clears.

Which isn't that big of deal because, despite what Windows tells you, you actually do have a network/internet connection. Web pages load, you can access network drives, Skype/Teams work. Almost everything function just fine without that encryption.

Notice I said "almost". The exception? Office - most notable to the end user - Outlook. It simply refuses to connect to the Exchange server if the encryption is not finalized. Office products will also not connect to your Office account, but that is mostly transparent and you can still use them, so users tend to not notice.

The only fix is to wait to open Outlook until the yellow triangle goes away. And if it doesn't clear in 10-15 minutes, disconnecting from vpn and reconnecting in the hopes of getting another server.

We (Desktop) noticed this problem back in July/August, and sent a ticket up to 3rd level - Collaborations and Network Security. In the 8/9 months since, the ticket has not been updated by either team. We have linked literally hundreds of tickets to this one ticket, and get multiple calls on it daily.

The teams obviously are aware of the problem, as evidenced by the occasional ticket that gets escalated to them when, despite our best efforts, we cannot get a user connected to Exchange. Those tickets are rapidly kicked back with the note "Known issue - have user switch vpn connections per ticket #xxxxx".

On a related note, I am tired of listening to users insist that connecting to the vpn "disconnects them from the internet", even after I have demonstrated that they are, indeed, still connected to the internet. "But Windows says No Internet Connection!"

mllaneza
Apr 28, 2007

Veteran, Bermuda Triangle Expeditionary Force, 1993-1952




CollegeCop posted:

"But Windows says No Internet Connection!"

Windows says a lot of things.

Sickening
Jul 16, 2007

Black summer was the best summer.

The Iron Rose posted:

The level of internal pushback I get on any security item is absolutely appalling. If I wasn't on this team, I'm pretty sure nobody would give a poo poo. We're failing on the basics - encryption, patching, offboarding - and every time I try and campaign for it I get told to sit down and shut up in nicer words.

It's incredibly frustrating.

I have encountered this in the past and the one way I saw this as effective is just to keep a running tally of all your "risk". Something as simple as a protected spreadsheet where you list all the outstanding work left to be done in this area. Sooner or later two things will happen. You will either get a new leader who gives a poo poo and you can communicate this list to them, or you will get owned and you can point back to the list. Its good to at least get organized in this area.

Raygereio
Nov 12, 2012

Internet Explorer posted:

If that worked for you, that's awesome, but I would say it is not a given that a breach would lead to things changing. I certainly have seen it not. In fact, I'm not sure I've ever seen a breach or really any other catastrophic event lead to improvement. And I have seen my fair share of "company-ending" level of events.
Same. :negative:
The company I work for had their whole network ransomwared 4 years ago. You would think that would mean we would do our best to secure our netwerk better afterwards. But last week I was attempting to explain to our "Head of IT" why we needed to get rid our existing Win7 machines. And I lost my temper with the fucker after I explained in the most basic terms possible, with the aid of a whiteboard, what ransomware is and he still obviously didn't understand.
It doesn't help that he an accountant, so he counters requests to invest in upgrades with "It's working now, we clearly don't need to change anything".

FlapYoJacks
Feb 12, 2009

Sickening posted:

I have encountered this in the past and the one way I saw this as effective is just to keep a running tally of all your "risk". Something as simple as a protected spreadsheet where you list all the outstanding work left to be done in this area. Sooner or later two things will happen. You will either get a new leader who gives a poo poo and you can communicate this list to them, or you will get owned and you can point back to the list. Its good to at least get organized in this area.

To add to this: make sure you email a higher up on every single item and put their response on the list with a link to the saved reply.

Sickening
Jul 16, 2007

Black summer was the best summer.

ratbert90 posted:

To add to this: make sure you email a higher up on every single item and put their response on the list with a link to the saved reply.

There is personal risk to that. In a just environment, this would help you. In a toxic environment, asking for CYA email responses is going to put a target on your back as people will notice.

Johnny Aztec
Jan 30, 2005

by Hand Knit

Jaded Burnout posted:

A company I worked for finally listened to my requests for rotating passwords (and not sharing them, please) when a pissed off ex-employee logged into the webmail of one of the owners and forwarded an email chain to the entire company, wherein she and her son (another director, of course) were arguing about how she needs to give him more cash from the business (which he was literally spending on hookers and blow) or else he'd destroy the company.

That man is a real hero.

What was the fallout from that?



(laffo, fallout. That son probably is the CEO now)

FlapYoJacks
Feb 12, 2009

Sickening posted:

There is personal risk to that. In a just environment, this would help you. In a toxic environment, asking for CYA email responses is going to put a target on your back as people will notice.

The easiest way is to just email them to begin with: “Hey $manager, I noticed this $security_issue, what would you like me to do?”

If they talk to you in person then email them after the conversation: “Hey $manager, just to make sure I understood you correctly, you want me to do $x. Is that correct?”

Totally non confrontational and as an added bonus makes you look like you are a team player and value your managers input.

Sickening
Jul 16, 2007

Black summer was the best summer.

ratbert90 posted:

The easiest way is to just email them to begin with: “Hey $manager, I noticed this $security_issue, what would you like me to do?”

If they talk to you in person then email them after the conversation: “Hey $manager, just to make sure I understood you correctly, you want me to do $x. Is that correct?”

Totally non confrontational and as an added bonus makes you look like you are a team player and value your managers input.

I think even the average moron is going to see email number 2 for what it is and paint you up. If the most basic security practices aren't being done, you don't need extensive CYA email chain as proof. The state of the systems is proof enough that leaders weren't doing the right things.

Methylethylaldehyde
Oct 23, 2004

BAKA BAKA

Sickening posted:

I think even the average moron is going to see email number 2 for what it is and paint you up. If the most basic security practices aren't being done, you don't need extensive CYA email chain as proof. The state of the systems is proof enough that leaders weren't doing the right things.

Still won't help you when they lie about why you were fired so you don't get to collect unemployment.

Raerlynn
Oct 28, 2007

Sorry I'm late, I'm afraid I got lost on the path of life.

Methylethylaldehyde posted:

Still won't help you when they lie about why you were fired so you don't get to collect unemployment.

Let's level with each other, the alternative is you getting fired for "incompetence" when the security shortfalls are discovered and you are blamed roundly for failing to bring this to light. And then effectively blacklisted since no one wants to hire IT personnel that recently worked at a breached company.

FlapYoJacks
Feb 12, 2009

Sickening posted:

I think even the average moron is going to see email number 2 for what it is and paint you up. If the most basic security practices aren't being done, you don't need extensive CYA email chain as proof. The state of the systems is proof enough that leaders weren't doing the right things.

Sure, they may see through the email, but it’s still polite and friendly.


Raerlynn posted:

Let's level with each other, the alternative is you getting fired for "incompetence" when the security shortfalls are discovered and you are blamed roundly for failing to bring this to light. And then effectively blacklisted since no one wants to hire IT personnel that recently worked at a breached company.

Yep, I would rather get fired for some light CYA and have the CYA as proof the company was incompetent than be blacklisted because the company lied and pinned everything on you.

Ham Equity
Apr 16, 2013

The first thing we do, let's kill all the cars.
Grimey Drawer
Remember the multi-billion dollar Equifax breach, and how the company blamed one loving IT person for it?

Not enough CYA in the universe to save yourself from that.

FlapYoJacks
Feb 12, 2009

Thanatosian posted:

Remember the multi-billion dollar Equifax breach, and how the company blamed one loving IT person for it?

Not enough CYA in the universe to save yourself from that.

There is indeed: IT employee releases emails from his manager saying “don’t fix this obvious security flaw” is a pretty drat good way of instantly shifting the story from “IT guy fucks up everything” to “company management fucks up and tries to blame a single employee.”

Jaded Burnout
Jul 10, 2004


Johnny Aztec posted:

That man is a real hero.

What was the fallout from that?



(laffo, fallout. That son probably is the CEO now)

Absolutely nothing, other than suddenly password rotation was deemed important and the guy had it out from me from then on (he thought I did it; I didn't).

The only way he's CEO is if both his parents died.

Methylethylaldehyde
Oct 23, 2004

BAKA BAKA

ratbert90 posted:

There is indeed: IT employee releases emails from his manager saying “don’t fix this obvious security flaw” is a pretty drat good way of instantly shifting the story from “IT guy fucks up everything” to “company management fucks up and tries to blame a single employee.”

The guy who did that would be soundly blacklisted from loving every company bigger than 100 people, because nobody wants to employ the guy who was willing to embarrass their bosses in the national news.

klosterdev
Oct 10, 2006

Na na na na na na na na Batman!
Guess the lesson is you're hosed long as a skeezebag exists more powerful than you

bull3964
Nov 18, 2000

DO YOU HEAR THAT? THAT'S THE SOUND OF ME PATTING MYSELF ON THE BACK.


klosterdev posted:

Guess the lesson is you're hosed long as a skeezebag exists more powerful than you

Harsh truths of US corporate culture.

a_pineapple
Dec 23, 2005


My favorite thing in the world is wasting my time in weeks of regular meetings describing the same thing over and over again to people who make twice my salary something that's already implemented in production that is recommended by the vendor and well-documented while they straight up tell me they don't believe me and instead speculate how it could possibly even work like that blahrheggegwgugihiuewthu;qwe

Next time I'm gonna take a drink every time someone says "I'm just trying to understand" and be plastered halfway through the meeting party time hell yeah

EoRaptor
Sep 13, 2003

by Fluffdaddy

CollegeCop posted:

What's pissing me off today?

"Known Issue" that 3rd level teams "know" about, but apparently do not plan to do anything about

When we connect to our VPN, there is apparently some back-end encryption going on. While this encryption is being negotiated, Win10 gives the yellow Alert! triangle on the network connection, which shows No Internet Connection when you hover over it. Sometimes it takes a little time to clear. And sometimes, the encryption never completes and the icon never clears.

Which isn't that big of deal because, despite what Windows tells you, you actually do have a network/internet connection. Web pages load, you can access network drives, Skype/Teams work. Almost everything function just fine without that encryption.

Notice I said "almost". The exception? Office - most notable to the end user - Outlook. It simply refuses to connect to the Exchange server if the encryption is not finalized. Office products will also not connect to your Office account, but that is mostly transparent and you can still use them, so users tend to not notice.

The only fix is to wait to open Outlook until the yellow triangle goes away. And if it doesn't clear in 10-15 minutes, disconnecting from vpn and reconnecting in the hopes of getting another server.

We (Desktop) noticed this problem back in July/August, and sent a ticket up to 3rd level - Collaborations and Network Security. In the 8/9 months since, the ticket has not been updated by either team. We have linked literally hundreds of tickets to this one ticket, and get multiple calls on it daily.

The teams obviously are aware of the problem, as evidenced by the occasional ticket that gets escalated to them when, despite our best efforts, we cannot get a user connected to Exchange. Those tickets are rapidly kicked back with the note "Known issue - have user switch vpn connections per ticket #xxxxx".

On a related note, I am tired of listening to users insist that connecting to the vpn "disconnects them from the internet", even after I have demonstrated that they are, indeed, still connected to the internet. "But Windows says No Internet Connection!"

There is a lot going on here:

Microsoft recently acknowledged there is a bug in Windows 10 NCSI, the 2020-03 Update Rollup KB4551762 has a possible fix. Does not work in all circumstances.

If your VPN client doesn't show up as a VPN client in Windows, and instead shows up as an ethernet connection, there can be some issues with Windows running queries against the device before the connection is established. This is the VPN software reporting 'connected' prematurely, the vendor should be able to provide an updated client that defers reporting connected to Windows for a few seconds among some other things.

NCSI depends on being able to do a DNS lookup and series of web queries against https://www.msftconnecttest.com (article). If you gate Internet access behind a proxy or firewall with USER access control, you need to allow a COMPUTER account to complete those tests for NCSI to succeed.

If you do use a Proxy, you need to tell the WinHTTP service about it. Use the 'netsh winhttp set proxy fqdn:port "exceptions"' command to set it, or use GPO for wider coverage.

Now, if Windows is detecting your VPN connections as public, instead of domain/private, that's a whole different bug.

mllaneza
Apr 28, 2007

Veteran, Bermuda Triangle Expeditionary Force, 1993-1952




a_pineapple posted:

Next time I'm gonna take a drink every time someone says "I'm just trying to understand" and be plastered halfway through the meeting party time hell yeah

Thank god for WFH !

CollegeCop
Jul 11, 2005

You're right. I'm not a real cop. Those are imaginary handcuffs. And in a minute, we'll be going to the make-believe jail.

EoRaptor posted:

There is a lot going on here:

Microsoft recently acknowledged there is a bug in Windows 10 NCSI, the 2020-03 Update Rollup KB4551762 has a possible fix. Does not work in all circumstances.

If your VPN client doesn't show up as a VPN client in Windows, and instead shows up as an ethernet connection, there can be some issues with Windows running queries against the device before the connection is established. This is the VPN software reporting 'connected' prematurely, the vendor should be able to provide an updated client that defers reporting connected to Windows for a few seconds among some other things.

NCSI depends on being able to do a DNS lookup and series of web queries against https://www.msftconnecttest.com (article). If you gate Internet access behind a proxy or firewall with USER access control, you need to allow a COMPUTER account to complete those tests for NCSI to succeed.

If you do use a Proxy, you need to tell the WinHTTP service about it. Use the 'netsh winhttp set proxy fqdn:port "exceptions"' command to set it, or use GPO for wider coverage.

Now, if Windows is detecting your VPN connections as public, instead of domain/private, that's a whole different bug.

All of this is above my pay grade (and for the most part, outside my realm of expertise)

What I do know is that we have gotten 0 updates in 9 months on an issue that we deal with multiple times a day.

At this point, I'd be happy with "Windows/Microsoft/VPN bug. Working with the vendor on a fix"

I'd be loving tickled pink with "Try this and let us know if it works"

It's the radio silence that is pissing me off.

Edit:

Holy. poo poo. I plugged some of the keywords from your post into Google, and the first loving result was a Knowledge Base article from our VPN vendor outlining this exact problem and a fix to try. It's dated 2018.

I also found several articles about the Win10 bug and the fix that they just rolled out. Since these problems really started creeping up about the time we did our last massive Win10 migration, I'm tempted to say this might be it as well.

I've forwarded the information to our Subject Matter Expert to send up to 3rd level.

CollegeCop fucked around with this message at 18:25 on May 13, 2020

skooma512
Feb 8, 2012

You couldn't grok my race car, but you dug the roadside blur.

a_pineapple posted:

My favorite thing in the world is wasting my time in weeks of regular meetings describing the same thing over and over again to people who make twice my salary something that's already implemented in production that is recommended by the vendor and well-documented while they straight up tell me they don't believe me and instead speculate how it could possibly even work like that blahrheggegwgugihiuewthu;qwe

Next time I'm gonna take a drink every time someone says "I'm just trying to understand" and be plastered halfway through the meeting party time hell yeah

I was in the other IT thread talking about this phenomenon. People just don't believe instructions are real or that anything could possibly work as described. There always has to be workaround or special ritual you have to do, even if everything is working perfectly fine and nothing is wrong.

22 Eargesplitten
Oct 10, 2010



CollegeCop posted:

All of this is above my pay grade (and for the most part, outside my realm of expertise)

What I do know is that we have gotten 0 updates in 9 months on an issue that we deal with multiple times a day.

At this point, I'd be happy with "Windows/Microsoft/VPN bug. Working with the vendor on a fix"

I'd be loving tickled pink with "Try this and let us know if it works"

It's the radio silence that is pissing me off.

Have you brought it up with your manager? That's my go-to when poo poo's getting ignored by other departments.

Bob Morales
Aug 18, 2006


Just wear the fucking mask, Bob

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

HI THIS IS BOB FROM IT

*CHEWING NOISES*

CollegeCop
Jul 11, 2005

You're right. I'm not a real cop. Those are imaginary handcuffs. And in a minute, we'll be going to the make-believe jail.

22 Eargesplitten posted:

Have you brought it up with your manager? That's my go-to when poo poo's getting ignored by other departments.

Every week for the past 6 months in our weekly team huddle.

I just spoon-fed some info up to 3rd level thanks to EoRaptor.

Maybe someone can take the info, claim it as their own, and finally fix the problem.

Ratmtattat
Mar 10, 2004
the hairdryer

The joke was to call it Office-359, but I'm thinking that it's more like Office-355.

Mr. Clark2
Sep 17, 2003

Rocco sez: Oh man, what a bummer. Woof.

Sickening posted:

I would love to know what security issues they believe that make teams a no go. :allears:

It's probably not that Teams itself is a security risk, but that it installs to what is typically considered to be an 'unsafe' location. This leads to consternation when IT teams have to create rules that allow Teams to work, while still disallowing the running of other things that users themselves have put into this same unsafe location, some of which could potentially cause harm.

silicone thrills
Jan 9, 2008

I paint things
The network guy changed the vpn address in the middle of the work day with no change management ticket.

Immediately the ticket queue starts pinging.

I test and im like "yep vpn is broked, hey network guy, whats up - I test <network address> and it isnt responding"

"Oh I thought the vpn address was X so I started cleaning up DNS entries"

"We've been using this address for over a year and its built into every profile of every computer we build"

"oh. my bad"



Half of our team is currently furloughed and a bunch of people were laid off already so its skeleton crew and he decides to trim dns entries at 2pm on a loving wendesday. EXCUSE ME?

Internet Explorer
Jun 1, 2005





Well, I found the guy you can probably get rid of!

The Fool
Oct 16, 2003


silicone thrills posted:

The network guy changed the vpn address in the middle of the work day with no change management ticket.

Immediately the ticket queue starts pinging.

I test and im like "yep vpn is broked, hey network guy, whats up - I test <network address> and it isnt responding"

"Oh I thought the vpn address was X so I started cleaning up DNS entries"

"We've been using this address for over a year and its built into every profile of every computer we build"

"oh. my bad"



Half of our team is currently furloughed and a bunch of people were laid off already so its skeleton crew and he decides to trim dns entries at 2pm on a loving wendesday. EXCUSE ME?

Well, if he’s that bored he probably doesn’t need the job

silicone thrills
Jan 9, 2008

I paint things

Internet Explorer posted:

Well, I found the guy you can probably get rid of!

He does something like this every week and he's been there 17 years. I just can't even.

lol here's what I sent my manager

"I just want to very calmly express that i'm upset that <networkguy> is making production system changes in the middle of the work day. "

If I had been at the office I would have just walked off property and sat in my car for a while. Instead I just took a couple of shots. Then we had a team meeting on teams where I scowled the entire time he talked and then blamed service desk for building all the vpn profiles with that address even though thats the address he told us to use in every build over a year ago. No. ticket. No change management. Nothing. Not even a loving heads up that he was trimming entries. We're a 4 person loving entire IT org.


edit: his counterpart who was more of a moron and we're pretty sure had a serious drinking problem because he randomly assigned 802.1x and external networks to random jacks through out the org was laid off 6 months ago so this guys the better of the 2.

silicone thrills fucked around with this message at 23:44 on May 13, 2020

Ham Equity
Apr 16, 2013

The first thing we do, let's kill all the cars.
Grimey Drawer

silicone thrills posted:

He does something like this every week and he's been there 17 years. I just can't even.

lol here's what I sent my manager

"I just want to very calmly express that i'm upset that <networkguy> is making production system changes in the middle of the work day. "

If I had been at the office I would have just walked off property and sat in my car for a while. Instead I just took a couple of shots. Then we had a team meeting on teams where I scowled the entire time he talked and then blamed service desk for building all the vpn profiles with that address even though thats the address he told us to use in every build over a year ago. No. ticket. No change management. Nothing. Not even a loving heads up that he was trimming entries. We're a 4 person loving entire IT org.


edit: his counterpart who was more of a moron and we're pretty sure had a serious drinking problem because he randomly assigned 802.1x and external networks to random jacks through out the org was laid off 6 months ago so this guys the better of the 2.

I have definitely done poo poo like this, but the fact that his reaction is anything other than a profuse apology and a writeup of steps he's taken to insure this doesn't happen again should really be grounds for immediate termination.

I'd be losing my poo poo. You have a remarkable level of self-control.

silicone thrills
Jan 9, 2008

I paint things

Thanatosian posted:

I have definitely done poo poo like this, but the fact that his reaction is anything other than a profuse apology and a writeup of steps he's taken to insure this doesn't happen again should really be grounds for immediate termination.

I'd be losing my poo poo. You have a remarkable level of self-control.

oh thank god we are all working from home because I just started laughing uncontrollably for a solid 20 minutes. Like how else do you react to that? I'm mad as gently caress but what the hell am I going to do about it from here? Nada. I get to sit here and be mad.

Adbot
ADBOT LOVES YOU

Ham Equity
Apr 16, 2013

The first thing we do, let's kill all the cars.
Grimey Drawer

silicone thrills posted:

oh thank god we are all working from home because I just started laughing uncontrollably for a solid 20 minutes. Like how else do you react to that? I'm mad as gently caress but what the hell am I going to do about it from here? Nada. I get to sit here and be mad.
I'd've been on the phone with my manager. Who would have been trying to calm me down; we have a good relationship, it's a great good cop/bad cop dynamic.

Though, in a case like that, I probably would have been the one talking her down. I don't understand how a manager can just sit back and be okay with that. Like, it would be one thing if he clearly recognized he hosed up and all, you don't need to harp on or chew out someone for poo poo like that, we're all human; but to just loving continue in denial like that? Jesus loving Christ.

You should've just put all those tickets in his queue.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply