|
pixaal posted:This; it's free for 100 sensors, which with switches if you use SNMP should get you pretty much everything you need, 1-3 sensors per device should be enough for networking monitoring. You can also point it at servers and get all sorts of fun information with the WMI ones in a windows environment! Want a bunch of info on your WSUS status? Want to know if replication between two DCs failed? I wish I could see what PRTG is like at 100 sensors. At over 16,000 across many probes it's dogshit. Their support is just awful, some kind of crowdsourced bullshit. Half their sensors are in beta mode and we have to restart it all the time because it lies about SNMP sensors. They even say in their documentation to run on physical machines. I can't believe the network team convinced the CTO to drop so much money on this. At least Nagios was free and had a ton of documentation.
|
#
?
Sep 1, 2016 23:37
|
|
|
|
| # ? Jun 10, 2024 09:06 |
|
|
Those are problems with the company culture and incentive structure though, which he specifically mentioned as necessary factors to consider. If the company is completely dysfunctional then you've got bigger problems that how much control your infosec department has, because it is probably also dysfunctional. Setting up IT and infosec as opposing forces fighting for control over information management is rarely going to have good business outcomes. Edit: to be clear, this was in response to Arsten YOLOsubmarine fucked around with this message at 23:45 on Sep 1, 2016 |
|
#
?
Sep 1, 2016 23:42
|
|
|
Too many businesses neuter their smart people with too much policy and bureaucracy. A smart person who can straddle the line of IT and erm and is empowered to make decisions that stick is a valuable asset.
|
|
#
?
Sep 2, 2016 00:52
|
|
|
psydude posted:e^: I'm working with a customer right now whose desktop/systems group is fighting tooth and nail to keep us from turning on 802.1X enforcement because they're worried some users will lose access if their machines aren't registered to the domain or aren't whitelisted. I have another customer that doesn't want to take a 5 minute service outage to upgrade the IPS module on a firewall so I can register it to the management center because it's the end of the fiscal year and they don't want ANY service disruptions, even if it means being unable to gain visibility into traffic.  NippleFloss posted:Those are problems with the company culture and incentive structure though, which he specifically mentioned as necessary factors to consider. If the company is completely dysfunctional then you've got bigger problems that how much control your infosec department has, because it is probably also dysfunctional. And that's kinda missing my point. Infosec, just like every other aspect of a company (management style, IT needs, structure, salary, benefits, incentives, performance, perks, engagement within each strata, etc) is inextricably tied to the culture within a company. IT and Infosec being the best buds can still lead to scenarios where your info isn't sec, where you are destroying profitability, or even actively forcing people to leave through bureaucratic oppression. A good culture and incentive program, in contrast, doesn't even mean that you will have good Infosec. You need to be ready not only for your culture to erode (It can take less than a quarter to obliterate the culture within a company. And it can take up to 4x the time you spent floundering to get that culture, or at least an approximation, back) and change, but you need to assume that your people are reasonably smart and willing to circumvent you if you get in their way. This isn't malicious - on the contrary, they consider you malicious - they simply don't care about your goals. They care about productivity and profitability because that's what they are incentivized on. Arsten fucked around with this message at 04:28 on Sep 2, 2016 |
|
#
?
Sep 2, 2016 04:26
|
|
|
Thanks Ants posted:PRTG I happily wrote them a check for 7500 bucks a couple of months ago and haven't looked back yet. PRTG + Graylog together is nirvana.
|
|
#
?
Sep 2, 2016 04:53
|
|
|
Spent a decent amount of time tonight talking the on-call junior sysadmin down from jumping off a bridge ok not literally, but he took down a production site personally for the first time, and was FLIPPING THE gently caress OUT about it. If there's one thing I've learned from 10 years in IT, it's to
|
|
#
?
Sep 2, 2016 05:54
|
|
|
Knocking out services for a major national company or federal agency is pretty much a rite of passage in IT. It's also where you hone your problem solving skills, because few things are better motivators than not being fired. Source: me taking down all private-sector claims processing for a national health insurance provider for an entire weekend during a botched F5 migration.
|
|
#
?
Sep 2, 2016 05:59
|
|
|
Taking down something site or company wide is definitely a rite of passage, and as I approach veteran status there's something joyful about watching it happen to new people 
|
|
#
?
Sep 2, 2016 06:01
|
|
|
I've never shut down a major website. I feel left out. I've got stories of friends that have but never been me.
|
|
#
?
Sep 2, 2016 06:07
|
|
|
jaegerx posted:I've never shut down a major website. I feel left out. I've got stories of friends that have but never been me. If you can get all the way to senior level without taking something down stupidly at least once that's pretty impressive.
|
|
#
?
Sep 2, 2016 07:23
|
|
|
http://www.nytimes.com/2016/09/02/technology/when-im-mistakenly-put-on-an-email-chain-should-i-hit-reply-all-asking-to-be-removed.html
|
|
#
?
Sep 2, 2016 10:40
|
|
|
rafikki posted:http://www.nytimes.com/2016/09/02/technology/when-im-mistakenly-put-on-an-email-chain-should-i-hit-reply-all-asking-to-be-removed.html I still have a folder called 'hilarity' with a 600 reply email chain. It was sent to approximately 4,000 employees and it escalates to incandescent rage in a fairly quick manner. It peaked with an executive or two threatening people to stop using "reply all". Finally the messaging team caught wind and killed it after about an hour.
|
|
#
?
Sep 2, 2016 12:35
|
|
|
And then you have people trying to be helpful by replying to all and telling people to stop replying to all. 
|
|
#
?
Sep 2, 2016 12:39
|
|
|
Yesterday my boss sent an email to the whole company that the offices would close early today. A few minutes later the CEO's brother in law replied to all with "Ok"
|
|
#
?
Sep 2, 2016 12:45
|
|
|
You have a large, technology dependent business. You run a very large CRM/analytics platform. You hire a developer to join your team of developers. 1. Do you test your dev? Why not? 2. Do you insist your dev have OS and Middleware related skills beyond his self proclaimed Java Ninjaness? Why not? 3. Do you give dev root on production CRM/analytics host? Why?
|
|
#
?
Sep 2, 2016 12:50
|
|
|
Docjowles posted:Spent a decent amount of time tonight talking the on-call junior sysadmin down from jumping off a bridge Typical manager + user response to a major program being down at everyplace I've been  Main program down fixit! fixit! fixit! fixit! Working on it you aren't taking this seriously! Oh god! I'm working on it and it has my full attention (phone): I can't work I get error in program about being in timeout! Working on it (phone): My email is thing is down That can wait, I need my email no get in line unplugs phone, closes door and puts in headphones to drown out the knocking Problem resolved 10 minutes later after 20 minutes of people interrupting to tell me it was down and that I wasn't panicking.
|
|
#
?
Sep 2, 2016 13:12
|
|
|
Why aren't mailing lists locked down to only approved senders? That's an easy way to stop reply-all.
|
|
#
?
Sep 2, 2016 13:14
|
|
|
As much trouble as it causes the Reply-All function is a beautiful gift, because some ignorant dingus will always shoot themselves in the foot in front of the whole company; HR Manager "We regret to inform everyone that person X has left the business" Dumbass "Hi manager who was X" HR Manager "We must inform everyone there is to be no smoking on the premises, and this also includes company vehicles" Dumbass "Thanks for the update but I don't smoke anyway, kind regards"
|
|
#
?
Sep 2, 2016 13:48
|
|
|
My previous employer sent out a mail for a new position to everyone a few years back. Requirements were that you spoke both French and (primarily) Dutch, as well as proficiency with MS Office. We used Outlook for mail at the time(later migrated to Office365 webmail for cost reasons according to IT) The next day someone reply all'd with their full resume and cover letter attached proudly claiming over a decade of experience with MS Office, along with a note that they didn't speak Dutch but were willing to learn. An hour or two later, another guy reply all'd with his resume as well, and the same excuse for not speaking Dutch. I somehow doubt they even got an interview or realized the reply all.
|
|
#
?
Sep 2, 2016 14:06
|
|
|
flosofl posted:I still have a folder called 'hilarity' with a 600 reply email chain. It was sent to approximately 4,000 employees and it escalates to incandescent rage in a fairly quick manner. It peaked with an executive or two threatening people to stop using "reply all". Finally the messaging team caught wind and killed it after about an hour. HR Email: Jim Bob has left the company to pursue other opportunities and we wish him well. Jim Bob, Executive Manager: I've done what, now? HR Email: Karen Bob has left the company to pursue other opportunities and we wish her well. Karen Bob, Regional General Manager: Huh? Do you just not like people with the last name Bob? HR Email: Please disregard previous retirement emails. Day 2 HR Email: Jim Bob has left the company to pursue other opportunities and we wish him well. Jim Bob: Still wrong, guys. Day 2 HR Email: Jim Bob will continue his job functions. It is actually Karen Bob that is leaving us. Please wish her well. Karen Bob: Really? Day 3 HR Email: Michael Bob has decided to step down as CEO to pursue other opportunities. We wish him well and than him for his tenure! Day 3 HR Email: Please disregard the previous retirement email. Jane Smith (HR Director) is no longer with the company, so please wish her the best. Day 3 HR Email: Guy McGuy (HR Asst Director) is no longer with the company. Please wish him well for all his future endeavors! Day 3 HR Email: Girl ChickieGirl (HR Clerk) is no longer with the company. Please wish her well for all her future endeavors! Tell the CEO he's retiring when he's not. Go on, I dare ya!  Edit: Oh, and the guy who actually left the company was a bottom-rung supervisor in charge of a logistics piece somewhere. He shared the (common) last name, but his first name was completely different. Like "Jim" vs "Ralph" sort of difference. Arsten fucked around with this message at 14:49 on Sep 2, 2016 |
|
#
?
Sep 2, 2016 14:46
|
|
|
Arsten posted:My hilarity reply all email chain was this: My guess is they were either only given the last name Mr.Bob is retiring! That or they were verbally told only remembered the last name and were too lazy to ask whoever told them "Hey there's like 30 mr.bobs I'm just want to verify the first name with you, is it Jim? no it's Ralph? good thing I didn't send that out!"
|
|
#
?
Sep 2, 2016 14:58
|
|
|
pixaal posted:My guess is they were either only given the last name Mr.Bob is retiring! That or they were verbally told only remembered the last name and were too lazy to ask whoever told them "Hey there's like 30 mr.bobs I'm just want to verify the first name with you, is it Jim? no it's Ralph? good thing I didn't send that out!" I don't know.... Even if you didn't know that Jim Bob was some executive manager, how do you not only not recognize your CEO's name, but actually type out "is stepping down as CEO" in your email without being completely and totally brain dead?
|
|
#
?
Sep 2, 2016 15:12
|
|
|
Arsten posted:I don't know.... Even if you didn't know that Jim Bob was some executive manager, how do you not only not recognize your CEO's name, but actually type out "is stepping down as CEO" in your email without being completely and totally brain dead? They got fired over this right? It's hard to tell with the 3 false HR leaving things. e: We don't even bother with an email. I usually find out someone is quitting I get a knock on my door at lunch asking if I want some cake and pizza because someone is leaving. That or I on board their replacement because they weren't good enough to get cake or didn't give notice. pixaal fucked around with this message at 15:17 on Sep 2, 2016 |
|
#
?
Sep 2, 2016 15:13
|
|
|
Looks like 3 HR people got fired over it.
|
|
#
?
Sep 2, 2016 15:15
|
|
|
rafikki posted:http://www.nytimes.com/2016/09/02/technology/when-im-mistakenly-put-on-an-email-chain-should-i-hit-reply-all-asking-to-be-removed.html Gotta add what it looked like in the actual paper: https://twitter.com/benshpigel/status/771678413274877952
|
|
#
?
Sep 2, 2016 15:16
|
|
|
pixaal posted:They got fired over this right? It's hard to tell with the 3 false HR leaving things. Oh, yeah. The CEO went down there and cleaned house. (You can tell because there were no recall emails )
|
|
#
?
Sep 2, 2016 15:16
|
|
|
air- posted:Gotta add what it looked like in the actual paper: That's perfect.
|
|
#
?
Sep 2, 2016 15:20
|
|
|
Che Delilas posted:It's not a debate or anything but this probably hasn't been seen by some of the people in this thread and it's good for a chuckle. I have a client whose idea of security is to capitalize the P in password as their "secure" password. they asked me to pen test their site and after the first 5 thing I tried all got me in I stopped and gave them my diagnosis as "overhaul required" they seem to think moving to the cloud will solve their inept coding issues....
|
|
#
?
Sep 2, 2016 15:26
|
|
|
pixaal posted:Typical manager + user response to a major program being down at everyplace I've been One of the reasons I love my current boss, when there's an outage he forwards all IT calls to his phone. If anyone comes into the IT area, he ushers them out. I've seen him (politely mind you) tell the CEO go GTFO. Re: taking down large companies, I think we could use a round of stories, so fess up. Here's mine; 1997, I had just sold my ISP and consulting businesses, got my first job at a real company, working for the switch manufacturer Xylan as a Network Engineer. Day one goes by normally, HR stuff, meet the other team members, find out there's no other Network Engineers for a 2,200 person company that makes network gear.. that's a bit weird. Day two, I'm given credentials to our network gear. Its the same username/password for every switch and router. admin/admin... ok. I've got a copy of the manual and I'm going through and just doing some basic network diagramming, since there was 0 documentation about the network, like none. No monitoring, no connection maps, nothing. I'm in the main core router and I issue a command that is *supposed* to go out to all the connected routers (36 of them, 19 domestic, 17 international) and pull their configs via TFTP to a directory on the core router. That's what the command was supposed to do according to the user manual. What the command actually did was push the router config. By 11am on day two of my new job, I took down a 400 million dollar multinational company. I'm fairly sure I started hyperventilating right in the server room, certain I'm going to be fired. Turned out it was a known bug and since all the engineers had logins to all of the network equipment, the network going down regularly was kind of expected, but I'll never forget that feeling as the TFTP command pushed configs out in front of me.
|
|
#
?
Sep 2, 2016 15:59
|
|
|
Arsten posted:As a person who evaluates business processes, I'm just going to laugh until I end up in dry heaves and pass out.
|
|
#
?
Sep 2, 2016 16:07
|
|
|
air- posted:Gotta add what it looked like in the actual paper:
|
|
#
?
Sep 2, 2016 16:46
|
|
|
"I just wanted to follow up on my previous email regarding ZeroFOX...." I give ZeroFOX about your company.
|
|
#
?
Sep 2, 2016 16:52
|
|
|
Two stand out to me, mostly because these were really bush league mistakes: Took down the #1 fragrance retailer for a a half hour because I thought I could solve a spanning tree loop by reintroducing the loop during prime business hours Knocked out public wireless for 3 states for 12 hours because my CoPP policy backfired and only allowed 100Mb traffic through when it is typically 80Gb of traffic Haven't rekt any clients infrastructure so far at the new place, trying to keep it that way so I can keep up this aura of perfection I've got going on.
|
|
#
?
Sep 2, 2016 16:53
|
|
|
Vulture Culture posted:I absolutely agree. Most business units of most companies are hosed to Idiocracy levels. It doesn't change the central point that there are entire fields of study dedicated to organizational behaviors and why people don't just do the rationally-self-interested bare minimum constantly. Except....that they do. It's why "Idiocracy levels" of fuckup exists at most companies in the first place. People want to have a job for a variety of reasons, but if they aren't engaged, they cut all the corners they can. If they are trying to over-achieve, they will cut all the corners they can, too, just in ways you don't immediately see. If this wasn't true, you could have Infosec work on an honor system internally. "Use this format for your passwords." could go out as a memo, no need to actually set the password policy on your domain to match the memo, right? Then your people's passwords will be perfect, right? You can trust them to do 1 upper, 1 lower, 1 number, and 1 special character with a different password for each internal system.....right? Certainly multiple studies over the years saying that the most common password is "password" doesn't actually mean the people in a good culture with good incentives, hey? Of course not. People will cut every corner to save themselves time, effort, or attention so that they can function at a lower level in any of the three. You need to assume, even if you have a good culture with good incentives, that you won't live in the Brothers' Grimm book for very long and that people will actually ignore anything they can to reduce the effort they put in. Worse is that this is functionally viral: if someone gives them a "good trick" such as "Enter just the number 1 for your password and it'll be quicker across all 5 systems you have to login to!" they will take it, use it, and spread it.
|
|
#
?
Sep 2, 2016 17:41
|
|
|
DigitalMocking posted:Re: taking down large companies, I think we could use a round of stories, so fess up. Here's mine I was working in govt at the time when I did my best one. Was working with the server crew on setting up a new virtual environment so I was chillin working on getting them all configured via putty. Meanwhile, some annoying request from the courthouse gets escalated to me, so I pop open a putty window to their MDF switch and pull some logs/stats to show that it was indeed not a network problem, which it was not. This is a common thing since their network over there is a piece of poo poo even by local government standards and in desperate need of an upgrade so sometimes it really is a network problem. Then I went to get a drink or go to the bathroom or something, don't remember exactly. Then I come back in and continue where I left off on the new switch for our new virtual environment, which was trunking the ports to the hosts. int po 1 spanning-tree portfast trunk are you sure idiot don't do this in the wrong spot - hit Y int po 2 spanning-tree portfast trunk are you sure idiot don't do this in the wrong spot - hit Y Phone immediately lights up, NOC screen goes red, courthouse is down. Realize immediately after that that I had the wrong putty window open and just portfast trunked the links from their main switch down to their two main distribution switches (all 3 of which were in a ring) which unsurprisingly made their entire horrible network poo poo the bed. I unfucked it immediately but the damage was done. They had a bunch of old rear end access switches over there all in a single broadcast domain, some old Ciscos that had err-disabled but didn't have autorecovery enabled, other unmanaged/garbage tier switches that had simply locked up and needed to be power cycled to get them to come back. There were a bunch of these many of which were not actually where the map said they were, my favorite of which was a 2950 which had been sealed inside a wall for some unknown reason that was serving a clerk area. It took us a while to find that one and we had to remove a bit of drywall to get to it. Apparently there had been an access panel for something there that had been removed and patched while leaving the switch inside. It took multiple hours for us to get the whole network back online and we had some extremely pissed off attorneys, judges and staff for a bit. They ended up using some of their asset forfeiture money to buy some shiny new gear to make the network not suck after that and I never did get reprimanded or anything cause lol local government. Nuclearmonkee fucked around with this message at 17:59 on Sep 2, 2016 |
|
#
?
Sep 2, 2016 17:56
|
|
|
Nuclearmonkee posted:my favorite of which was a 2950 which had been sealed inside a wall for some unknown reason This happened to the NOC at an elementary school. The reason? The NOC was in a closet with a large window in the door and it was right across the hall from the Principals' office. The "blinky lights" disturbed her. When they were doing some upgrades, she used some of the discretionary construction budget to take the door out and drywall it. Instead of, you know, buying a solid door or even construction-papering the inside of the window. 
|
|
#
?
Sep 2, 2016 18:27
|
|
|
Arsten posted:Except....that they do. It's why "Idiocracy levels" of fuckup exists at most companies in the first place. People want to have a job for a variety of reasons, but if they aren't engaged, they cut all the corners they can. If they are trying to over-achieve, they will cut all the corners they can, too, just in ways you don't immediately see. You're deeply cynical and also what you're saying isn't actually born out by research.
|
|
#
?
Sep 2, 2016 19:18
|
|
|
NippleFloss posted:You're deeply cynical and also what you're saying isn't actually born out by research. Unfortunately, calling someone cynical doesn't hold up in court. Which is where a lot of security incidents have a good chance of winding up, especially if the company is publicly traded or deals in HPI.
|
|
#
?
Sep 2, 2016 19:37
|
|
|
I don't think it's cynical to believe that individuals all have their own goals and priorities, and will act accordingly. IT security is not a priority or goal to most people, so if they can minimize the amount of time they spend on tasks that support that goal, they probably will.
|
|
#
?
Sep 2, 2016 19:40
|
|
|
|
| # ? Jun 10, 2024 09:06 |
|
|
psydude posted:Unfortunately, calling someone cynical doesn't hold up in court. Which is where a lot of security incidents have a good chance of winding up, especially if the company is publicly traded or deals in HPI. The issue is that this attitude leads to rigid and inflexible security policy that doesn't take into account business needs and leads to a lot of shadow IT where people feel like they have to work around onerous policies so that they can do their jobs effectively. So then you spend more money on more tools to chase down these things (unauthorized devices, cloud sync services, mobile data protection) and the people in turn find different tools and you end up in an arms race with your own users because you've essentially been put into opposition with one another by the corporate structure. You simply can't consider security in isolation from broader business goals and the impact on employees and productivity, but that's exactly what you get when the infosec team is a little isolated silo that "owns" the network. There's always a balance between security and flexibility and it's almost impossible for a single team focused very narrowly on network policy to decide what the correct balance is, and how to achieve it. That needs to be a conversation involving all of the stakeholders. Dr. Arbitrary posted:I don't think it's cynical to believe that individuals all have their own goals and priorities, and will act accordingly. An employee's goals and priorities aren't something they just invent for themselves, they are determined by how the company incentivizes or dis-incentivizes behavior. For instance if you make it very hard for people to do their jobs and maintain security then they will find ways to do it insecurely. If you make it easier they are less like to attempt to bypass those policies. The issue is trying to fix human organizational problems with technology, which is backwards.
|
|
#
?
Sep 2, 2016 20:05
|
|