|
I worked for a company that was terrified of their app going down because no one knew how to bring it back up. The whole thing was a mess a circular dependencies no one had ever mapped, code written by people who left the company a decade before, and undocumented exceptions.
|
#
?
Apr 23, 2017 17:49
|
|
|
|
| # ? May 17, 2024 12:09 |
|
|
CommieGIR posted:The biggest issue I'm running into, and I have 10 years of doing consulting with IT Operations groups and Datacenter groups, is that their processes are not documented, they have no Standards and Practices, nothing is documented. Even their infrastructure is nothing more than a drawining on a whiteboard, the only hard infrasctructure documentation they have is more than 4 years old now. 
|
|
#
?
Apr 23, 2017 17:57
|
|
|
Having flashbacks to the app that was used for the project I was on a couple jobs ago. Every server in the cluster made its own specific connections to an oracle database, and each time you added a server to a specific role the config had to have all the previous servers listed or else it wouldn't work right. To do a full restart of the thing you had to shut everything down, and then start one of each server role in the right order. Then you could start the rest. It also took 20-40 minutes for each instance to fully start so it was stupid slow. I was just glad once other people took that responsibility on and the graveyard shift people were doing all that. You are being paid because you have all this knowledge and then would be using it to babysit this dumb service for half your shift and then making tickets with the vender and calling people in the middle of the night when it didn't go as planned.
|
|
#
?
Apr 23, 2017 18:00
|
|
|
MC Fruit Stripe posted:Yep yep, that's another thing about it - 3 people who do the work that 50 people should do, don't have time for your silly rear end documenting. The problem is they have been hiring and have expanded their headcount a lot, but are too busy putting out fires that shouldn't be fires because they don't put the things in place like Logstash or Splunk and Monitoring that will help pre-empt the fires.
|
|
#
?
Apr 23, 2017 18:09
|
|
|
MC Fruit Stripe posted:They're going through what my team went through over the last few years. We exploded in growth from a mom and pop, to mid sized, to enterprise, all within a 5 year period. Go easy on them - they WANT to grow, and they realize a lot of their processes are garbage, but a lot of them were Good Enough for the time.
|
|
#
?
Apr 23, 2017 18:56
|
|
|
Gave my notice last week.  Was working for a few years for a very toxic and systemically mismanaged fintech company. Glad to be getting out of finance. Just some of the dumb poo poo I experienced throughout the years:
Pretty sure this company is slowly transforming into losing significant market share, and they'll use this as an excuse to shrink the # of products we have and thereby shrink the size of the company. They're constantly screwing over their employees, and people are quitting more and more frequently. gently caress 'em, glad I'm out. Weaponized Autism fucked around with this message at 21:10 on Apr 23, 2017 |
|
#
?
Apr 23, 2017 21:07
|
|
|
Super Slash posted:My place is going through the same thing, when I started there were like 20 people tops and now there's about 100 or more but they still run the place like a pokey little business where lots of process flow through a single person. Bugs the hell out of me because I can imagine this place growing a lot more (we need to move office again as we're out of space) and I'm in a good position to move up, but I don't want to hang on to hopes while having low pay when there's better stuff out there. In your case though, with it up to 100 people and still relying on single points of failure it's possible that they're going to take so long adapting to their new reality that it's going to damage your career, so of course, your mileage varies.
|
|
#
?
Apr 24, 2017 05:14
|
|
|
Boss strikes again. Last week. Hey Sefal, I don't know what happened when you were gone but suddenly these servers stopped working. I restored them, but I can't logon to the KMS server. It says: The trust relationship between this workstation and the primary domain failed. Do you know what this is? Me: Remove the server from the domain and then join it back to the domain. boss: Nah. it works fine now. Logged in as local admin Me: it's harmless to remove it from the domain and then join it back. When you restored it. The Sid that the domain has of the server doesn't match up with the sid of the Server itself. A week passes. Boss: Man I have no idea why this happened? domain trust lost. Me: Did you join it back to the domain? boss: No Me: I'll do it. Boss. Hey! look at that error is gone now. Logging in now as domain admin, let's see what happens  Apparently a server was struck with a virus (not crypto) on my vacation. Instead of wiping the server clean and restoring a backup. Which in turn would verify the backup procedure. Boss decided to just run a virus scanner that found and quarantined the virus. And the server has been running ever since. I have e-mailed my boss that I want to wipe the server and restore it from a backup prior to the date that it was infected. Because I believe the server can no longer be trusted after an infection. Mostly CYA. Hasn't responded yet. I expect a no. Would it be ok for me to e-mail people who are higher up about this if he does say no? I can't sit idly by on a server that i see as a ticking time bomb. Edit: Talked with boss about this "I'm not sure if a restore would work. If it happens again, we wipe it" Sefal fucked around with this message at 10:27 on Apr 24, 2017 |
|
#
?
Apr 24, 2017 09:57
|
|
|
I think if you use this as a practise run in being able to switch off what you give a gently caress about then you will improve your stress levels quite nicely. You've identified a potential risk, informed the person above you who makes the decisions, they've ignored it. Maybe bring it up in a department meeting in a "hey, not had a response on this" way but then try to let it go and stop worrying about it, otherwise you will drown under the stress of carrying everybody else's lovely decisions around with you.
|
|
#
?
Apr 24, 2017 10:27
|
|
|
Do it yourself and log it as an emergency repair. Better to ask forgiveness, etc.
|
|
#
?
Apr 24, 2017 10:28
|
|
|
Sefal posted:Apparently a server was struck with a virus (not crypto) on my vacation. Instead of wiping the server clean and restoring a backup. Which in turn would verify the backup procedure. Boss decided to just run a virus scanner that found and quarantined the virus. And the server has been running ever since. Not really something that warrants escalation. If the virus is contained, there's not even really any problem with the machine, that's the point of AV. A CYA doesn't hurt, but I don't see how this is in any way a special event if it's just a generic virus.
|
|
#
?
Apr 24, 2017 10:29
|
|
|
Thanks Ants posted:I think if you use this as a practise run in being able to switch off what you give a gently caress about then you will improve your stress levels quite nicely. You've identified a potential risk, informed the person above you who makes the decisions, they've ignored it. Maybe bring it up in a department meeting in a "hey, not had a response on this" way but then try to let it go and stop worrying about it, otherwise you will drown under the stress of carrying everybody else's lovely decisions around with you. Thank you. I will do that. At this point, I've let it go after talking to my boss about it. Which actually pulled up some more worrying stuff. But i'll deal with that when I can
|
|
#
?
Apr 24, 2017 10:29
|
|
SEKCobra posted:Not really something that warrants escalation. If the virus is contained, there's not even really any problem with the machine, that's the point of AV. A CYA doesn't hurt, but I don't see how this is in any way a special event if it's just a generic virus. This is Geek Squad working on Grandmas computer levels of bad advice. Don't ever trust antivirus programs to do anything correctly and if it detected anything engage in full risk mitigation mode.
|
|
|
#
?
Apr 24, 2017 11:59
|
|
|
milk milk lemonade posted:This is Geek Squad working on Grandmas computer levels of bad advice. Don't ever trust antivirus programs to do anything correctly and if it detected anything engage in full risk mitigation mode. Ok, you be paranoid about generic viruses which are well researched and can be traced back to an origin and have a specific behavior, I'll just work with objective facts and not get grey hair over nothing. I'm not saying ignore viruses, but a virus that was successfully quarantined never infected anything and poses no threat. Either you stick to your "AV ain't worth poo poo", then we can just ignore the virus alert all together I suppose, or you say you trust the AV to have detected something and you use their recommended solution. If you are gonna use AV, you are retarded if you then decide that you don't trust it and need to spend your precious time on a non-existent problem. Do you also call the police over unauthorized attempts to connect to port 22?
|
|
#
?
Apr 24, 2017 12:07
|
|
|
You're a loving idiot and you have no idea what you're talking about. Hope this helps! (USER WAS PUT ON PROBATION FOR THIS POST)
|
|
|
#
?
Apr 24, 2017 12:13
|
|
|
I would agree with a contained virus on a workstation not connected to my domain, but on a server, not a chance, that poo poo is getting wiped as soon as it's flagged. It's like backup thread says, RAID isn't a backup, and in the same sense the AV software on a server shouldn't be the end all solution. It should be used as a warning and then fixed properly like a failed disk on a RAID array that you have backups of. Even with known good backups I'm sure your manager would replace a failed disk in a RAID array? Rudager fucked around with this message at 12:23 on Apr 24, 2017 |
|
#
?
Apr 24, 2017 12:19
|
|
|
milk milk lemonade posted:You're a loving idiot and you have no idea what you're talking about. Hope this helps! Ok, uninstalling AV now and reinstalling every server because there were viruses touching them. Hell, I'm going to flatten our file servers, because user shares were touched by cryptoviruses, too! milk milk lemonade posted:You're a loving idiot and you have no idea what you're talking about. Hope this helps! Explain your 0-tolerance policy instead of being a oval office.
|
|
#
?
Apr 24, 2017 12:24
|
|
|
Yah I would've gladly explained it to you if you hadn't included some iditiotic comment about unauthorized port scanning/whatever you're trying to say there. It has nothing to do with a perception that virus's are magic dude. It was in my first post - risk mitigation. And even more importantly how is a server responsible for internal licensing getting hit with virus's? What was the actual attack vector? Are more servers compromised? You roll it back/destroy it in case AV didn't get everything (it never does, even the ones that can guess at what's a virus now). It's 2017 it should take you way less time to rebuild a server than deal with the potential fallout of a breach.
|
|
|
#
?
Apr 24, 2017 12:30
|
|
|
If you have a functioning backup infrastructure then why the gently caress would you bother to clean a virus out of a server 
|
|
#
?
Apr 24, 2017 13:10
|
|
|
Virus on a desktop? Contain it and clean it. Its fine. I usually wipe my desktops if they get infected, but its fine. Virus on a Server? 
|
|
#
?
Apr 24, 2017 13:49
|
|
|
I want to know how you actually get a virus on a server these days. I'm not even mad, that's impressive.
|
|
#
?
Apr 24, 2017 14:17
|
|
|
milk milk lemonade posted:Yah I would've gladly explained it to you if you hadn't included some iditiotic comment about unauthorized port scanning/whatever you're trying to say there. I don't really see the risk. I mean I run a segregated infrastructure with no AV on most servers anyway. But somehow playing up the fact that ANY virus touched a server is ridiculous. First of all you have to look at what kind of virus it was. And what kind of detection. My file server has AV so I know when a user has a virus and copies it on there if they have an outdated AV or whatever. Just because the AV detected a virus in a user folder doesn't mean I'm going into panic mode and revert to backups. The OP gave no details about what kind of virus or how it was detected. If his boss is even willing to take the liability, you really don't have to give much of a gently caress when it's possibly a harmless virus on a sheltered spooler at best.
|
|
#
?
Apr 24, 2017 14:25
|
|
|
SEKCobra posted:I don't really see the risk. I mean I run a segregated infrastructure with no AV on most servers anyway. But somehow playing up the fact that ANY virus touched a server is ridiculous. In this day and age, with breaches left and right, if the infection is on the server itself and its not a file server, its better safe than sorry to just restore the system to a prior point. I still want to know how it got infected in the first place, is it a file share?
|
|
#
?
Apr 24, 2017 14:43
|
|
|
CommieGIR posted:I still want to know how it got infected in the first place, is it a file share? My bet is a terminal server without re-directed profiles, hence the reluctance to wiping it.
|
|
#
?
Apr 24, 2017 14:53
|
|
SEKCobra posted:I don't really see the risk. I mean I run a segregated infrastructure with no AV on most servers anyway. But somehow playing up the fact that ANY virus touched a server is ridiculous. You don't see the risk because you're either not working at an enterprise level or you are on the path to getting slammed someday. Like I said, the virus itself is actually secondary to the vector. If someone can get a virus on a server they might be able to do a whole lot more and do something more effective next time. But the first thing you do is rollback or destroy that server to mitigate any additional risk that endpoint may introduce. Your scenario isn't really worth the time picking apart because it's missing the forest for the trees.
|
|
|
#
?
Apr 24, 2017 14:57
|
|
|
It is a file server. And we got hit with Derusbi malware.
|
|
#
?
Apr 24, 2017 15:01
|
|
|
Sefal posted:It is a file server. And we got hit with Derusbi malware. http://www.novetta.com/wp-content/uploads/2014/11/Derusbi.pdf Interesting reading.
|
|
#
?
Apr 24, 2017 15:15
|
|
|
Anybody have experience with Trend Micro Business blocking DRM software from running? I've got an engineer who needs to run some serial communications software for testing, but it won't run at all with our AV client running. I'm working on getting the program whitelisted with the client, but I recommended that he contact the software company because this seems really sketch as to why Trend Micro would block the new version of this software.
|
|
#
?
Apr 24, 2017 15:21
|
|
|
CommieGIR posted:http://www.novetta.com/wp-content/uploads/2014/11/Derusbi.pdf Yeah it's actively making changes to the server. Burn it and restore. No point loving around once it gains access to services and firewall. milk milk lemonade posted:You're a loving idiot and you have no idea what you're talking about. Hope this helps! But this is unnecessary in this thread, friend. IT is a wide industry full of every personally and opinion you can imagine, but we're all here because we like to share information and viewpoints.
|
|
#
?
Apr 24, 2017 15:49
|
|
|
I am sorry. That is my fault. I should have included all info in the original post
|
|
#
?
Apr 24, 2017 15:51
|
|
|
Not content to just say "I disagree", goons must attack your intelligence as well.
|
|
#
?
Apr 24, 2017 15:55
|
|
|
Thanks Ants posted:PaperCut seems to be the most recommended one - http://www.papercut.com/products/ng/manual/applicationserver/topics/account-popups.html I was recommended papercut previously. I checked it out and it looks alright. It claims to do what I need but who knows how it is to implement and get working properly. Xerox also tried to sell me a 3rd party solution; Cirrato. It seemed stupidly complex and expensive but might actually work? In other news; Cheeki Breeki phishing attacks are getting really sneaky. https://9to5mac.com/2017/04/20/how-to-spot-a-phishing-attempt-fake-apple-site/ quote:It is possible to register domains such as “xn--pple-43d.com”, which is equivalent to “аpple.com”. It may not be obvious at first glance, but “аpple.com” uses the Cyrillic “а” (U+0430) rather than the ASCII “a” (U+0061). This is known as a homograph attack.
|
|
#
?
Apr 24, 2017 16:00
|
|
|
If an unprivileged user manages to get some malware on a fully patched system, where I can be reasonably confident that there was no escalation because no one's going to be wasting zero days on my customers, I'll take a whack at cleaning it up. I'm usually willing to put about an hour in to it, which is more than enough time to clean up the simple stuff. Any more than that and it's  time because it'll be a quicker way back to a clean system.If it hit a privileged user or if there's any reasonable chance of privilege escalation there's no bothering to try to clean it up. I'd assume most people don't generally have unprivileged users logging in to the server where they can get hit by malware, so...
|
|
#
?
Apr 24, 2017 16:24
|
|
|
Terminal servers are another thing, those should be locked down so much the virus can't escape the user context even if for some retarded reason you let them run software. And that's why the attack vector and virus are important, if it's using a zero-day, you might be hosed, if it's gome generic police-lockdown thing, destroy the session and dump the profile, done.
|
|
#
?
Apr 24, 2017 16:56
|
|
|
 after nearly three years at this place. Which wasn't bad after the first couple of months and my managers (with exactly one exception) were freaking amazing, but no amount of good people is going to help when you're underpaid and have an itch to get out of the industry in the next five years or so. Man it's been forever since I've had to give notice. OWLS! fucked around with this message at 17:18 on Apr 24, 2017 |
|
#
?
Apr 24, 2017 17:15
|
|
|
Vargatron posted:Not content to just say "I disagree", goons must attack your intelligence as well.
|
|
#
?
Apr 24, 2017 17:41
|
|
|
anthonypants posted:When the shoe fits, You get to live happily ever after with your prince? OWLS! posted:
I spent a little over 10 years at my last job. The only reason I had any idea of what to expect from the job search process was because of reading these threads. The entire job search/interviewing/giving notice process was so anxiety inducing I'm surprised I followed through with it. Now that I'm 8-ish months into the new job, it is clear to me that I spent 3-5 years too long and the last one, and am going to have to work hard to make up that time.
|
|
#
?
Apr 24, 2017 18:00
|
|
|
SEKCobra posted:Terminal servers are another thing, those should be locked down so much the virus can't escape the user context even if for some retarded reason you let them run software. And that's why the attack vector and virus are important, if it's using a zero-day, you might be hosed, if it's gome generic police-lockdown thing, destroy the session and dump the profile, done. Yeah, terminal servers are an entirely different beast. I have one customer that has them and we do nightly exports of the VM images for both. If despite our efforts to idiot-proof it nature manages to send a better idiot our way we can have everyone switch over to the unaffected server and it'll take about two hours to restore the broken one from the previous night's backup. So far the worst we've ever had happen is someone managed to get an honest to goodness P2P virus which just filled the user's home directory with "britney spears sex tape.mpg.scr" type stuff.
|
|
#
?
Apr 24, 2017 19:31
|
|
|
Anyone using ADFS with O365, I've been getting some mileage out of this tool: http://jackstromberg.com/o365-smart-linksso-link-generator/
|
|
#
?
Apr 24, 2017 19:56
|
|
|
|
| # ? May 17, 2024 12:09 |
|
|
wolrah posted:If despite our efforts to idiot-proof it nature manages to send a better idiot 
|
|
#
?
Apr 24, 2017 20:11
|
|