|
Powdered Toast Man posted:One of you expressed concern for my livelihood (thanks), so I thought I'd let y'all know that rumors of my death have been greatly exaggerated. Good to hear, and good luck with the hopefully more long-lived job!
|
#
?
Oct 31, 2013 19:21
|
|
|
|
| # ? May 9, 2024 22:22 |
|
|
Something that annoys the poo poo out of me: DocuSign. An email with the name of some large business partner of ours, but not actually from that business partner, with a shady looking link to a website that's not the website of that business partner, asking for personal information? This is textbook phishing and all my users are trained well enough to pick up on it - I get asked every time if it's legit. Of course every time I confirm that it's okay, it wears away on their conditioning and they become less sensitive to real phishing. DocuSign claims their "the fastest way to get a signature." No, you rear end in a top hat, that's still printing, signing, and scanning. Oldschool, but it loving works.
|
|
#
?
Oct 31, 2013 19:39
|
|
|
Helushune posted:Over the summer almost every single one of our inkjets' print heads gummed up (due to low use because I work for a school, I guess Throw them away. Inkjets are poo poo.
|
|
#
?
Oct 31, 2013 19:40
|
|
).
|
evol262 posted:Having recently moved from systems admin/engineering into development, it's hard to argue with the idea that change management is a dying process, but I can't help but believe that there's a lot more tradecraft than babysitting logrotate and cronjobs. Particularly in web shops, there's an argument for spinning up more frontend instances which proxy back to your actual app servers when performance problems come up, but AWS isn't that cheap in the long run, and scaling out/up rather than looking more closely at your infrastructure turns you into Twitter 5 years ago.
|
|
#
?
Oct 31, 2013 19:51
|
|
|
Rhymenoserous posted:Throw them away. Inkjets are poo poo. I wholeheartedly agree. I've been pushing laserjets pretty hard for anyone who needs a personal printer and they're finally starting to listen. I'm going to scream the next time I see a clogged up print head. The best part is that a lot of the newer ones we were buying don't have any user serviceable parts. Caged posted:Get auditing enabled on those shares. Even if you aren't allowed to call people on deleting stuff accidentally it at least gives you a bit of a lead in tracking down if it's some dodgy software wiping things out. I should have done this right when you suggested it. It looks like someone logged in and nuked all of our shadow copies on that server in the middle of restoring it. Now I'm trying to figure out how to go back to "inconvenience" from "catastrophic failure". Does anyone know how to possibly restore deleted shadow copies? All the drives in the server's raid 6 array are healthy. Our previous sysadmin thinks it might have been a software hiccup somewhere but I think it looks malicious. And before anyone says anything, I've already changed all of our domain admin passwords.
|
|
#
?
Oct 31, 2013 19:54
|
|
|
I hope whoever is responsible for your backups has been doing their job. You could probably run getdataback and recover most of the data if you stop writing to the server right now, but it's going to be an unpleasant process.
|
|
#
?
Oct 31, 2013 19:57
|
|
|
Since you can't pull the disks out and analyse them individually you need to stop anything happening on that server that is writing to the disks. Cancel all scheduled tasks, turn off as much poo poo as possible. As for the recovery I have no idea. But if for example that box also caches Windows Updates then each minute that goes by is reducing the amount of stuff you can get back.
|
|
#
?
Oct 31, 2013 19:59
|
|
|
Misogynist posted:Infrastructure engineering will always have a place, and there will always be people who focus on problems of scalability. That doesn't necessarily translate into "software shops need full-time employees whose job is just to manage systems", though. I think we're in agreement here anyway. I'm not trying to make the argument that a full-time sysadmin is necessary, but rather that there are some sysadmin-ish tasks which a purely development shop does badly, and an operations person or part-time infrastructure person smooths over those bumps.
|
|
#
?
Oct 31, 2013 20:05
|
|
|
Isn't that where people turn to PaaS over IaaS or just sign contracts with MSPs if they really need an on-site element?
|
|
#
?
Oct 31, 2013 20:07
|
|
|
Thankfully, the machine is pretty much just a bulk file server with a large raid array in it since it was built before we moved over to virtualizing everything. Unfortunately, it's also the host to a bunch of our DFS shares and it's too vital to take down for the day. It's been on our list to phase out and re purpose but we haven't gotten around to it yet. What's really great is that my boss is out for the rest of the day, I can't reach him, and I'm flying solo. 
|
|
#
?
Oct 31, 2013 20:12
|
|
|
evol262 posted:I think we're in agreement here anyway. I'm not trying to make the argument that a full-time sysadmin is necessary, but rather that there are some sysadmin-ish tasks which a purely development shop does badly, and an operations person or part-time infrastructure person smooths over those bumps. Good luck finding someone someone good to do part time skill position work.
|
|
#
?
Oct 31, 2013 20:15
|
|
|
Sickening posted:Good luck finding someone someone good to do part time skill position work. That person is honestly Misogynist. Or me. Or any number of other people who have a background in infrastructure but are capable of fulfilling other roles as well because systems administration is static and all the cool stuff was happening in configuration management or weird projects so they branched out. I still enjoy infrastructure work, but I'd off myself if I were doing it 40 hours a week, and the bits of it I get in my current job are enough to keep my skills up to snuff but minor enough that I can do other things as well. It's not hard to find systems people who can wear other hats. E: By "part-time" I mean "spend 5 hours a week on this", not "hire a part-time employee".
|
|
#
?
Oct 31, 2013 20:19
|
|
|
Yeah but would you want to pull them off whatever projects they were doing for some dumb infrastructure problem?
|
|
#
?
Oct 31, 2013 20:20
|
|
|
Caged posted:Yeah but would you want to pull them off whatever projects they were doing for some dumb infrastructure problem? As opposed to living with an infrastructure problem or keeping a dedicated person on staff who spends most of their day on SA because your automated shop doesn't need constant handholding?
|
|
#
?
Oct 31, 2013 20:26
|
|
|
evol262 posted:As opposed to living with an infrastructure problem or keeping a dedicated person on staff who spends most of their day on SA because your automated shop doesn't need constant handholding? People work in places that don't need hand holding? What utopia is this? Sickening fucked around with this message at 20:33 on Oct 31, 2013 |
|
#
?
Oct 31, 2013 20:31
|
|
|
I'm trying to get an understanding of how this would be handled myself so maybe I'm just way off the mark here, but wouldn't a contract with another company to look after that sort of stuff be better in the long run? It doesn't involve someone extra being employed, and it means your guys who don't do infrastructure day to day don't have to drop into infrastructure mode to fight fires.
|
|
#
?
Oct 31, 2013 20:37
|
|
|
Sickening posted:Good luck finding someone someone good to do part time skill position work. Caged posted:I'm trying to get an understanding of how this would be handled myself so maybe I'm just way off the mark here, but wouldn't a contract with another company to look after that sort of stuff be better in the long run? It doesn't involve someone extra being employed, and it means your guys who don't do infrastructure day to day don't have to drop into infrastructure mode to fight fires. There will always be problems with production applications, but the idea is to limit the scope of problems you're responsible for to problems you're actually good at fixing. Vulture Culture fucked around with this message at 20:46 on Oct 31, 2013 |
|
#
?
Oct 31, 2013 20:40
|
|
|
Pissing me off: Support guy keeps asking me when I'm going to coordinate a site visit to fix what we've clearly established is a client-side problem. (One client isn't resolving a certain DNS record, but the one right next to it is, and the problem started after creating a new local profile.) I've already told him I'm not; I'm running all over the place until the end of the year and do not have time for this. I fought hard to limit the scope of my job to administration a few years ago, and things keep slip-sliding into "network admin + guy to call when the underskilled support staff can't figure something out." If you can't reach a website - but can reach others - and you're trying to hand this off without even doing so much as a quick nslookup, you need to stop whatever you're doing and get to work on castrating yourself. CatsOnTheInternet fucked around with this message at 20:55 on Oct 31, 2013 |
|
#
?
Oct 31, 2013 20:48
|
|
|
Caged posted:I'm trying to get an understanding of how this would be handled myself so maybe I'm just way off the mark here, but wouldn't a contract with another company to look after that sort of stuff be better in the long run? It doesn't involve someone extra being employed, and it means your guys who don't do infrastructure day to day don't have to drop into infrastructure mode to fight fires. The idea is that everything gets automated. You run Openstack. Or Eucalyptus. Or vCenter. Or whatever. Foreman or something similar provisions machines straight from the hypervisor layer, kickstarts them, and applies a chef cookbook/puppet manifest. They register themselves as ${whatever} servers. You do all your testing on identical machines triggered from Gerrit -> Jenkins -> jClouds -> fresh Openstack instance using the same definitions as your production environment. If it fails testing, nack the commit. The only "infrastructure work" you should need is occasionally setting up a new Jenkins instance, user account, DNS record, or whatever. You don't fight fires because you can redeploy your entire environment any time you want, and you never deploy an application without a complete rebuild, plus that application's already been tested in an identical environment. What fires are there? It's also fair to say that I don't think there's a big mental switch to "drop into infrastructure mode". I already think about that class of problems when something comes up. It's second nature. evol262 fucked around with this message at 20:52 on Oct 31, 2013 |
|
#
?
Oct 31, 2013 20:49
|
|
|
Holy poo poo I hate humans. Working on clearing out an old 4 post rack. The entire back of it (1 foot of room) is a loving lattice of cables. I am yanking out unused cables and power cords. All the sudden helpdesk guy comes in and tells me the Intranet is down. I start staring at things and notice one of the XenServer hosts is powered off, WFT. Go begin the rack and start tracing the power cords for it. It is running to a 1u PDU that was laying on the ground below the lattice of cables. The power plugs and power switch are on opposite sides of the unit, so this thing has been laying on its power switch for god knows how long. So about 10 different production VMs got powered off and didn't restart on a different host because the people before me never setup HA (and our whole XenServer environment is massively misconfigured). Must move faster on these V2V migrations into the new vSphere cluster I have built in a new proper rack with some OCD cable management and all servers running to alternate vertical mount PDUs. I hate humans.
|
|
#
?
Oct 31, 2013 20:54
|
|
|
Caged posted:I'm trying to get an understanding of how this would be handled myself so maybe I'm just way off the mark here, but wouldn't a contract with another company to look after that sort of stuff be better in the long run? It doesn't involve someone extra being employed, and it means your guys who don't do infrastructure day to day don't have to drop into infrastructure mode to fight fires. We, as an MSP, support a few companies that produce web applications as their primary business. They range in size from 5 to perhaps 75, so not huge by any means. They all have their Misogynists on staff though, because the amount of client specific knowledge that we'd need to gain in order to actually meaningfully contribute to how their platform operates doesn't work so well for our business model. A windows network is a windows network, 10 is the same as 50. In comparison the web apps guys have nearly nothing in common. What we do for them is take care of their internal systems (customer service/hr/payroll users etc) leaving the operations guys to focus on the product. It works well. I can see how someone could consult on the operations side, probably in a very lucrative way, but it would have to be with a handful of clients at a time.
|
|
#
?
Oct 31, 2013 20:55
|
|
|
And I guess the movement towards treating email as any other service you'd buy in falls into that trend, because why the gently caress would you want to purchase and replace Exchange servers and employ someone to manage it when you can get someone else to do it for a flat fee a month.
|
|
#
?
Oct 31, 2013 21:01
|
|
|
Caged posted:And I guess the movement towards treating email as any other service you'd buy in falls into that trend, because why the gently caress would you want to purchase and replace Exchange servers and employ someone to manage it when you can get someone else to do it for a flat fee a month. The only reason we run in-house is because we have a lot of customization for mitigating data breaches. Outbound emails get scanned for a few types of signatures (PII, trade secrets, etc) to make sure nothing unapproved walks out the front door. To that extent, managed email services aren't up to par. If we just had a vanilla Exchange setup, though, I'd be screaming to hand it off to someone else. Jesus, I hate dealing with Exchange.
|
|
#
?
Oct 31, 2013 21:05
|
|
|
No, I will not order a apple-branded monitor for your dell computer. If you want a monitor for your personal mac laptop, you'll just have to get a full-time research position so you'll have the authority to abuse the ordering system like everyone else.
|
|
#
?
Oct 31, 2013 21:44
|
|
P.S. STILL COMPLETELY DEVOID OF MERIT
|
Caged posted:And I guess the movement towards treating email as any other service you'd buy in falls into that trend, because why the gently caress would you want to purchase and replace Exchange servers and employ someone to manage it when you can get someone else to do it for a flat fee a month. If you're a SMB, managing your own email infrastructure is throwing money away, and there's zero benefit to paying a small shop to manage it instead of just using Office 365 or another hosted service for a flat fee per month.
|
|
#
?
Oct 31, 2013 21:44
|
|
|
evol262 posted:If you're a SMB, managing your own email infrastructure is throwing money away, and there's zero benefit to paying a small shop to manage it instead of just using Office 365 or another hosted service for a flat fee per month. Yeah, I don't see the appeal of in-house Exchange anymore at all unless you're That goes for any physical infrastructure, really. The amount of times I went to a place at my old job where the "fileserver" crashed, and it turns out the "fileserver" is just some off-the-shelf Compaq running XP Home SP1 sitting in the corner of a closet, with enough dust in it that I literally have to scoop it out was...well it was more than enough to make me write all of this. Inspector_666 fucked around with this message at 03:48 on Nov 1, 2013 |
|
#
?
Oct 31, 2013 22:34
|
|
|
Inspector_666 posted:That goes for any physical infrastructure, really. The amount of times I went to a place at my old job where the "fileserver" crashed, and it turns out the "fileserver" is just some off-the-shelf Compaq running XP Home SP1 sitting in the corner of a closet, with enough dust in it that I literally have to scoop it out was...well it was more than enough to make me write all of this. The problem with that is (for the fileserver part) you'll have crazy SMB apps that still require a local file server, or worse yet a local MS SQL server. I have some side jobs that are not worth doing in offices like this (and I only do them because these are my friend's businesses). I've tried to find suitable replacements, but nothing has been practical as of yet. One of the worst offenders is in the insurance industry (small family office of agents). We went from on PC being the "server" when they first asked for help (YEARS ago) to a debian box running samba that was cron jobbing everything to an external drive on a schedule and now on to an SMB RAID 1 NAS solution + MozyPro. I wish I could do better for them, but some of the crap they need to run is ancient and no one is making suitable SaaS replacements. At least not a year or so ago the last time I got called for help that took long enough that I revisited the basic problem and potential solutions. I know it's easy to say "there has to be a way to move them off of <x>" and while that may be true in a technical sense.....these places aren't interested in moving in many cases or it's financially burdonsome. Mom & Dad have been using the same drat software to do the same things for a couple of decades and it works for them. They are ready to retire in the next decade or less and have no interest whatsoever in learning new software. Forcing Win 7 on them because they needed new desktops was bad enough. Fortunately, there no real money to be made in these markets so most of us (who aren't suckers for their friends) won't have to deal with that mess.
|
|
#
?
Oct 31, 2013 22:44
|
|
|
Most small businesses could get away with a couple of Meraki PoE switches, APs and a gateway, something not totally poo poo to store files on, a VPN link into Azure/AWS to run domain controllers on and spend the money they didn't spend buying servers on an internet connection. Unfortunately a shitbox server with no redundancy running Windows Server Essentials is £600 so there will be a lot of people cleaning up after someone who 'knew something about computers' for a long time to come.
|
|
#
?
Oct 31, 2013 22:46
|
|
|
Motronic posted:The problem with that is (for the fileserver part) you'll have crazy SMB apps that still require a local file server, or worse yet a local MS SQL server. I have some side jobs that are not worth doing in offices like this (and I only do them because these are my friend's businesses). I've tried to find suitable replacements, but nothing has been practical as of yet. One of the worst offenders is in the insurance industry (small family office of agents). Yeah but if you have to run that server, at least put it on a shelf and try to dust the room every now and then or something. I'm not saying "Oh if you can't afford a full rack and separate redundant power you should just use Dropbox you loving plebes" but you also can't just ignore your infrastructure responsibilities, especially if whatever is on that one box is mission critical. Which of course it always was. Which of course would never have backups. And of course the people never want to pay for updated hardware even when the existing poo poo is dead/dying, it's always just "Well fix this one!" Again, at my old job I once had to restart a Server 2003 box and before doing it I straight up told the business owner "Listen, there is a very real chance that this thing will simply not come back on." This was one of the few times we knew we had good backups, so that's not quite as apocalyptic as it sounds, but the guy I told it too still refused to buy a new server even though he readily admitted that "if" (it was when) this one died he would pretty much be out of business. Inspector_666 fucked around with this message at 22:58 on Oct 31, 2013 |
|
#
?
Oct 31, 2013 22:55
|
|
|
Inspector_666 posted:Yeah, I don't see the appeal of in-house Exchange anymore at all unless you're huge. If you don't have the resources to commit to a proper environment for the physical server and the staff to manage it, it will be much more trouble than it can ever be worth. Some businesses just can not trust 3rd party mail providers for confidentiality reasons etc. Most specifically lawyers and doctors are the two main ones I can think of who should avoid 3rd party hosted providers for mail and document storage. Not to be  about it but one example was a lawyer working on assisting an immigration case for someone who unknown to the lawyer is on a list for extra scrutiny from the national security agencies.The other issue with externally hosted services like that is the old issue of a contractor getting carried away with a digger and knocking out the fibres carrying the data for the area. Which has happened to us and our clients a couple of times in the last 18 months or so.
|
|
#
?
Oct 31, 2013 23:04
|
|
|
Inspector_666 posted:Yeah but if you have to run that server, at least put it on a shelf and try to dust the room every now and then or something. I'm not saying "Oh if you can't afford a full rack and separate redundant power you should just use Dropbox you loving plebes" but you also can't just ignore your infrastructure responsibilities, especially if whatever is on that one box is mission critical. Oh, I didn't think you meant that at all. We totally agree on these points, and for the people I'm not so happily helping they are at least doing the bare minimum. Inspector_666 posted:Which of course it always was. Which of course would never have backups. And of course the people never want to pay for updated hardware even when the existing poo poo is dead/dying, it's always just "Well fix this one!" I started out in the business dealing with poo poo like that. It took me some time before I learned how to fire customers. Not all of them are worth keeping. Inspector_666 posted:Again, at my old job I once had to restart a Server 2003 box and before doing it I straight up told the business owner "Listen, there is a very real chance that this thing will simply not come back on." This was one of the few times we knew we had good backups, so that's not quite as apocalyptic as it sounds, but the guy I told it too still refused to buy a new server even though he readily admitted that "if" (it was when) this one died he would pretty much be out of business. Been there, shuddered at your description. Those are the ones I fire now. They CAN NOT be made happy long term, and when things go pear shaped YOU will be the one they blame. F that.
|
|
#
?
Nov 1, 2013 00:13
|
|
|
Helushune posted:I should have done this right when you suggested it. It looks like someone logged in and nuked all of our shadow copies on that server in the middle of restoring it.  How do you do that accidentally?
|
|
#
?
Nov 1, 2013 02:21
|
|
|
Varkk posted:Some businesses just can not trust 3rd party mail providers for confidentiality reasons etc. Most specifically lawyers and doctors are the two main ones I can think of who should avoid 3rd party hosted providers for mail and document storage. Not to be Document retention and confidentiality is no worse on hosted email. If you don't have in house expertise, you're probably safer hosted with Google than with "Bob, IT consultant" grasping the nuances of SOX. Your lawyer comment is a bit . If there's one thing the NSA stuff should have showed you, it's that you're no safer on servers you host yourself, and security requires that you use GPG or another encryption method yourself on the actual text. Anything across the wire is questionable unless they only used self-signed certs and the client did as well. Illegal discovery still applies regardless.Google and Microsoft don't get their fiber cut. We had ours cut at my first job. Then the copper two weeks later. Hosting yourself isn't a safety net from municipal workers.
|
|
#
?
Nov 1, 2013 02:44
|
|
|
evol262 posted:Document retention and confidentiality is no worse on hosted email. If you don't have in house expertise, you're probably safer hosted with Google than with "Bob, IT consultant" grasping the nuances of SOX. I think Inspector_666 fucked around with this message at 03:45 on Nov 1, 2013 |
|
#
?
Nov 1, 2013 02:52
|
|
|
Inspector_666 posted:Yeah, I don't see the appeal of in-house Exchange anymore at all unless you're huge. There are still tons of reasons to run in house exchange and doing so is easier than ever if you have any sort of competent on site staff.
|
|
#
?
Nov 1, 2013 03:35
|
|
|
Syano posted:There are still tons of reasons to run in house exchange and doing so is easier than ever if you have any sort of competent on site staff. Yeah, sorry, I didn't realize I actually did say "huge" in my initial post.
|
|
#
?
Nov 1, 2013 03:45
|
|
|
stubblyhead posted:
There's no way it was accidental. How does three months worth of shadow copies just disappear without a trace when the raid array's healthy and I'm the only IT person on staff at that moment and I was in the middle of a restore? There's a couple other factors, such as an admin account logging on to the server at 7:19am when none of the IT staff was on campus yet, but this screams malicious intent to me.
|
|
#
?
Nov 1, 2013 04:59
|
|
|
Helushune posted:There's a couple other factors, such as an admin account logging on to the server at 7:19am when none of the IT staff was on campus yet, but this screams malicious intent to me. Oh dear. Time to start documenting EVERYTHING you encounter, get the boss in on it too so he doesn't think you're the idiot who can't handle a simple accidental deletion. Oh, and change all your "extra-rights" passwords, from backup accounts to test accounts to admins. (You've already done this, I assume, but it still bears mentioning)
|
|
#
?
Nov 1, 2013 10:01
|
|
|
Sounds like someone's covering something up maybe? Buckle in, this could get interesting.
|
|
#
?
Nov 1, 2013 12:20
|
|
|
|
| # ? May 9, 2024 22:22 |
|
|
Calling it now, it's going to be something terrible like child porn.
|
|
#
?
Nov 1, 2013 13:31
|
|
