|
Jaded Burnout posted:Camels are also an invasive species in Australia from what I remember and trucking around the interior in swarms, so they're probably glad to be rid of them.
|
#
?
Jul 8, 2020 20:56
|
|
|
|
| # ? May 30, 2024 16:33 |
|
|
Arquinsiel posted:Oh I know why Australia wants rid, but not why Saudi Arabia wants more... Saudis eat camel meat.
|
|
#
?
Jul 8, 2020 21:05
|
|
|
Well that explains that then. Now I know.
|
|
#
?
Jul 8, 2020 21:11
|
|
|
Agrikk posted:Saudis eat camel meat. What do they do with the sand?
|
|
#
?
Jul 9, 2020 00:08
|
|
|
bell jar posted:What do they do with the sand? Desert sand is not useful for a lot of building activities. You need ocean sand, which they don't have enough of.
|
|
#
?
Jul 9, 2020 00:16
|
|
|
Motronic posted:Desert sand is not useful for a lot of building activities. You need ocean sand, which they don't have enough of. You want sand that has a sharp grain structure so when it's packed together it holds it's shape. Sand dune sand is more rounded and doesn't do well as fill for your fake-UN map of the world artificial island project. There is actually a thriving black market for sand, and sand thieves are a thing. They stripped the beaches clean overnight on one island, got shot at by the locals, it was a fun read.
|
|
#
?
Jul 9, 2020 00:36
|
|
|
https://en.wikipedia.org/wiki/Sand_theft This is a THING?? Really??? 
|
|
#
?
Jul 9, 2020 01:36
|
|
|
my cat is norris posted:https://en.wikipedia.org/wiki/Sand_theft https://www.npr.org/sections/money/2018/07/13/628894815/episode-853-peak-sand
|
|
#
?
Jul 9, 2020 01:42
|
|
|
Musical accompaniment for sandchat. https://www.youtube.com/watch?v=YGqdjaz2Upg
|
|
#
?
Jul 9, 2020 01:47
|
|
|
Jaded Burnout posted:Camels are also an invasive species in Australia from what I remember and trucking around the interior in swarms, so they're probably glad to be rid of them. I hit a camel in a mine-spec LandCruiser once, driving through scrubby territory near the Tanami Desert. Just turned a corner at 40km/h and he was sitting in the middle of the road, I guess chilling? It ruined his day (dead) and ruined mine (had to spend an hour pressure washing him out of the bullbar and grill). Emus aren't invasive, they're endemic, but because they don't eat much and they don't really live anywhere anyone else wants to live they're fine.
|
|
#
?
Jul 9, 2020 01:59
|
|
|
bell jar posted:What do they do with the sand? Sand blasting, for one. Australian sand is coarse enough to scrub paint off the surface yet fine enough to be blown out of a nozzle.
|
|
#
?
Jul 9, 2020 02:14
|
|
|
Agrikk posted:Sand blasting, for one. Australian sand is coarse enough to scrub paint off the surface yet fine enough to be blown out of a nozzle. Much like the Australian people themselves, really
|
|
#
?
Jul 9, 2020 03:29
|
|
|
One of the major exports of North Korea, apart from drugs, counterfeit money and cyber crime, is sand. They're sanctioned to hell though so there are illegal sand deals being done in the ocean where it's transfered between ships to hide it's origin. https://www.msn.com/en-us/news/world/north-korea-is-making-millions-selling-sand-yes-sand/ar-BB15gxrD
|
|
#
?
Jul 9, 2020 07:40
|
|
|
Sand sanctions is Tom Clancy level poo poo
|
|
#
?
Jul 9, 2020 09:27
|
|
|
This was a fun RFO to see today:quote:It was identified that while working on card replacement activity on the gateway, the engineer disconnected the VoIP Gateway from the network. Somebody probably got fired for that one.
|
|
#
?
Jul 9, 2020 14:48
|
|
|
Kaethela posted:This was a fun RFO to see today: We had someone disconnect the power from the production storage for our core business system at 2 in the afternoon. He's still here.
|
|
#
?
Jul 9, 2020 15:42
|
|
|
About six months before I started, one of the other engineers managed to delete the VMware lun that had our at the time primary on premise business application/SQL server. In the middle of the day. He’s still employed, but no longer an engineer or has access to mess anything up except for the area he is now responsible for.
|
|
#
?
Jul 9, 2020 15:55
|
|
|
my cat is norris posted:https://en.wikipedia.org/wiki/Sand_theft there used to be plans for a big construction some 300m away from where i live, but when yugoslavia went hosed, so did the plans. the giant piles of sand they had prepared for construction stayed for a couple years though. when they finally decided to maybe go pick up their sand, like half of it was missing, every month you'd see a car or two with a trailer and no plates stop by and pick some up. also, it's how one of my neighbours renovated their house real cheap.  fake edit: i see someone already posted about the jamaican beach too
|
|
#
?
Jul 9, 2020 16:07
|
|
|
Kaethela posted:This was a fun RFO to see today: quote:Somebody probably got fired for that one. A few years back we had a zero-day vuln in a web interface lead to someone managing to install a PHP shell on a few of our systems. I was using MultiSSH to look for and clean it up on all of the relevant machines at once. Turns out the folder that it landed in didn't even exist on some of those machines, and I didn't notice when a few of them threw an error to my "cd" command. Went to delete that part of the folder structure to replace with a known good copy and noticed "hmm, this delete command is taking longer than expected" which then immediately turned in to a sinking feeling. Fortunately this was at rear end-thirty in the morning so I had a few hours before the customers using those systems would be in, so I got to give our backups a real-world test. I think everyone who's been in this industry for long enough has either been directly or indirectly responsible for some kind of outage at least once. As long as we're all learning and not repeating the same mistakes it is what it is.
|
|
#
?
Jul 9, 2020 16:33
|
|
|
Kaethela posted:This was a fun RFO to see today: My network guy took out our whole virtual environment by disconnecting the "backup" network in the middle of the day. Had a horrible cascade effect. He also changed the VPN routing address mid day a few weeks ago. Gg taking an entire company offline in the middle of a pandemic while everyone is working from home. He still works here. People like that never get fired in my experience.
|
|
#
?
Jul 9, 2020 16:40
|
|
|
wolrah posted:I think everyone who's been in this industry for long enough has either been directly or indirectly responsible for some kind of outage at least once. As long as we're all learning and not repeating the same mistakes it is what it is. And if you haven’t, you clearly haven’t been trusted with access to critical systems. That’s my favorite interview question: “tell me about a time you caused an outage, the impact to the business, the recovery process, and what you learned from it?” If they can’t think of one they’re either lying or never had enough access to break things.
|
|
#
?
Jul 9, 2020 17:24
|
|
|
I took our entire phone system down for a full day because I power cycled our ucm server. gently caress ucm.
|
|
#
?
Jul 9, 2020 17:26
|
|
|
What did you think was gonna happen?
|
|
#
?
Jul 9, 2020 17:29
|
|
|
The Fool posted:I took our entire phone system down for a full day because I power cycled our ucm server. Yes but when approached about this did you apologize and try to fix your mistake as quickly as possible OR did you immediately blame the client engineering folks for not reading your mind on what you THOUGHT the new VPN address should have been in every client in the company? Because my network dude immediately implied this was ~my fault~ despite never putting in a CM to change the VPN or add a backup VPN address into our clients.
|
|
#
?
Jul 9, 2020 17:32
|
|
|
GreenNight posted:What did you think was gonna happen? The entire unit was already non-responsive on the network. Phones weren't working, the vm's weren't responsive, and vsphere wasn't responsive. I was expecting that power-cycling the unit would allow everything to boot back up. I was not expecting the ucm vm to become corrupt and then have to spend my entire day restoring from backups. silicone thrills posted:Yes but when approached about this did you apologize and try to fix your mistake as quickly as possible OR did you immediately blame the client engineering folks for not reading your mind on what you THOUGHT the new VPN address should have been in every client in the company? Because my network dude immediately implied this was ~my fault~ despite never putting in a CM to change the VPN or add a backup VPN address into our clients. Oh, I immediately went to my boss and told him I hosed up and what I was going to do to fix it. e: I think this was the issue: https://quickview.cloudapps.cisco.com/quickview/bug/CSCvh55176 I don't remember the bit about vmtools, but the bug has been updated since we had the problem The Fool fucked around with this message at 17:56 on Jul 9, 2020 |
|
#
?
Jul 9, 2020 17:50
|
|
|
I think I've told this story before, but I rebuilt a door access controller as a VM after the old PC running it failed, restored a backup, everything looked good, then I connected it up to the network so it could see all the doors. Obviously I had rebuilt on the latest version of the software and each door needed a firmware update, so they all dutifully sat around waiting for their turn to update, helpfully not processing any badge reads in the meantime and remaining locked.
|
|
#
?
Jul 9, 2020 18:12
|
|
|
Thanks Ants posted:helpfully not processing any badge reads in the meantime and remaining locked. Our work had a good solution for a secure door which lost power to it's access control striker for a bit... Put a key in it and leave it there. Hope no one removes it.
|
|
#
?
Jul 9, 2020 18:27
|
|
|
devmd01 posted:That’s my favorite interview question: “tell me about a time you caused an outage, the impact to the business, the recovery process, and what you learned from it?” Had to think for a minute but I remember the one time I had been using the Solaris modular debugger on the kernel of a production mission-critical DB server to get performance data so I could troubleshoot a problem. I left the window open, thinking I'd close it once I was sure the problem was solved. At the end of the day, when I wanted to shut down, I went and typed "exit" into every open window, disconnected my screen session, and logged out. The next morning I re-opened screen and saw that my mdb session was still open. Oh, right, exit isn't how you exit mdb on Solaris. Control-d is how you do that. I pressed control-d. What I found out later is that exit means nothing to mdb in and of itself, but it was in the command buffer when I hit control-d, so the mdb just passed that along to the kernel. Which is how I killed the kernel of a production mission-critical Oracle database first thing in the morning. From then on, I only ever touched mdb by using echo "exactly what I want, and yes, I checked twice" | mdb -k.
|
|
#
?
Jul 9, 2020 20:24
|
|
|
wolrah posted:That wasn't by chance somewhere in the general area of Northeast Ohio or Western Pennsylvania. was it? Had some issues with one of my carriers recently that would fit this description. Nope, this was in Europe.
|
|
#
?
Jul 9, 2020 21:15
|
|
|
I've never caused a major outage but I did accidentally push software on every computer in a 4000 computer SCCM instance by misclicking a collection once. Then we actually fixed it by rebuilding our collections and changing how we were targeting software. I also had to create a package to silently delete it. luckily it was very smol and easy though.
|
|
#
?
Jul 9, 2020 22:40
|
|
|
I once saw a sysadmin accidentally drop the production DB in the middle of the day. Restored it from a backup, wasn't too bad. Personally I once typed something like `install-bsd DESTINATION= /path/to/the/jail` instead of `install-bsd DESTINATION=/path/to/the/jail`, which the OS interpreted as `install-bsd DESTINATION=/` and installed a fresh freebsd install right over the top of itself while it was running, in a data centre 250 miles away.
|
|
#
?
Jul 9, 2020 22:54
|
|
|
silicone thrills posted:I've never caused a major outage but I did accidentally push software on every computer in a 4000 computer SCCM instance by misclicking a collection once. At least you didn’t accidentally push an image job to every computer and server on campus like that one place a few years back.
|
|
#
?
Jul 9, 2020 23:05
|
|
|
On two different occasions I (a network guy) was trying to clear a BGP peer because one of the ISPs on that router was having issues/I was troubleshooting. I typed in "clear ip bgp" with every intention of putting in the peer IP address. But hit enter. Yes, that clears all peers. And this is how you drop 10,000 active simultaneous phone calls on the floor. That can't even call back again for several minutes.
|
|
#
?
Jul 10, 2020 00:13
|
|
|
wolrah posted:That wasn't by chance somewhere in the general area of Northeast Ohio or Western Pennsylvania. was it? Had some issues with one of my carriers recently that would fit this description. I once took down the main internal tool of a regional bank for a couple hours and didn't get fired. My boss: "Restore everything from backup." Backup team: "We need to send a request to the VP for approval before we can do that." My boss: "I wasn't asking; I was telling. DO IT NOW." Motronic posted:On two different occasions I (a network guy) was trying to clear a BGP peer because one of the ISPs on that router was having issues/I was troubleshooting. I still remember when the BGP table went over (2? 4? whatever, some even number) GB. I was working at a large IT distributor at the time and all our calls for the next few days were people who suddenly found budget for a new router. KillHour fucked around with this message at 00:52 on Jul 10, 2020 |
|
#
?
Jul 10, 2020 00:46
|
|
|
I somehow can't think of any serious outages I've caused, although I'm sure there have been some, but I know I lost 32TB of security logs (that weren't backed up) due to a puppet module that made a bad assumption about home directories always being unique.
|
|
#
?
Jul 10, 2020 01:12
|
|
|
the only outage I ever caused was the the badge in system for a bunch of gyms for like 5 minutes. I was told we needed to take a core dump of the java app while it was building up to a crash we couldn't track down, so as soon as the node started acting up, I hit the core dump. I didn't realize that this froze the process
|
|
#
?
Jul 10, 2020 01:18
|
|
|
I once accidentally reverted a router to default config, including wiping PPPoE creds, thinking I was backing up its config, due, I would argue, to a very badly written web interface which made extremely ambiguous use of the term "mirror config".
|
|
#
?
Jul 10, 2020 01:22
|
|
|
I learned that a major insurance company kept the European domain controller and backup domain controller on the same switch by unplugging every ethernet cable on said switch.
|
|
#
?
Jul 10, 2020 01:41
|
|
|
My company isn't big enough for me to blow up anything too huge; we're confined to a metropolitan area. I did bring down our core services during the middle of the day for a couple of minutes one time (did an IIS reset thinking I was logged into Test; I was not logged in to Test). Not my fault but I did the actual deed: our software vendor told me a change to a calendaring function in order to automate some arduous, manual stuff when dealing with holidays would be fine. I asked for clarification, because I had a bad feeling, but was pretty new with the system, so when they came back in what seemed a vaguely annoyed tone to tell me "yes, it's fine," I made the change (one of our senior VPs wanted it done on New Years, I pushed it back to MLK day). It turns out the change they had me make made our core system skip the day entirely, meaning that when we came back on Tuesday, according to the system it was Wednesday. I discovered this the evening before, but had to push it through because once it started, we couldn't stop it. In that vendor's defense, when they saw what happened, it was all-hands-on-deck in a way I had never, ever seen from them before (and definitely not since). I put in a ticket at one point and got a call back three minutes later from THREE people on a conference, two of whom I knew were middle-managers. Do not recommend that week, though. Those meetings that you're sitting in with multiple VPs/C-levels asking a lot of questions, many of which you don't know the answers to are very, very unpleasant. Fixing it was... not fun. But honestly easier than I expected. And I learned a lot. And it isn't really something I hold against the vendor (unlike a lot of other poo poo) because they really stepped up. EDIT: Oh, poo poo, deleted our entire training database one time. Had to rebuild it, took me like a week. Completely forgot about that. The first six months in my current position were... eventful. But I learned a lot! Ham Equity fucked around with this message at 02:10 on Jul 10, 2020 |
|
#
?
Jul 10, 2020 02:06
|
|
|
|
| # ? May 30, 2024 16:33 |
|
|
Good judgement comes from experience. Experience comes from poor judgement. Let's see.... hosed up where the deny-all went in in a firewall ruleset. While working from home. Nobody was in the office thank goodness. hosed up a cd followed by an rm -rf *. I was trying to clear /Library/Caches on a dying OS X box, so reinstalling wasn't a major chore. Assigned the desktop backup scheme (which excluded videos) to the website (which had hundreds of hours of valuable digital strategy content). Then while the execs were all at a conference - on digital strategy - the RAID card on the web server died. Almost did get fired for that one. Didn't have to argue very hard in favor of cloud services after that though. Blocked our largest client's IP range in an enthusiastic anti-spam measure targeted at Northern Europe in general.
|
|
#
?
Jul 10, 2020 02:08
|
|
