[SPAM] FW: RE: The logs could do with a courtesy flush

The Something Awful Forums > Archives > Comedy Goldmine > The Goodmine > [SPAM] FW: RE: The logs could do with a courtesy flush

«‹›501 »

orange juche: Mar 14, 2012

Hotel Kpro posted:

That sucks man. We were up against the wall trying to get a good image going. For some reason the one we were using broke outlook, so we upgraded another one and poo poo was down to the wire more than once. Except it broke active client so that was fun to fix too

Chromium edge is coming down the pipe soon too, to be forced on us by the deprecation of classic edge or whatever they're calling it. I can't wait for it to break half of everything because everything in the government is built on loving pick up sticks.

# ? Mar 12, 2021 07:10

Adbot: ADBOT LOVES YOU

# ? May 26, 2024 00:03

orange juche: Mar 14, 2012

Btw if you have on prem exchange servers loving bend over champ, cause its :goatsecx:

time

https://www.lawfareblog.com/microsoft-exchange-hack-and-great-email-robbery

quote:

As I write this, the world is probably days away from the �Great Email Robbery,� where a large number of threat actors around the globe are going to pillage and ransom the email servers of tens of thousands of businesses and local governments. Or at least pillage those that the purported Chinese actors haven�t already pillaged.

On Mar. 5, the investigative journalist Brian Krebs reported that an �unusually aggressive Chinese cyber espionage unit� had gained access to more than 30,000 U.S. organizations. The New York Times detailed on Mar. 6 that �The number of victims is estimated to be in the tens of thousands and could rise.� How did the attackers breach the companies? The Chinese actors developed a way to hack Microsoft Exchange and then attacked the organizations from there. And many of those attacked are still vulnerable to follow-on attacks not just by the Chinese but numerous criminals. The impact of the Exchange hack will certainly be greater than SolarWinds and researchers aren�t even close to the end of the story. But it�s a complicated story, with a lot to untangle.

What is Microsoft Exchange?

If you send an email, your computer contacts an email server. This server both stores your email and communicates with other email servers. A large number of companies outsource their mail servers to cloud-service companies like Microsoft or Google, commonly spending $6-12 per user per month. Your email account�firstname.lastname@companyx.com�may well rely on these Google or Microsoft servers.

Running a mail server is often difficult, so fraught with peril that many professional computing institutions (such as both the University of California Berkeley and the International Computer Science Institute, where I work) outsource our email to Google, others outsource it to Microsoft. Yet it is hard to argue with economics, so many companies will just run their own mail server, either buying the software from Microsoft or some other company. This can save $100,000 a year for a 1,000-person business.

Microsoft Exchange is one of the most popular mail servers because it works very well within a Windows environment. It also includes substantial features such as integration with voicemail, a webmail interface, and is practically guaranteed to work with Microsoft Outlook and Office.

Exchange centralizes all of a company�s emails. This means that if you compromise a company�s Microsoft Exchange server you can now see every email sent or received. This makes the mail server a very very tempting target for attackers. On Mar. 2, Microsoft released a series of patches for four exploits that were under active targeting by Chinese threat actors. Microsoft released the patches earlier than expected, opting not to wait for the traditional �Patch Tuesday.� Microsoft made the decision to go forward with the early release because the vulnerabilities were actively exploited; the early and unexpected release of the patch was an attempt to stop future exploitation.

What are the vulnerabilities?

The Chinese actors were not using a single vulnerability but actually a sequence of four �zero-day� exploits. The first allowed an unauthorized user to basically tell the server �let me in, I�m the server� by tricking the server into contacting itself. After the unauthorized user gained entry, the hacker could use the second vulnerability, which used a malformed voicemail that, when interpreted by the server, allowed them to execute arbitrary commands. Two further vulnerabilities allow the attacker to write new files, which is a common primitive that attackers use to increase their access: An attacker uses a vulnerability to write a file and then uses the arbitrary command execution vulnerability to execute that file.

Using this access, the attackers could read anybody�s email or indeed take over the mail server completely. Critically, they would almost always do more, introducing a �web shell,� a program that would enable further remote exploitation even if the vulnerabilities are patched.

What is the timeline?

The investigative journalist Brian Krebs has produced a handy timeline of events and a few things stand out from the chronology. The attacker was first detected by one group on Jan. 5 and another on Jan. 6, and Microsoft acknowledged the problem immediately. During this time the attacker appeared to be relatively subtle, exploiting particular targets (although we generally lack insight into who was targeted). Microsoft determined on Feb. 18 that it would patch these vulnerabilities on the March 9th �Patch Tuesday� release of fixes.

Somehow, the threat actor either knew that the exploits would soon become worthless or simply guessed that they would. So, in late February, the attacker changed strategy. Instead of simply exploiting targeted Exchange servers, the attackers stepped up their pace considerably by targeting tens of thousands of servers to install the web shell, an exploit that allows attackers to have remote access to a system. Microsoft then released the patch with very little warning on Mar. 2, at which point the attacker simply sought to compromise almost every vulnerable Exchange server on the Internet. The result? Virtually every vulnerable mail server received the web shell as a backdoor for further exploitation, making the patch effectively useless against the Chinese attackers; almost all of the vulnerable systems were exploited before they were patched.

This is a rational strategy for any actor who doesn�t care about consequences. When a zero-day is confidential and undiscovered, the attacker tries to be careful, only using it on attackers of sufficient value. But if the attacker knows or has reason to believe their vulnerabilities may be patched, they will increase the pace of exploits and, once a patch is released, there is no reason to not try to exploit everything possible.

So what is next?

Unfortunately these vulnerabilities are reportedly easy to exploit. To make matters worse, the patches that fix this problem provide a guide to reproducing the exploit. I would expect these exploits to be in criminal toolkits shortly and that the world is, at most, days away from ransomware gangs mass-exploiting Exchange servers, encrypting the contents, and offering the victims a choice: pay up, or your emails will be published for everyone else and deleted from your own servers.

Even patched servers aren�t out of the woods: There�s a very high probability that they were already compromised and a web shell installed before administrators applied the patches. Since the web shell is a backdoor into the server not removed by patches, the resulting systems remain vulnerable. This web shell can be used by the original installer or, possibly, by the same ransomware gangs about to mass-exploit the still unpatched servers.

So any company running an Exchange server, whether or not they might be a target of Chinese espionage, needs to look for and remove such backdoors. And those companies whose Exchange servers give way to even a remote chance for spying should probably rebuild their mail-servers completely. The web shell is simply the first of many possible backdoors the attacker might have installed.

And now the Biden administration has a real hard policy problem: What now? The SolarWinds hack may have been significant, but this will affect far more institutions. The SolarWinds hackers stayed subtle. They targeted traditional intelligence targets and never transitioned to a �pillage everything� model, which made that attack more of a �Spies Gonna Spy� operation. The Exchange attack showed complete disregard for possible consequences on behalf of those responsible for the breach.

Without consequences, such broad attacks will simply continue. There are currently no reasons why an attacker who has access to a zero-day shouldn�t simply press a button and exploit every possible target at the moment when they know their exploit is about to lose value. I don�t know how to change this calculus, but the U.S. must do so somehow.

If you have on-prem exchange and are the admin of it, get ready to rebuild your exchange servers.

This poo poo is going to keep happening, and it's going to get bigger and bigger and bigger.

orange juche fucked around with this message at 07:22 on Mar 12, 2021

# ? Mar 12, 2021 07:16

Entropist: Dec 1, 2007; I'm very stupid.

Ah, so that's why my organisation made Outlook available from their network/vpn only and is moving things to the cloud this week.

# ? Mar 12, 2021 12:21

BaseballPCHiker: Jan 16, 2006

Entropic posted:

Anyone have any experience with Cisco support? I was assigned to try to troubleshoot an issue with a client where the web interface on their router just spins when they try to login, and I went and created a Cisco support account and made a ticket, but I've been ghosted by the support rep since their first reply on Tuesday despite multiple followup attempts.

Does the device have support, as in is it current and someone pays for support for the device? Otherwise I think Cisco just does hardware support and nothing else. In any event how this goes down is you email TAC, they respond asking for a bunch of information you probably already included in your first email, they'll assign the ticket to someone, they'll ask you for all the same information again, you give it to them, then they ask for the output of a show tech, then they just tell you to do a code upgrade. At least thats been the case for me 90% of the time.

# ? Mar 12, 2021 14:45

Soylent Pudding: Jun 22, 2007; We've got people!

orange juche posted:

Btw if you have on prem exchange servers loving bend over champ, cause its time

https://www.lawfareblog.com/microsoft-exchange-hack-and-great-email-robbery

If you have on-prem exchange and are the admin of it, get ready to rebuild your exchange servers.

This poo poo is going to keep happening, and it's going to get bigger and bigger and bigger.

If you have on prem exchange you really should be assuming domain admin compromise at this point.

# ? Mar 12, 2021 15:02

orange juche: Mar 14, 2012

Soylent Pudding posted:

If you have on prem exchange you really should be assuming domain admin compromise at this point.

Yep, exchange service accounts are domain admins, throw the whole loving thing out.

# ? Mar 12, 2021 19:02

Sickening: Jul 16, 2007; Black summer was the best summer.

orange juche posted:

Btw if you have on prem exchange servers loving bend over champ, cause its time

https://www.lawfareblog.com/microsoft-exchange-hack-and-great-email-robbery

If you have on-prem exchange and are the admin of it, get ready to rebuild your exchange servers.

This poo poo is going to keep happening, and it's going to get bigger and bigger and bigger.

It makes for a very entertaining article, each time something like this happens doom and gloom are promised and it turns into :flaccid:

# ? Mar 12, 2021 19:23

RFC2324: Jun 7, 2012; http 418

Y2k was nothing but a wet fart

# ? Mar 12, 2021 19:43

Jewel: May 2, 2009

RFC2324 posted:

Y2k was nothing but a wet fart

bc people did a lot of work to make the impact minimal, though

e: or offset the two digit display year by 20 so we started to get the bugs again recently in a few cases

Jewel fucked around with this message at 20:19 on Mar 12, 2021

# ? Mar 12, 2021 20:16

orange juche: Mar 14, 2012

Sickening posted:

It makes for a very entertaining article, each time something like this happens doom and gloom are promised and it turns into .

Well i mean its going to continue to be :flaccid:

until

, and when that happens i hope i'm not working in IT when it does.

# ? Mar 12, 2021 20:26

rujasu: Dec 19, 2013

every time a new virus comes out, it's all doom and gloom, they told us West Nile and SARS were going to be these huge pandemics and look what happened, nothing

# ? Mar 12, 2021 20:55

RFC2324: Jun 7, 2012; http 418

Jewel posted:

bc people did a lot of work to make the impact minimal, though

thats my point. the doom and gloom scenarios assume people don't do the work to fix poo poo, which never ends up being the case. people like us work our asses off the make sure the apocalypse scenarios don't actually play out

# ? Mar 12, 2021 21:24

Raerlynn: Oct 28, 2007; Sorry I'm late, I'm afraid I got lost on the path of life.

RFC2324 posted:

thats my point. the doom and gloom scenarios assume people don't do the work to fix poo poo, which never ends up being the case. people like us work our asses off the make sure the apocalypse scenarios don't actually play out

This. I asked our Exchange admin about this today and he explained that the moment the breach went live they closed down external facing services and audited the entire the to high holy hell for the last few weeks.

It is a big deal, but we should also recognize the number of heroic efforts put in by IT teams around the world to mitigate this poo poo every time it happens.

# ? Mar 12, 2021 21:32

Internet Explorer: Jun 1, 2005

We patched our on-prem Exchange server pretty much instantly and just today got an alert from what looks like a secondary payload. So that's cool. With this having been in the wild for so long, I have a feeling the fallout from this one is going to be fairly extensive.

rujasu posted:

every time a new virus comes out, it's all doom and gloom, they told us West Nile and SARS were going to be these huge pandemics and look what happened, nothing

This is amazing, frame it.

# ? Mar 12, 2021 21:43

Potato Salad: Oct 23, 2014; nobody cares

Raerlynn posted:

This. I asked our Exchange admin about this today and he explained that the moment the breach went live they closed down external facing services and audited the entire the to high holy hell for the last few weeks.

It is a big deal, but we should also recognize the number of heroic efforts put in by IT teams around the world to mitigate this poo poo every time it happens.

I don't know, dude. I doubt that most orgs running on-prem mail of any variety have mature incident response /investigation capabilities.

# ? Mar 12, 2021 21:58

i am a moron: Nov 12, 2020; "I think if there’s one thing we can all agree on it’s that Penn State and Michigan both suck and are garbage and it’s hilarious Michigan fans are freaking out thinking this is their natty window when they can’t even beat a B12 team in the playoffs lmao"

They probably don�t even know they have OWA running

# ? Mar 12, 2021 22:01

RFC2324: Jun 7, 2012; http 418

i am a moron posted:

They probably don�t even know they have OWA running

I once heard a story about an IT org that discovered OWA was running when a C-level called bitching it was down

# ? Mar 12, 2021 22:36

Sickening: Jul 16, 2007; Black summer was the best summer.

RFC2324 posted:

I once heard a story about an IT org that discovered OWA was running when a C-level called bitching it was down

You can't ever turn it off right? All you can hope for is to make it inaccessible.

# ? Mar 12, 2021 22:48

orange juche: Mar 14, 2012

Sickening posted:

You can't ever turn it off right? All you can hope for is to make it inaccessible.

block external access to owa from your firewall and then redirect users searching for it internally in the intranet to hello.jpg

# ? Mar 12, 2021 22:52

i am a moron: Nov 12, 2020; "I think if there’s one thing we can all agree on it’s that Penn State and Michigan both suck and are garbage and it’s hilarious Michigan fans are freaking out thinking this is their natty window when they can’t even beat a B12 team in the playoffs lmao"

RFC2324 posted:

I once heard a story about an IT org that discovered OWA was running when a C-level called bitching it was down

All it takes is an unlabeled NAT and some brain drain to the right org. I�ve seen it more than once, and I�ve seen some permutation of �let�s shut OWA off no one uses it� only to get 5% of the company screaming bloody murder over it more times than I can count

# ? Mar 12, 2021 23:49

Thanks Ants: May 21, 2004; #essereFerrari

Has there been any comment on whether the five people running OWA behind Azure Application Proxy were vulnerable to this exploit?

# ? Mar 13, 2021 00:03

i am a moron: Nov 12, 2020; "I think if there’s one thing we can all agree on it’s that Penn State and Michigan both suck and are garbage and it’s hilarious Michigan fans are freaking out thinking this is their natty window when they can’t even beat a B12 team in the playoffs lmao"

You�d think that Azure AD auth in front would stop it

# ? Mar 13, 2021 00:19

dragonshardz: May 2, 2017

i am a moron posted:

You�d think that Azure AD auth in front would stop it

How would that stop a 0-day that makes the server think it's being contacted by itself?

# ? Mar 13, 2021 01:41

Wizard of the Deep: Sep 25, 2005; Another productive workday

dragonshardz posted:

How would that stop a 0-day that makes the server think it's being contacted by itself?

If it's in the context of Azure App Proxy, you have to auth to Azure AD before any outside signal gets to the server. No AAD auth, no packets to the server at all.

# ? Mar 13, 2021 02:13

Entropic: Feb 21, 2007; patriarchy sucks

https://twitter.com/emollick/status/1370547742313955329?s=20

# ? Mar 13, 2021 04:01

Internet Explorer: Jun 1, 2005

Maybe I'm old, but I remember having to fight to get OWA running. It was BES only for a long time and I literally had to cherry pick a champion and do some serious finagaling to get it into even a small pilot.

# ? Mar 13, 2021 04:10

Super Nintendo 64: Feb 18, 2012

Entropic posted:

https://twitter.com/emollick/status/1370547742313955329?s=20

Haha people and technology suck rear end (no offence)

# ? Mar 13, 2021 07:04

Thanks Ants: May 21, 2004; #essereFerrari

Internet Explorer posted:

Maybe I'm old, but I remember having to fight to get OWA running. It was BES only for a long time and I literally had to cherry pick a champion and do some serious finagaling to get it into even a small pilot.

I briefly worked at a company where the only way to access email outside the office in 2011 was via OWA or a PPTP VPN, on Exchange 2003. Bounced out fairly rapidly because you can't fix the underlying philosophy that leads to that being the policy. The IT director was one of those who would give everybody 200MB mailbox quotas and disable PSTs via GPO, and then flat out not listen to any problems that this caused.

# ? Mar 13, 2021 12:22

ihafarm: Aug 12, 2004

Entropic posted:

https://twitter.com/emollick/status/1370547742313955329?s=20

Exactly why I started our new physical inventory numbering scheme at 1000000 and not 0000000(Excel in general, not this specific incident). Not that I personally use Excel for this, but can�t anticipate what others do.

# ? Mar 13, 2021 18:26

Guy Axlerod: Dec 29, 2008

I like when excel outputs a csv where your numeric identifiers have been turned into scientific notation.

# ? Mar 13, 2021 21:27

Data Graham: Dec 28, 2009; 📈📊🍪😋

Guy Axlerod posted:

I like when excel outputs a csv where your numeric identifiers have been turned into scientific notation.

And you don't discover this until 5 or 6 transformations down the interdepartmental chain

e: yes we have a brand safety violation on campaign id 5.13E+10

# ? Mar 13, 2021 21:29

Thanks Ants: May 21, 2004; #essereFerrari

I can't understand why the default behaviour of Excel when opening a CSV file isn't to ask you what the data type in each column is. Maybe it could use some intelligence by looking at the header row or whatever to suggest some sane defaults.

# ? Mar 13, 2021 21:44

SlowBloke: Aug 14, 2017

Thanks Ants posted:

I can't understand why the default behaviour of Excel when opening a CSV file isn't to ask you what the data type in each column is. Maybe it could use some intelligence by looking at the header row or whatever to suggest some sane defaults.

If you open a csv when the separator is not comma, you will be have to do the conversion using the string to column tool which lets you to that

# ? Mar 13, 2021 21:47

Kurieg: Jul 19, 2012; RIP Lutri: 5/19/20-4/2/20

ihafarm posted:

Exactly why I started our new physical inventory numbering scheme at 1000000 and not 0000000(Excel in general, not this specific incident). Not that I personally use Excel for this, but can�t anticipate what others do.

One of our external systems starts at 000000000000, since we start ours with L, that's a text field for us and we maintain the value. However i'm pretty sure I'm the only person who actually puts in effort to maintain that value when doing csvs, or even just normal rear end excel spreadsheets, as I constantly get emails asking me why 124650 isn't in the system.

# ? Mar 13, 2021 22:14

Bob Morales: Aug 18, 2006; ~~Just wear the fucking mask, Bob~~

I don't care how many people I probably infected with COVID-19 while refusing to wear a mask, my comfort is far more important than the health and safety of everyone around me!

Excel loved our part numbers that had decimals in them

# ? Mar 13, 2021 22:50

RFC2324: Jun 7, 2012; http 418

SlowBloke posted:

If you open a csv when the separator is not comma, you will be have to do the conversion using the string to column tool which lets you to that

I've always used semicolons when given the option, and always wondered why excel gave me that option but didn't seem to give it to anyone else

# ? Mar 14, 2021 00:13

Super Nintendo 64: Feb 18, 2012

I met Excel once when I was working at Microsoft and he was a total rear end in a top hat.

# ? Mar 14, 2021 02:29

Renegret: May 26, 2007; THANK YOU FOR CALLING HELP DOG, INC.

YOUR POSITION IN THE QUEUE IS *pbbbbbbbbbbbbbbbbt*

Cat Army Sworn Enemy

Super Nintendo 64 posted:

I met Excel once when I was working at Microsoft and he was a total rear end in a top hat.

Word

# ? Mar 14, 2021 02:34

kensei: Dec 27, 2007; He has come home, where he belongs. The Ancient Mariner returns to lead his first team to glory, forever and ever. Amen!

Super Nintendo 64 posted:

I met Excel once when I was working at Microsoft and he was a total rear end in a top hat.

He had a lovely Outlook when I spoke to him

# ? Mar 14, 2021 02:56

Adbot: ADBOT LOVES YOU

# ? May 26, 2024 00:03

SyNack Sassimov: May 4, 2006; Let the robot win.
��--Captain James T. Vader

kensei posted:

He had a lovely Outlook when I spoke to him

Do you think he has good days too, or Visiolways that cranky?

# ? Mar 14, 2021 03:11

The Something Awful Forums > Archives > Comedy Goldmine > The Goodmine > [SPAM] FW: RE: The logs could do with a courtesy flush

«‹›501 »