Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
how long does trump go to jail for?
life
no jail time
elected president from a jail cell
goku
View Results
 
  • Post
  • Reply
emfive
Aug 6, 2011

Hey emfive, this is Alec. I am glad you like the mummy eating the bowl of shitty pasta with a can of 'parm.' I made that image for you way back when. I’m glad you enjoy it.

god will smite this down





Adbot
ADBOT LOVES YOU

smarxist
Jul 26, 2018

by Fluffdaddy

because i'm a weirdo i spotted the AV nerd (dechristefaro) and googled, and he has a youtube channel where he was making wacky little TV show skit poo poo, and uploaded this exact outro lmfao

Shifty Nipples
Apr 8, 2007



hosed up if you ask me

El Palo Mota
Aug 5, 2009

Plinkey posted:

im gunna say it, and it's not gunna be popular

cool ranch seasoning tastes better on thin lays chips than it does on Doritos corn chips

triple sulk
Sep 17, 2014



torn achilles lol couldn't happen to a better guy

Grey Fox
Jan 5, 2004

doritos have a cardboard texture so i bet all of those seasonings on other chips would be an improvement

Shifty Nipples
Apr 8, 2007

crunchy cardboard

tacodaemon
Nov 27, 2006



slow motion video of crashing HO scale trains into die-cast metal toy cars


https://www.youtube.com/watch?v=WrMsvYD4t4I

smarxist
Jul 26, 2018

by Fluffdaddy

Grey Fox posted:

doritos have a cardboard texture so i bet all of those seasonings on other chips would be an improvement

they made a pivot like 5ish years ago to be more like thin/crisp restaurant style chips, and it was a huge mistake imo

alnilam
Nov 10, 2009

Plinkey posted:

yeah, we had a kentacohut somewhere in Oakland around pitt

That kentacohut closed in 2005 :911:

Spoondick
Jun 9, 2000

Plinkey posted:

you can do better



sorry, but im afraid this isn't working out, you're not taco bell material because you lack the live mas mentality... maybe the kfc 10 feet to the left is more your speed

FunOne
Aug 20, 2000
I am a slimey vat of concentrated stupidity

Fun Shoe

Plinkey posted:

you can do better



Be still my heart.

RandomBlue
Dec 30, 2012

hay guys!


Biscuit Hider

tacodaemon posted:

slow motion video of crashing HO scale trains into die-cast metal toy cars


https://www.youtube.com/watch?v=WrMsvYD4t4I

jesus, just how big were hos back then?

smarxist
Jul 26, 2018

by Fluffdaddy

Plinkey posted:

you can do better



doritos locos taco stuffed with popcorn chicken and dunked in marinara sauce

SomethingBeautiful
Oct 22, 2008

Some celestial event. No- no words. No words to describe it. Poetry! They should have sent a poet. So beautiful. So beautiful... I had no idea

tacodaemon posted:

slow motion video of crashing HO scale trains into die-cast metal toy cars


https://www.youtube.com/watch?v=WrMsvYD4t4I

should use those old lionel trains those things are heavy

Plinkey
Aug 4, 2004

by Fluffdaddy

alnilam posted:

That kentacohut closed in 2005 :911:

i was only there for like fall of 2002, loving hated that place

transfered to penn state instead

Spergin Morlock
Aug 8, 2009
Probation
Can't post for 39 minutes!



YOSPOS

limp dick calvin
Sep 1, 2006

Strepitoso. Vedete? Una meraviglia.
bill cassidy fucks

titty_baby_
Nov 11, 2015


Melting pot

Shifty Nipples
Apr 8, 2007

titty_baby_ posted:

Melting pot

limp dick calvin
Sep 1, 2006

Strepitoso. Vedete? Una meraviglia.

titty_baby_ posted:

Melting pot

you’re supposed to smoke it

Grey Fox
Jan 5, 2004

street tacos (registered trademark)

Fuckt Tupp
Apr 19, 2007

Science

Plinkey posted:

lol what a typo



obama bin biden

Fuckt Tupp
Apr 19, 2007

Science

MY BRANDS :negative:

emfive
Aug 6, 2011

Hey emfive, this is Alec. I am glad you like the mummy eating the bowl of shitty pasta with a can of 'parm.' I made that image for you way back when. I’m glad you enjoy it.

is he related to David Cassidy

he was Shaun Cassidy's half-brother and star of the blockbuster TV hit "The Partridge Family" with his stepmom Shirley Jones, noted for her starring role in the musical "Oklahoma!"

Father Jack Cassidy won a Tony award for his work in the musical "She Loves Me", and was in three Columbo episodes!

smarxist
Jul 26, 2018

by Fluffdaddy

emfive posted:

is he related to David Cassidy

he was Shaun Cassidy's half-brother and star of the blockbuster TV hit "The Partridge Family" with his stepmom Shirley Jones, noted for her starring role in the musical "Oklahoma!"

Father Jack Cassidy won a Tony award for his work in the musical "She Loves Me", and was in three Columbo episodes!

did you Rainman the TV Guide or something

Inspector Hound
Jul 14, 2003

Schmoe Cwead posted:

great ai takes over every other job but athletes are still getting tbi's while everyone just sees birds and stars spinning over a dizzy baby yoda

On the other hand, why can't we eventually have AI rollerball too, snuffing out all other sports overnight

Plinkey
Aug 4, 2004

by Fluffdaddy

Shifty Nipples posted:

crunchy cardboard

why do you keep talking about your dick like that?

tacodaemon
Nov 27, 2006



this tale of how Elon shut down the Sacramento data center is one for all the sys admins and data center managers out there


https://www.cnbc.com/amp/2023/09/11...of-urgency.html

"Does this timeframe seem like something that I would find remotely acceptable?" Musk asked. "Obviously not. If a timeline is long, it's wrong."

It was late at night on December 22, and the meeting in Musk's 10th floor conference room at X, formerly Twitter, had become tense. He was talking to two X infrastructure managers who had not dealt with him much before, and certainly not when he was in a foul mood.

One of them tried to explain the problem. The data-services company that housed one of X's server farms, located in Sacramento, had agreed to allow them some short-term extensions on their lease so they could begin to move out during 2023 in an orderly fashion. "But this morning," the nervous manager told Musk, "they came back to us and said that plan was no longer on the table because, and these are their words, they don't think that we will be financially viable."

The facility was costing X more than $100 million a year. Musk wanted to save that money by moving the servers to one of X's other facilities, in Portland, Oregon. Another manager at the meeting said that couldn't be done right away. "We can't get out safely before six to nine months," she said in a matter-of-fact tone. "Sacramento still needs to be around to serve traffic."

Over the years, Musk had been faced many times with a choice between what he thought was necessary and what others told him was possible. The result was almost always the same. He paused in silence for a few moments, then announced, "You have 90 days to do it. If you can't make that work, your resignation is accepted."

The manager began to explain in detail some of the obstacles to relocating the servers to Portland. "It has different rack densities, different power densities," she said. "So the rooms need to be upgraded." She started to give a lot more details, but after a minute, Musk interrupted.

"This is making my brain hurt," he said.


"I'm sorry, that was not my intention," she replied in a measured monotone.

"Do you know the head-explosion emoji?" he asked her. "That's what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It's trivial to move servers one place to another."

The X managers again tried to explain the constraints. Musk interrupted. "Can you have someone go to our server centers and send me videos of the insides?" he asked.

It was three days before Christmas, and the manager promised the video in a week. "No, tomorrow," Musk ordered. "I've built server centers myself, and I can tell if you could put more servers there or not. That's why I asked if you had actually visited these facilities. If you've not been there, you're just talking bulls---."

SpaceX and Tesla were successful because Musk relentlessly pushed his teams to be scrappier, more nimble, and to launch fire-drill surges that extruded all obstacles. That's how they had cobbled together a car production line in a tent in Fremont and a test facility in the Texas desert and a launch site at Cape Canaveral made of used parts.

"All you need to do is just move the f---ing servers to Portland," he said. "If it takes longer than 30 days, that would blow my mind." He paused and recalculated. "Just get a moving company, and it will take a week to move the computers and another week to plug them in. Two weeks. That's what should happen."

Everyone was silent. But Musk was still warming up. "If you got a godd--- U-Haul, you could probably do it by yourself." The two X managers looked to see if he was serious. Two of Musk's top loyalists, Steve Davis and Omead Afshar were also at the table. They had seen him like this many times before, and they knew that he might be.

"Why don't we do it right now?" James Musk asked.

James and his brother Andrew, younger first cousins of Musk, were flying with him from San Francisco to Austin on Friday evening, December 23, the day after the frustrating infrastructure meeting about how long it would take to move the servers out of the Sacramento facility.
Avid skiers, they had planned to go by themselves to Tahoe for Christmas, but Elon that day invited them to come to Austin instead.

James was reluctant. He was mentally exhausted and didn't need more intensity, but Andrew convinced him that they should go. So that's how they ended up on the plane listening to Elon complain about the servers.

They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved. It was already late evening, but he told his pilot to divert, and they made a loop back up to Sacramento.

The only rental car they could find when they landed was a Toyota Corolla. They were not sure how they would even get inside the data center at night, but one very surprised X staffer, a guy named Alex from Uzbekistan, was still there. He merrily let them in and showed them around.

The facility, which housed rooms of servers for many other companies as well, was very secure, with a retinal scan required for entry into each of the vaults. Alex the Uzbek was able to get them into the X vault, which contained about 5,200 refrigerator-size racks of 30 computers each.

"These things do not look that hard to move," Elon announced. It was a reality-distorting assertion, since each rack weighed about 2,500 pounds and was eight feet tall.

"You'll have to hire a contractor to lift the floor panels," Alex said. "They need to be lifted with suction cups." Another set of contractors, he said, would then have to go underneath the floor panels and disconnect the electric cables and seismic rods.

Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved.

"Well that doesn't seem super hard," he said as Alex the Uzbek and the rest of the gang stared. Musk was totally jazzed by this point. It was, he said with a loud laugh, like a remake of Mission: Impossible, Sacramento edition.


The next day — Christmas Eve — Musk called in reinforcements. Ross Nordeen, who worked with his friend James at Tesla, drove from San Francisco. He stopped at the Apple Store in Union Square and spent $2,000 to buy out the entire stock of AirTags so the servers could be tracked on their journey, and then stopped at Home Depot, where he spent $2,500 on wrenches, bolt-cutters, headlamps, and the tools needed to unscrew the seismic bolts.

Steve Davis, a loyal Musk lieutenant, got someone to procure a semi truck and line up moving vans. Other enlistees arrived from SpaceX. The server racks were on wheels, so the team was able to disconnect four of them and roll them to the waiting truck. This showed that all fifty-two hundred or so could probably be moved within days. "The guys are kicking rear end!" Musk exulted.

Other workers at the facility watched with a mix of amazement and horror. Musk and his renegade team were rolling servers out without putting them in crates or swaddling them in protective material, then using store-bought straps to secure them in the truck. "I've never loaded a semi before," James admitted. Ross called it "terrifying." It was like cleaning out a closet, "but the stuff in it is totally critical."

At 3 p.m., after they had gotten four servers onto the truck, word of the caper reached the top executives at NTT, the company that owned and managed the data center. They issued orders that Musk's team halt. Musk had the mix of glee and anger that often accompanied one of his manic surges. He called the CEO of the storage division, who told him it was impossible to move server racks without a bevy of experts. "Bulls---," Musk explained. "We have already loaded four onto the semi."

The CEO then told him that some of the floors could not handle more than 500 pounds of pressure, so rolling a 2,000-pound server would cause damage. Musk replied that the servers had four wheels, so the pressure at any one point was only 500 pounds. "The dude is not very good at math," Musk told the musketeers.

Having ruined the Christmas Eve of the NTT managers, as well as hitting them with a potential loss of more than $100 million in revenue for the coming year, Musk showed pity and said he would suspend moving the servers for two days. But they would resume, he warned, the day after Christmas.

After Christmas, Andrew and James headed back to Sacramento to see how many more servers they could move. They hadn't brought enough clothes, so they went to Walmart and bought jeans and T-shirts.

The moving contractors that NTT wanted them to use charged $200 an hour. So James went on Yelp and found a company named Extra Care Movers that would do the work at one-tenth the cost. The motley company pushed the ideal of scrappiness to its outer limits. The owner had lived on the streets for a while, then had a kid, and he was trying to turn his life around. He didn't have a bank account, so James ended up using PayPal to pay him.

The second day, the crew wanted cash, so James went to a bank and withdrew $13,000 from his personal account. Two of the crew members had no identification, which made it hard for them to sign into the facility. But they made up for it in hustle. "You get a dollar tip for every additional server we move," James announced at one point. From then on, when they got a new one on a truck, the workers would ask how many they were up to.

The servers had user data on them, and James did not initially realize that, for privacy reasons, they were supposed to be wiped clean before being moved. "By the time we learned this, the servers had already been unplugged and rolled out, so there was no way we would roll them back, plug them in, and then wipe them," he says. Plus, the wiping software wasn't working. "F---, what do we do?" he asked. Elon recommended that they lock the trucks and track them.

So James sent someone to Home Depot to buy big padlocks, and they sent the combination codes on a spreadsheet to Portland so the trucks could be opened there.
"I can't believe it worked," James says. "They all made it to Portland safely."

By the end of the week, they had used all of the available trucks in Sacramento. Despite the area being pummeled by rain, they moved more than 700 of the racks in three days. The previous record at that facility had been moving 30 in a month. That still left a lot of servers in the facility, but the musketeers had proven that they could be moved quickly. The rest were handled by the X infrastructure team in January.

All very exciting and inspiring, right? An example of Musk's bold and scrappy approach! But as with all things Musk, it was, alas, not that simple. It was also an example of his recklessness, his impatience with pushback, and the way he intimidated people. X's infrastructure engineers had tried to explain to him, in that head-explosion-emoji meeting a week earlier, why a quick shutdown of the Sacramento center would be a problem, but he shot them down. He had a good track record of knowing when to ignore naysayers. But not a perfect one.

For the next two months, X was destabilized. The lack of servers caused meltdowns, including when Musk hosted a Twitter Spaces for presidential candidate Ron DeSantis. "In retrospect, the whole Sacramento shutdown was a mistake," Musk would admit in March 2023. "I was told we had redundancy across our data centers. What I wasn't told was that we had 70,000 hard-coded references to Sacramento. And there's still poo poo that's broken because of it."

His most valuable lieutenants at Tesla and SpaceX had learned ways to deflect his bad ideas and drip-feed him unwelcome information, but the legacy employees at X didn't know how to handle him. That said, X survived. And the Sacramento caper showed X employees that he was serious when he spoke about the need for a maniacal sense of urgency.

tacodaemon has issued a correction as of 05:26 on Sep 12, 2023

WrasslorMonkey
Mar 5, 2012


ABU (always be uppercutting)

limp dick calvin
Sep 1, 2006

Strepitoso. Vedete? Una meraviglia.
knowing elon was in my hometown is causing me psychic pain

Bilirubin
Feb 16, 2014

The sanctioned action is to CHUG


tacodaemon posted:

slow motion video of crashing HO scale trains into die-cast metal toy cars


https://www.youtube.com/watch?v=WrMsvYD4t4I

I had that engine

Bilirubin
Feb 16, 2014

The sanctioned action is to CHUG


FunOne posted:

Be still my heart.

Just keep eating there then

19 o'clock
Sep 9, 2004

Excelsior!!!

Bilirubin
Feb 16, 2014

The sanctioned action is to CHUG


tacodaemon posted:

this tale of how Elon shut down the Sacramento data center is one for all the sys admins and data center managers out there


https://www.cnbc.com/amp/2023/09/11...of-urgency.html

"Does this timeframe seem like something that I would find remotely acceptable?" Musk asked. "Obviously not. If a timeline is long, it's wrong."

It was late at night on December 22, and the meeting in Musk's 10th floor conference room at X, formerly Twitter, had become tense. He was talking to two X infrastructure managers who had not dealt with him much before, and certainly not when he was in a foul mood.

One of them tried to explain the problem. The data-services company that housed one of X's server farms, located in Sacramento, had agreed to allow them some short-term extensions on their lease so they could begin to move out during 2023 in an orderly fashion. "But this morning," the nervous manager told Musk, "they came back to us and said that plan was no longer on the table because, and these are their words, they don't think that we will be financially viable."

The facility was costing X more than $100 million a year. Musk wanted to save that money by moving the servers to one of X's other facilities, in Portland, Oregon. Another manager at the meeting said that couldn't be done right away. "We can't get out safely before six to nine months," she said in a matter-of-fact tone. "Sacramento still needs to be around to serve traffic."

Over the years, Musk had been faced many times with a choice between what he thought was necessary and what others told him was possible. The result was almost always the same. He paused in silence for a few moments, then announced, "You have 90 days to do it. If you can't make that work, your resignation is accepted."

The manager began to explain in detail some of the obstacles to relocating the servers to Portland. "It has different rack densities, different power densities," she said. "So the rooms need to be upgraded." She started to give a lot more details, but after a minute, Musk interrupted.

"This is making my brain hurt," he said.


"I'm sorry, that was not my intention," she replied in a measured monotone.

"Do you know the head-explosion emoji?" he asked her. "That's what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It's trivial to move servers one place to another."

The X managers again tried to explain the constraints. Musk interrupted. "Can you have someone go to our server centers and send me videos of the insides?" he asked.

It was three days before Christmas, and the manager promised the video in a week. "No, tomorrow," Musk ordered. "I've built server centers myself, and I can tell if you could put more servers there or not. That's why I asked if you had actually visited these facilities. If you've not been there, you're just talking bulls---."

SpaceX and Tesla were successful because Musk relentlessly pushed his teams to be scrappier, more nimble, and to launch fire-drill surges that extruded all obstacles. That's how they had cobbled together a car production line in a tent in Fremont and a test facility in the Texas desert and a launch site at Cape Canaveral made of used parts.

"All you need to do is just move the f---ing servers to Portland," he said. "If it takes longer than 30 days, that would blow my mind." He paused and recalculated. "Just get a moving company, and it will take a week to move the computers and another week to plug them in. Two weeks. That's what should happen."

Everyone was silent. But Musk was still warming up. "If you got a godd--- U-Haul, you could probably do it by yourself." The two X managers looked to see if he was serious. Two of Musk's top loyalists, Steve Davis and Omead Afshar were also at the table. They had seen him like this many times before, and they knew that he might be.

"Why don't we do it right now?" James Musk asked.

James and his brother Andrew, younger first cousins of Musk, were flying with him from San Francisco to Austin on Friday evening, December 23, the day after the frustrating infrastructure meeting about how long it would take to move the servers out of the Sacramento facility.
Avid skiers, they had planned to go by themselves to Tahoe for Christmas, but Elon that day invited them to come to Austin instead.

James was reluctant. He was mentally exhausted and didn't need more intensity, but Andrew convinced him that they should go. So that's how they ended up on the plane listening to Elon complain about the servers.

They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved. It was already late evening, but he told his pilot to divert, and they made a loop back up to Sacramento.

The only rental car they could find when they landed was a Toyota Corolla. They were not sure how they would even get inside the data center at night, but one very surprised X staffer, a guy named Alex from Uzbekistan, was still there. He merrily let them in and showed them around.

The facility, which housed rooms of servers for many other companies as well, was very secure, with a retinal scan required for entry into each of the vaults. Alex the Uzbek was able to get them into the X vault, which contained about 5,200 refrigerator-size racks of 30 computers each.

"These things do not look that hard to move," Elon announced. It was a reality-distorting assertion, since each rack weighed about 2,500 pounds and was eight feet tall.

"You'll have to hire a contractor to lift the floor panels," Alex said. "They need to be lifted with suction cups." Another set of contractors, he said, would then have to go underneath the floor panels and disconnect the electric cables and seismic rods.

Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved.

"Well that doesn't seem super hard," he said as Alex the Uzbek and the rest of the gang stared. Musk was totally jazzed by this point. It was, he said with a loud laugh, like a remake of Mission: Impossible, Sacramento edition.


The next day — Christmas Eve — Musk called in reinforcements. Ross Nordeen, who worked with his friend James at Tesla, drove from San Francisco. He stopped at the Apple Store in Union Square and spent $2,000 to buy out the entire stock of AirTags so the servers could be tracked on their journey, and then stopped at Home Depot, where he spent $2,500 on wrenches, bolt-cutters, headlamps, and the tools needed to unscrew the seismic bolts.

Steve Davis, a loyal Musk lieutenant, got someone to procure a semi truck and line up moving vans. Other enlistees arrived from SpaceX. The server racks were on wheels, so the team was able to disconnect four of them and roll them to the waiting truck. This showed that all fifty-two hundred or so could probably be moved within days. "The guys are kicking rear end!" Musk exulted.

Other workers at the facility watched with a mix of amazement and horror. Musk and his renegade team were rolling servers out without putting them in crates or swaddling them in protective material, then using store-bought straps to secure them in the truck. "I've never loaded a semi before," James admitted. Ross called it "terrifying." It was like cleaning out a closet, "but the stuff in it is totally critical."

At 3 p.m., after they had gotten four servers onto the truck, word of the caper reached the top executives at NTT, the company that owned and managed the data center. They issued orders that Musk's team halt. Musk had the mix of glee and anger that often accompanied one of his manic surges. He called the CEO of the storage division, who told him it was impossible to move server racks without a bevy of experts. "Bulls---," Musk explained. "We have already loaded four onto the semi."

The CEO then told him that some of the floors could not handle more than 500 pounds of pressure, so rolling a 2,000-pound server would cause damage. Musk replied that the servers had four wheels, so the pressure at any one point was only 500 pounds. "The dude is not very good at math," Musk told the musketeers.

Having ruined the Christmas Eve of the NTT managers, as well as hitting them with a potential loss of more than $100 million in revenue for the coming year, Musk showed pity and said he would suspend moving the servers for two days. But they would resume, he warned, the day after Christmas.

After Christmas, Andrew and James headed back to Sacramento to see how many more servers they could move. They hadn't brought enough clothes, so they went to Walmart and bought jeans and T-shirts.

The moving contractors that NTT wanted them to use charged $200 an hour. So James went on Yelp and found a company named Extra Care Movers that would do the work at one-tenth the cost. The motley company pushed the ideal of scrappiness to its outer limits. The owner had lived on the streets for a while, then had a kid, and he was trying to turn his life around. He didn't have a bank account, so James ended up using PayPal to pay him.

The second day, the crew wanted cash, so James went to a bank and withdrew $13,000 from his personal account. Two of the crew members had no identification, which made it hard for them to sign into the facility. But they made up for it in hustle. "You get a dollar tip for every additional server we move," James announced at one point. From then on, when they got a new one on a truck, the workers would ask how many they were up to.

The servers had user data on them, and James did not initially realize that, for privacy reasons, they were supposed to be wiped clean before being moved. "By the time we learned this, the servers had already been unplugged and rolled out, so there was no way we would roll them back, plug them in, and then wipe them," he says. Plus, the wiping software wasn't working. "F---, what do we do?" he asked. Elon recommended that they lock the trucks and track them.

So James sent someone to Home Depot to buy big padlocks, and they sent the combination codes on a spreadsheet to Portland so the trucks could be opened there.
"I can't believe it worked," James says. "They all made it to Portland safely."

By the end of the week, they had used all of the available trucks in Sacramento. Despite the area being pummeled by rain, they moved more than 700 of the racks in three days. The previous record at that facility had been moving 30 in a month. That still left a lot of servers in the facility, but the musketeers had proven that they could be moved quickly. The rest were handled by the X infrastructure team in January.

All very exciting and inspiring, right? An example of Musk's bold and scrappy approach! But as with all things Musk, it was, alas, not that simple. It was also an example of his recklessness, his impatience with pushback, and the way he intimidated people. X's infrastructure engineers had tried to explain to him, in that head-explosion-emoji meeting a week earlier, why a quick shutdown of the Sacramento center would be a problem, but he shot them down. He had a good track record of knowing when to ignore naysayers. But not a perfect one.

For the next two months, X was destabilized. The lack of servers caused meltdowns, including when Musk hosted a Twitter Spaces for presidential candidate Ron DeSantis. "In retrospect, the whole Sacramento shutdown was a mistake," Musk would admit in March 2023. "I was told we had redundancy across our data centers. What I wasn't told was that we had 70,000 hard-coded references to Sacramento. And there's still poo poo that's broken because of it."

His most valuable lieutenants at Tesla and SpaceX had learned ways to deflect his bad ideas and drip-feed him unwelcome information, but the legacy employees at X didn't know how to handle him. That said, X survived. And the Sacramento caper showed X employees that he was serious when he spoke about the need for a maniacal sense of urgency.
Congrats and/or condolences

Filthy Hans
Jun 27, 2008

by Fluffdaddy

(and can't post for 10 years!)


:owned:

Plinkey
Aug 4, 2004

by Fluffdaddy

Bilirubin posted:

I had that engine

me too

WrasslorMonkey
Mar 5, 2012

tacodaemon posted:

this tale of how Elon shut down the Sacramento data center is one for all the sys admins and data center managers out there


https://www.cnbc.com/amp/2023/09/11...of-urgency.html

"Does this timeframe seem like something that I would find remotely acceptable?" Musk asked. "Obviously not. If a timeline is long, it's wrong."

It was late at night on December 22, and the meeting in Musk's 10th floor conference room at X, formerly Twitter, had become tense. He was talking to two X infrastructure managers who had not dealt with him much before, and certainly not when he was in a foul mood.

One of them tried to explain the problem. The data-services company that housed one of X's server farms, located in Sacramento, had agreed to allow them some short-term extensions on their lease so they could begin to move out during 2023 in an orderly fashion. "But this morning," the nervous manager told Musk, "they came back to us and said that plan was no longer on the table because, and these are their words, they don't think that we will be financially viable."

The facility was costing X more than $100 million a year. Musk wanted to save that money by moving the servers to one of X's other facilities, in Portland, Oregon. Another manager at the meeting said that couldn't be done right away. "We can't get out safely before six to nine months," she said in a matter-of-fact tone. "Sacramento still needs to be around to serve traffic."

Over the years, Musk had been faced many times with a choice between what he thought was necessary and what others told him was possible. The result was almost always the same. He paused in silence for a few moments, then announced, "You have 90 days to do it. If you can't make that work, your resignation is accepted."

The manager began to explain in detail some of the obstacles to relocating the servers to Portland. "It has different rack densities, different power densities," she said. "So the rooms need to be upgraded." She started to give a lot more details, but after a minute, Musk interrupted.

"This is making my brain hurt," he said.


"I'm sorry, that was not my intention," she replied in a measured monotone.

"Do you know the head-explosion emoji?" he asked her. "That's what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It's trivial to move servers one place to another."

The X managers again tried to explain the constraints. Musk interrupted. "Can you have someone go to our server centers and send me videos of the insides?" he asked.

It was three days before Christmas, and the manager promised the video in a week. "No, tomorrow," Musk ordered. "I've built server centers myself, and I can tell if you could put more servers there or not. That's why I asked if you had actually visited these facilities. If you've not been there, you're just talking bulls---."

SpaceX and Tesla were successful because Musk relentlessly pushed his teams to be scrappier, more nimble, and to launch fire-drill surges that extruded all obstacles. That's how they had cobbled together a car production line in a tent in Fremont and a test facility in the Texas desert and a launch site at Cape Canaveral made of used parts.

"All you need to do is just move the f---ing servers to Portland," he said. "If it takes longer than 30 days, that would blow my mind." He paused and recalculated. "Just get a moving company, and it will take a week to move the computers and another week to plug them in. Two weeks. That's what should happen."

Everyone was silent. But Musk was still warming up. "If you got a godd--- U-Haul, you could probably do it by yourself." The two X managers looked to see if he was serious. Two of Musk's top loyalists, Steve Davis and Omead Afshar were also at the table. They had seen him like this many times before, and they knew that he might be.

"Why don't we do it right now?" James Musk asked.

James and his brother Andrew, younger first cousins of Musk, were flying with him from San Francisco to Austin on Friday evening, December 23, the day after the frustrating infrastructure meeting about how long it would take to move the servers out of the Sacramento facility.
Avid skiers, they had planned to go by themselves to Tahoe for Christmas, but Elon that day invited them to come to Austin instead.

James was reluctant. He was mentally exhausted and didn't need more intensity, but Andrew convinced him that they should go. So that's how they ended up on the plane listening to Elon complain about the servers.

They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved. It was already late evening, but he told his pilot to divert, and they made a loop back up to Sacramento.

The only rental car they could find when they landed was a Toyota Corolla. They were not sure how they would even get inside the data center at night, but one very surprised X staffer, a guy named Alex from Uzbekistan, was still there. He merrily let them in and showed them around.

The facility, which housed rooms of servers for many other companies as well, was very secure, with a retinal scan required for entry into each of the vaults. Alex the Uzbek was able to get them into the X vault, which contained about 5,200 refrigerator-size racks of 30 computers each.

"These things do not look that hard to move," Elon announced. It was a reality-distorting assertion, since each rack weighed about 2,500 pounds and was eight feet tall.

"You'll have to hire a contractor to lift the floor panels," Alex said. "They need to be lifted with suction cups." Another set of contractors, he said, would then have to go underneath the floor panels and disconnect the electric cables and seismic rods.

Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved.

"Well that doesn't seem super hard," he said as Alex the Uzbek and the rest of the gang stared. Musk was totally jazzed by this point. It was, he said with a loud laugh, like a remake of Mission: Impossible, Sacramento edition.


The next day — Christmas Eve — Musk called in reinforcements. Ross Nordeen, who worked with his friend James at Tesla, drove from San Francisco. He stopped at the Apple Store in Union Square and spent $2,000 to buy out the entire stock of AirTags so the servers could be tracked on their journey, and then stopped at Home Depot, where he spent $2,500 on wrenches, bolt-cutters, headlamps, and the tools needed to unscrew the seismic bolts.

Steve Davis, a loyal Musk lieutenant, got someone to procure a semi truck and line up moving vans. Other enlistees arrived from SpaceX. The server racks were on wheels, so the team was able to disconnect four of them and roll them to the waiting truck. This showed that all fifty-two hundred or so could probably be moved within days. "The guys are kicking rear end!" Musk exulted.

Other workers at the facility watched with a mix of amazement and horror. Musk and his renegade team were rolling servers out without putting them in crates or swaddling them in protective material, then using store-bought straps to secure them in the truck. "I've never loaded a semi before," James admitted. Ross called it "terrifying." It was like cleaning out a closet, "but the stuff in it is totally critical."

At 3 p.m., after they had gotten four servers onto the truck, word of the caper reached the top executives at NTT, the company that owned and managed the data center. They issued orders that Musk's team halt. Musk had the mix of glee and anger that often accompanied one of his manic surges. He called the CEO of the storage division, who told him it was impossible to move server racks without a bevy of experts. "Bulls---," Musk explained. "We have already loaded four onto the semi."

The CEO then told him that some of the floors could not handle more than 500 pounds of pressure, so rolling a 2,000-pound server would cause damage. Musk replied that the servers had four wheels, so the pressure at any one point was only 500 pounds. "The dude is not very good at math," Musk told the musketeers.

Having ruined the Christmas Eve of the NTT managers, as well as hitting them with a potential loss of more than $100 million in revenue for the coming year, Musk showed pity and said he would suspend moving the servers for two days. But they would resume, he warned, the day after Christmas.

After Christmas, Andrew and James headed back to Sacramento to see how many more servers they could move. They hadn't brought enough clothes, so they went to Walmart and bought jeans and T-shirts.

The moving contractors that NTT wanted them to use charged $200 an hour. So James went on Yelp and found a company named Extra Care Movers that would do the work at one-tenth the cost. The motley company pushed the ideal of scrappiness to its outer limits. The owner had lived on the streets for a while, then had a kid, and he was trying to turn his life around. He didn't have a bank account, so James ended up using PayPal to pay him.

The second day, the crew wanted cash, so James went to a bank and withdrew $13,000 from his personal account. Two of the crew members had no identification, which made it hard for them to sign into the facility. But they made up for it in hustle. "You get a dollar tip for every additional server we move," James announced at one point. From then on, when they got a new one on a truck, the workers would ask how many they were up to.

The servers had user data on them, and James did not initially realize that, for privacy reasons, they were supposed to be wiped clean before being moved. "By the time we learned this, the servers had already been unplugged and rolled out, so there was no way we would roll them back, plug them in, and then wipe them," he says. Plus, the wiping software wasn't working. "F---, what do we do?" he asked. Elon recommended that they lock the trucks and track them.

So James sent someone to Home Depot to buy big padlocks, and they sent the combination codes on a spreadsheet to Portland so the trucks could be opened there.
"I can't believe it worked," James says. "They all made it to Portland safely."

By the end of the week, they had used all of the available trucks in Sacramento. Despite the area being pummeled by rain, they moved more than 700 of the racks in three days. The previous record at that facility had been moving 30 in a month. That still left a lot of servers in the facility, but the musketeers had proven that they could be moved quickly. The rest were handled by the X infrastructure team in January.

All very exciting and inspiring, right? An example of Musk's bold and scrappy approach! But as with all things Musk, it was, alas, not that simple. It was also an example of his recklessness, his impatience with pushback, and the way he intimidated people. X's infrastructure engineers had tried to explain to him, in that head-explosion-emoji meeting a week earlier, why a quick shutdown of the Sacramento center would be a problem, but he shot them down. He had a good track record of knowing when to ignore naysayers. But not a perfect one.

For the next two months, X was destabilized. The lack of servers caused meltdowns, including when Musk hosted a Twitter Spaces for presidential candidate Ron DeSantis. "In retrospect, the whole Sacramento shutdown was a mistake," Musk would admit in March 2023. "I was told we had redundancy across our data centers. What I wasn't told was that we had 70,000 hard-coded references to Sacramento. And there's still poo poo that's broken because of it."

His most valuable lieutenants at Tesla and SpaceX had learned ways to deflect his bad ideas and drip-feed him unwelcome information, but the legacy employees at X didn't know how to handle him. That said, X survived. And the Sacramento caper showed X employees that he was serious when he spoke about the need for a maniacal sense of urgency.

Thread title

Plinkey
Aug 4, 2004

by Fluffdaddy

Bilirubin posted:

Congrats and/or condolences

it's a pretty good read if you know how to

Adbot
ADBOT LOVES YOU

Fuckt Tupp
Apr 19, 2007

Science
after reading the server story imagine it was instead about implanting computer chips into monkey brains

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply