- Ragequit
- Jun 1, 2006
-
-
Lipstick Apathy
|
this tale of how Elon shut down the Sacramento data center is one for all the sys admins and data center managers out there
https://www.cnbc.com/amp/2023/09/11...of-urgency.html
"Does this timeframe seem like something that I would find remotely acceptable?" Musk asked. "Obviously not. If a timeline is long, it's wrong."
It was late at night on December 22, and the meeting in Musk's 10th floor conference room at X, formerly Twitter, had become tense. He was talking to two X infrastructure managers who had not dealt with him much before, and certainly not when he was in a foul mood.
One of them tried to explain the problem. The data-services company that housed one of X's server farms, located in Sacramento, had agreed to allow them some short-term extensions on their lease so they could begin to move out during 2023 in an orderly fashion. "But this morning," the nervous manager told Musk, "they came back to us and said that plan was no longer on the table because, and these are their words, they don't think that we will be financially viable."
The facility was costing X more than $100 million a year. Musk wanted to save that money by moving the servers to one of X's other facilities, in Portland, Oregon. Another manager at the meeting said that couldn't be done right away. "We can't get out safely before six to nine months," she said in a matter-of-fact tone. "Sacramento still needs to be around to serve traffic."
Over the years, Musk had been faced many times with a choice between what he thought was necessary and what others told him was possible. The result was almost always the same. He paused in silence for a few moments, then announced, "You have 90 days to do it. If you can't make that work, your resignation is accepted."
The manager began to explain in detail some of the obstacles to relocating the servers to Portland. "It has different rack densities, different power densities," she said. "So the rooms need to be upgraded." She started to give a lot more details, but after a minute, Musk interrupted.
"This is making my brain hurt," he said.
"I'm sorry, that was not my intention," she replied in a measured monotone.
"Do you know the head-explosion emoji?" he asked her. "That's what my head feels like right now. What a pile of f---ing bulls---. Jesus H f---ing Christ. Portland obviously has tons of room. It's trivial to move servers one place to another."
The X managers again tried to explain the constraints. Musk interrupted. "Can you have someone go to our server centers and send me videos of the insides?" he asked.
It was three days before Christmas, and the manager promised the video in a week. "No, tomorrow," Musk ordered. "I've built server centers myself, and I can tell if you could put more servers there or not. That's why I asked if you had actually visited these facilities. If you've not been there, you're just talking bulls---."
SpaceX and Tesla were successful because Musk relentlessly pushed his teams to be scrappier, more nimble, and to launch fire-drill surges that extruded all obstacles. That's how they had cobbled together a car production line in a tent in Fremont and a test facility in the Texas desert and a launch site at Cape Canaveral made of used parts.
"All you need to do is just move the f---ing servers to Portland," he said. "If it takes longer than 30 days, that would blow my mind." He paused and recalculated. "Just get a moving company, and it will take a week to move the computers and another week to plug them in. Two weeks. That's what should happen."
Everyone was silent. But Musk was still warming up. "If you got a godd--- U-Haul, you could probably do it by yourself." The two X managers looked to see if he was serious. Two of Musk's top loyalists, Steve Davis and Omead Afshar were also at the table. They had seen him like this many times before, and they knew that he might be.
"Why don't we do it right now?" James Musk asked.
James and his brother Andrew, younger first cousins of Musk, were flying with him from San Francisco to Austin on Friday evening, December 23, the day after the frustrating infrastructure meeting about how long it would take to move the servers out of the Sacramento facility. Avid skiers, they had planned to go by themselves to Tahoe for Christmas, but Elon that day invited them to come to Austin instead.
James was reluctant. He was mentally exhausted and didn't need more intensity, but Andrew convinced him that they should go. So that's how they ended up on the plane listening to Elon complain about the servers.
They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved. It was already late evening, but he told his pilot to divert, and they made a loop back up to Sacramento.
The only rental car they could find when they landed was a Toyota Corolla. They were not sure how they would even get inside the data center at night, but one very surprised X staffer, a guy named Alex from Uzbekistan, was still there. He merrily let them in and showed them around.
The facility, which housed rooms of servers for many other companies as well, was very secure, with a retinal scan required for entry into each of the vaults. Alex the Uzbek was able to get them into the X vault, which contained about 5,200 refrigerator-size racks of 30 computers each.
"These things do not look that hard to move," Elon announced. It was a reality-distorting assertion, since each rack weighed about 2,500 pounds and was eight feet tall.
"You'll have to hire a contractor to lift the floor panels," Alex said. "They need to be lifted with suction cups." Another set of contractors, he said, would then have to go underneath the floor panels and disconnect the electric cables and seismic rods.
Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved.
"Well that doesn't seem super hard," he said as Alex the Uzbek and the rest of the gang stared. Musk was totally jazzed by this point. It was, he said with a loud laugh, like a remake of Mission: Impossible, Sacramento edition.
The next day — Christmas Eve — Musk called in reinforcements. Ross Nordeen, who worked with his friend James at Tesla, drove from San Francisco. He stopped at the Apple Store in Union Square and spent $2,000 to buy out the entire stock of AirTags so the servers could be tracked on their journey, and then stopped at Home Depot, where he spent $2,500 on wrenches, bolt-cutters, headlamps, and the tools needed to unscrew the seismic bolts.
Steve Davis, a loyal Musk lieutenant, got someone to procure a semi truck and line up moving vans. Other enlistees arrived from SpaceX. The server racks were on wheels, so the team was able to disconnect four of them and roll them to the waiting truck. This showed that all fifty-two hundred or so could probably be moved within days. "The guys are kicking rear end!" Musk exulted.
Other workers at the facility watched with a mix of amazement and horror. Musk and his renegade team were rolling servers out without putting them in crates or swaddling them in protective material, then using store-bought straps to secure them in the truck. "I've never loaded a semi before," James admitted. Ross called it "terrifying." It was like cleaning out a closet, "but the stuff in it is totally critical."
At 3 p.m., after they had gotten four servers onto the truck, word of the caper reached the top executives at NTT, the company that owned and managed the data center. They issued orders that Musk's team halt. Musk had the mix of glee and anger that often accompanied one of his manic surges. He called the CEO of the storage division, who told him it was impossible to move server racks without a bevy of experts. "Bulls---," Musk explained. "We have already loaded four onto the semi."
The CEO then told him that some of the floors could not handle more than 500 pounds of pressure, so rolling a 2,000-pound server would cause damage. Musk replied that the servers had four wheels, so the pressure at any one point was only 500 pounds. "The dude is not very good at math," Musk told the musketeers.
Having ruined the Christmas Eve of the NTT managers, as well as hitting them with a potential loss of more than $100 million in revenue for the coming year, Musk showed pity and said he would suspend moving the servers for two days. But they would resume, he warned, the day after Christmas.
After Christmas, Andrew and James headed back to Sacramento to see how many more servers they could move. They hadn't brought enough clothes, so they went to Walmart and bought jeans and T-shirts.
The moving contractors that NTT wanted them to use charged $200 an hour. So James went on Yelp and found a company named Extra Care Movers that would do the work at one-tenth the cost. The motley company pushed the ideal of scrappiness to its outer limits. The owner had lived on the streets for a while, then had a kid, and he was trying to turn his life around. He didn't have a bank account, so James ended up using PayPal to pay him.
The second day, the crew wanted cash, so James went to a bank and withdrew $13,000 from his personal account. Two of the crew members had no identification, which made it hard for them to sign into the facility. But they made up for it in hustle. "You get a dollar tip for every additional server we move," James announced at one point. From then on, when they got a new one on a truck, the workers would ask how many they were up to.
The servers had user data on them, and James did not initially realize that, for privacy reasons, they were supposed to be wiped clean before being moved. "By the time we learned this, the servers had already been unplugged and rolled out, so there was no way we would roll them back, plug them in, and then wipe them," he says. Plus, the wiping software wasn't working. "F---, what do we do?" he asked. Elon recommended that they lock the trucks and track them.
So James sent someone to Home Depot to buy big padlocks, and they sent the combination codes on a spreadsheet to Portland so the trucks could be opened there. "I can't believe it worked," James says. "They all made it to Portland safely."
By the end of the week, they had used all of the available trucks in Sacramento. Despite the area being pummeled by rain, they moved more than 700 of the racks in three days. The previous record at that facility had been moving 30 in a month. That still left a lot of servers in the facility, but the musketeers had proven that they could be moved quickly. The rest were handled by the X infrastructure team in January.
All very exciting and inspiring, right? An example of Musk's bold and scrappy approach! But as with all things Musk, it was, alas, not that simple. It was also an example of his recklessness, his impatience with pushback, and the way he intimidated people. X's infrastructure engineers had tried to explain to him, in that head-explosion-emoji meeting a week earlier, why a quick shutdown of the Sacramento center would be a problem, but he shot them down. He had a good track record of knowing when to ignore naysayers. But not a perfect one.
For the next two months, X was destabilized. The lack of servers caused meltdowns, including when Musk hosted a Twitter Spaces for presidential candidate Ron DeSantis. "In retrospect, the whole Sacramento shutdown was a mistake," Musk would admit in March 2023. "I was told we had redundancy across our data centers. What I wasn't told was that we had 70,000 hard-coded references to Sacramento. And there's still poo poo that's broken because of it."
His most valuable lieutenants at Tesla and SpaceX had learned ways to deflect his bad ideas and drip-feed him unwelcome information, but the legacy employees at X didn't know how to handle him. That said, X survived. And the Sacramento caper showed X employees that he was serious when he spoke about the need for a maniacal sense of urgency.
Love to be a huge loving idiot at work
|
#
?
Sep 12, 2023 05:49
what would this look like?
this actually made me really happy for some reason - ive been having a lovely stressful past few weeks
