quote:

Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon's vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon's dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:

- pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.

- every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.

- monitoring and QA are the same thing. You'd never think so until you try doing a big SOA. But when your service says "oh yes, I'm fine", it may well be the case that the only thing still functioning in the server is the little component that knows how to say "I'm fine, roger roger, over and out" in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. So they're a continuum.

- if you have hundreds of services, and your code MUST communicate with other groups' code via these services, then you won't be able to find any of them without a service-discovery mechanism. And you can't have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.

- debugging problems with someone else's code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.

Cocoa Crispies posted:

that's why you tie your goddamn shoes right

WHOIS John Galt posted:

tying two conversations together one of the most interesting things about yegge's post was this

how!! posted:

I'm going to tell you a true story. When I was in middle school, I was on the track team. One day during a track meet, I was running the 400 meter dash. I made the mistake of not making sure my shoes were tied tightly before the race started. After rounding the first turn, I looked down and noticed my shoe was untied. I couldn't just stop and tie my shoe, because I'm running a race for gods sake. After the third turn, my shoe had almost come completely off because the laces flinging around had made the shoe so loose. When I crossed the finish line I was practically limping because I couldn't run normally or else my shoe would come completely off. I was too afraid to stop and do what needed to be done.

If you have a project full of technical debt, its like running a race with a shoe coming half off your foot. Yeah stopping to tie the show (refactoring code) will slow you down, but not as much as not stopping. A lot of companies act like they're in a sprint, but many times they are not. Most companies are running a distance race, where a week or a month doesn't matter as much as they fear it will.

Shaggar posted:

yes

Wheany posted:

you should write a blog

how!! posted:

I'm going to tell you a true story. When I was in middle school, I was on the track team. One day during a track meet, I was running the 400 meter dash. I made the mistake of not making sure my shoes were tied tightly before the race started. After rounding the first turn, I looked down and noticed my shoe was untied. I couldn't just stop and tie my shoe, because I'm running a race for gods sake. After the third turn, my shoe had almost come completely off because the laces flinging around had made the shoe so loose. When I crossed the finish line I was practically limping because I couldn't run normally or else my shoe would come completely off. I was too afraid to stop and do what needed to be done.

If you have a project full of technical debt, its like running a race with a shoe coming half off your foot. Yeah stopping to tie the show (refactoring code) will slow you down, but not as much as not stopping. A lot of companies act like they're in a sprint, but many times they are not. Most companies are running a distance race, where a week or a month doesn't matter as much as they fear it will.

Win8 Hetro Experie posted:

it's more like "uses ALL of http, especially the nerdy bits"

like how well does some REST application work when a hotel wifi provider decides to turn one request into a temporary redirect to their login page?

Shaggar posted:

but yes its a good example of why http codes are unreliable and you should check the message content as well.

tef posted:

welcome to pager duty

Gazpacho posted:

i could tell you about the heap corruption bug

how!! posted:

I'm a much better programmer than I am writer. I spend most my free time programming.

rotor posted:

no, please dont.

how!! posted:

I'm a much better programmer than I am writer. I spend most my free time programming.

Shaggar posted:

but yes its a good example of why http codes are unreliable and you should check the message content as well.

WHOIS John Galt posted:

you write the way i thought about the world when i was like... 14? maybe 15?

how!! posted:

I'm a much better programmer than I am writer. I spend most my free time programming.

how!! posted:

I'm a much better programmer than I am writer. I spend most my free time programming.

Gazpacho posted:

oh that wasn't the worst of it, i could tell you about the heap corruption bug that woke devs up in the middle of the night for weeks on end and possibly was responsible for that dev leaving the team, thank god the incoming senior dev knew what a valgrind is

tef posted:

when I talked to amazon devs they all seemed to be going 'queues are awesome', then explaining how they dealt with all of the failure cases by getting paged.

how!! posted:

I'm going to tell you a true story. When I was in middle school, I was on the track team. One day during a track meet, I was running the 400 meter dash. I made the mistake of not making sure my shoes were tied tightly before the race started. After rounding the first turn, I looked down and noticed my shoe was untied. I couldn't just stop and tie my shoe, because I'm running a race for gods sake. After the third turn, my shoe had almost come completely off because the laces flinging around had made the shoe so loose. When I crossed the finish line I was practically limping because I couldn't run normally or else my shoe would come completely off. I was too afraid to stop and do what needed to be done.

If you have a project full of technical debt, its like running a race with a shoe coming half off your foot. Yeah stopping to tie the show (refactoring code) will slow you down, but not as much as not stopping. A lot of companies act like they're in a sprint, but many times they are not. Most companies are running a distance race, where a week or a month doesn't matter as much as they fear it will.

Shaggar posted:

who can I page about the horrific web design on amazon.com

stevey posted:

Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally -- wisely -- left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children.

Werthog 95 posted:

doesn't matter what distance race you're running if your shoe's half-off you kick it off and run barefoot you idiot

Werthog 95 posted:

oh right it's a metaphor. well it's a really lovely one because any half-decent runner knows you don't stop for poo poo

Gazpacho posted:

Gazpacho posted: