Amazon Web Services - Cloud Giant Hits Hard - The Something Awful Forums

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Amazon Web Services - Cloud Giant Hits Hard

«‹›61 »

Adhemar: Jan 21, 2004; Kellner, da ist ein scheussliches Biest in meiner Suppe.

And that reason is bucket sniping.

# ? Aug 23, 2022 16:33

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 16:37

fluppet: Feb 10, 2009

Best one I saw was <developer name>.<company name> hosting production assets. Every couple of months there someone would come along and ask about deleting that bucket.

Keith had long since departed the company

# ? Aug 23, 2022 16:56

Agrikk: Oct 17, 2003; Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

fluppet posted:

Best one I saw was <developer name>.<company name> hosting production assets. Every couple of months there someone would come along and ask about deleting that bucket.

Keith had long since departed the company

No no no. Keith is the name of the bucket. Keith named it after himself, but �Keith has a million objects, Keith is leaking PII, AWS got after us for bad API calls to Keith�.

Keith the developer is long gone but Keith the bucket is alive and well.

Keith is an rear end in a top hat.

# ? Aug 23, 2022 17:38

Ajaxify: May 6, 2009

Docjowles posted:

I just had to pull this out for appreciation. There is always a bucket called $companyname and it is always like the first thing some rando dev ever did in AWS years before the rest of the org thought about using the cloud. It will be a giant dumping ground of poo poo with no lifecycle policy and probably leaking PII.

There will also be some horrible reason you can't just easily fix it.

Verily.

Our company was serving assets from a self-named S3 bucket in the CEO/Founder's personal AWS account up until a couple years ago. The CEO hasn't written any code in probably a decade.

Given we have 2.5m concurrent users on average, It was probably quite the bill.

# ? Aug 23, 2022 18:11

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

That's certainly a bill worth putting on your Amazon credit card

# ? Aug 23, 2022 18:16

Arzakon: Nov 24, 2002; "I hereby retire from Mafia"
Please turbo me if you catch me in a game.

Does signing up for an AWS account still result in an Amazon.com account being created so you can have your corporate infrastructure spend next to your personal 100 gallon barrel of lube and dragon shaped dildo purchases?

# ? Aug 23, 2022 19:33

Woof Blitzer: Dec 29, 2012; [-]

Docjowles posted:

I just had to pull this out for appreciation. There is always a bucket called $companyname and it is always like the first thing some rando dev ever did in AWS years before the rest of the org thought about using the cloud. It will be a giant dumping ground of poo poo with no lifecycle policy and probably leaking PII.

There will also be some horrible reason you can't just easily fix it.

Too real for my taste.

# ? Aug 24, 2022 00:59

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

There is probably a museum gallery of horrible infrastructure patterns in the cloud now that makes the random server that was accidentally enclosed behind drywall and found out by having the cable traced when it finally went down one day seem like a nothingburger. If not, I'm going to goddamn start collecting them to make everyone's eyes bleed.

# ? Aug 24, 2022 01:16

Pile Of Garbage: May 28, 2007

90% of critical business infrastructure is dependent on a Excel spreadsheet and a Dell OptiPlex sitting under some dudes desk.

# ? Aug 24, 2022 01:52

ledge: Jun 10, 2003

Pile Of Garbage posted:

90% of critical business infrastructure is dependent on a ~~Excel spreadsheet~~ Access database and a Dell OptiPlex sitting under some dudes desk.

fixed that for you.

# ? Aug 24, 2022 01:57

Agrikk: Oct 17, 2003; Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

Arzakon posted:

Does signing up for an AWS account still result in an Amazon.com account being created so you can have your corporate infrastructure spend next to your personal 100 gallon barrel of lube and dragon shaped dildo purchases?

No. amazon.com accounts have been decoupled from AWS accounts for a while now.

# ? Aug 24, 2022 01:59

Agrikk: Oct 17, 2003; Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

necrobobsledder posted:

There is probably a museum gallery of horrible infrastructure patterns in the cloud now that makes the random server that was accidentally enclosed behind drywall and found out by having the cable traced when it finally went down one day seem like a nothingburger. If not, I'm going to goddamn start collecting them to make everyone's eyes bleed.

I�ve seen things you people wouldn�t believe.
Attack ships, etc.

# ? Aug 24, 2022 02:00

Woodsy Owl: Oct 27, 2004

The only nice thing about terraform are the non-cloud providers. Assuming they are actually maintained. That's it.

# ? Aug 24, 2022 05:26

12 rats tied together: Sep 7, 2006

First time I used the ns1 provider it set "the entire contents of this zone" to the 1 record that I asked it to provision, deleting thousands of records without indication. Very cool thank you, I'll stick with ansible for DNS though.

# ? Aug 24, 2022 05:42

BaseballPCHiker: Jan 16, 2006

Im always behind the times. Just learning Terraform and now I have to learn this Pulami poo poo!

I just want to coast 20 years and retire already.

Biggest spend at my company is ec2. I've seen instances that got spun up by a dev for some testing that were forgotten about that stay up running for months costing the company thousands. Scale that x2-300 and its a good chunk of change. Of course my company is dumb so no surprise there.

# ? Aug 24, 2022 15:58

The Fool: Oct 16, 2003

Terraform isn't going anywhere anytime soon.

I agree that it has stagnated and Hashi seems to be struggling to keep up with innovation in the IaC space, but it will continue to be an employable skill for years to come.

Just like PHP.

# ? Aug 24, 2022 16:09

vanity slug: Jul 20, 2010

I wish Terraform had import blocks, so that I wouldn't have to do it manually (hello aws_system_linked_role my old friend)

# ? Aug 24, 2022 18:16

Pile Of Garbage: May 28, 2007

As far as AWS is concerned learn CFN to understand the platform and then look at TF and Ansible if you want to do IaC. Only look at CDK and Pulumi if you want to do tightly-integrated IaC. If your applications are expected to operate at a platform level and interact with cloud then yeah use them, otherwise please don't you'll just make a nightmare for whoever picks up the pieces for support.

# ? Aug 24, 2022 18:25

12 rats tied together: Sep 7, 2006

Jeoh posted:

I wish Terraform had import blocks, so that I wouldn't have to do it manually (hello aws_system_linked_role my old friend)

Agreed, that is one of my favorite pulumi features: https://www.pulumi.com/docs/intro/concepts/resources/options/import/

TypeScript code:

let group = new aws.ec2.SecurityGroup("web-sg", {
    name: "web-sg-62a569b",
    ingress: [{ protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] }],
}, { import: "sg-04aeda9a214730248" });

imports are modifications, a big benefit of infracode is being able to propose and review modifications using version control. it's crazy that your only option for almost a decade has been to put a date-stamped bash script in a "migrations/" folder somewhere in your terraform repo

Pile Of Garbage posted:

As far as AWS is concerned learn CFN to understand the platform and then look at TF and Ansible if you want to do IaC. Only look at CDK and Pulumi if you want to do tightly-integrated IaC. If your applications are expected to operate at a platform level and interact with cloud then yeah use them, otherwise please don't you'll just make a nightmare for whoever picks up the pieces for support.

I don't necessarily disagree but what do you mean by "tightly-integrated?"

# ? Aug 24, 2022 18:37

Pile Of Garbage: May 28, 2007

12 rats tied together posted:

I don't necessarily disagree but what do you mean by "tightly-integrated?"

Where the core application/service/workload dictates required resources and then spins then up in a programmatic fashion.

This is the opposite of most orgs approach to IaC which is they hear about it, think they should be doing it and then never invest enough so the best infra ops can do is templating environments which is best left to CFN and TF.

# ? Aug 24, 2022 18:54

12 rats tied together: Sep 7, 2006

Cool, I broadly agree, there is a middle ground where the application does not need to specifically be orchestrating pulumi calls (e.g. what if the application does not run continuously), but I think we're on the same page.

Something that gets missed a lot is CloudFormation is way more than just a templating tool, it has really granular update policies, it has rollback triggers, it has event notifications, deployment management policies for some types of resource, etc. Even if you're using Pulumi, if you have a bunch of really important resources that need to live together and share state/data/fate, you should put them in a cfn stack.

Firing off naked api calls in sequence to like, a billing database, has always been a worst practice. It doesn't matter what tool is turning what type of input into the api calls.

# ? Aug 24, 2022 19:25

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

I think decoupled vs. coupled is the distinction. Integration oftentimes means an extension of features rather than something that is transformed by its integration. It's the difference to me between an application supporting K8S vs being deployed within K8S.

CFN to me serves as a vaguely atomic set of changes I can treat as a single black box unit within AWS constructs. If I write Pulumi, TF, or other orchestration code correctly I should be able to support anything that has a similar abstraction level. For example, I should be able to do an Azure Blueprint or GCP Deployment Manager and get similar feature parity overall. In such a situation I think Pulumi wins solidly over TF. Embedding Pulumi into a platform / application seems much easier to do than embedding TF from a programmatic standpoint IMO.

# ? Aug 25, 2022 02:17

jiffypop45: Dec 30, 2011

I'm trying to determine whether this is feasible or if using DynamoDB Local is the better option.

Right now we are using shared databases to do our dev/stage work (one per deployment stage).

We would like to be able to create a temporary table for each test we do to isolate it. In theory I think it would just be a matter of keeping the ARN in a variable and then having our docker connect to it but, I am not sure if DDB is designed to be used this way.

Has anyone done something like this and know of some documentation on it? Most of what I'm finding on line just says to use DDB Local but due to VPC's and other infra dependencies as well as not being a strictly valid test according to CI/CD tests due to the hardware not being the same (at least in Stage, Dev obviously its normative to have a non prod infra setup).

Edit: sorry should have mentioned our CI/CD and IaC is Jenkins with Terraform and Chef.

jiffypop45 fucked around with this message at 16:15 on Aug 30, 2022

# ? Aug 30, 2022 16:11

Docjowles: Apr 9, 2009

Since we were talking about managing AWS costs recently, today in "lol. lmao.":

Senior leadership has a goal to port the vast majority of our stuff out of the data centers into AWS. Naturally, after getting started on this, finance poo poo a brick when they saw the monthly bill. So they forced us to engage a third party consultant to analyze our spending and suggest improvements. It turned out that a handful of our accounts, for various bad reasons, had multiple CloudTrails set up. This was absolutely a lurking mistake, but had not incurred any meaningful cost before. And (I dunno if this is a bug or what) certain resources like SNS topics cause the tool this vendor uses to make roughly 500 ho-jillion API calls per day. Which we were now paying to log because of the aforementioned duplicate CloudTrails that go above and beyond the free tier. And of course, it was the couple accounts that had SNS topics with a zillion subscribers that also had extra CloudTrails. I caught it fairly quickly but still this "free POC" ended up costing us five figures in loving CloudTrail charges of all things.

They did not mention this 1000% increase in CloudTrail spending at all in their final report. Or really provide anything of value, it was basically "you could pay us to micromanage RIs and spending plans for you if you want".

lol. lmao.

The only silver lining is that I feel pretty good about all the cost optimizing work I've been leading. We can do better of course but they couldn't make a compelling case for themselves above and beyond what I've been doing.

# ? Aug 30, 2022 23:36

calusari: Apr 18, 2013; It's mechanical. Seems to come at regular intervals.

I am trying to update a CloudFormation template to whitelist some countries in a CloudFront distribution:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cloudfront-distribution-georestriction.html

How can I set a default list of strings like ['US', 'CA'] for my Ref?

code:

Params:
  GeoRestrictionLocations:
    Type: List<String>
    Default: [what goes here]

CloudFrontDistribution:
  Type: AWS::CloudFront::Distribution
  Properties:
    DistributionConfig:
      Restrictions:
        GeoRestriction:
          Locations: !Ref GeoRestrictionLocations
          RestrictionType: whitelist

# ? Sep 2, 2022 18:21

Thanks Ants: May 21, 2004; #essereFerrari

Isn�t it � rather than �

# ? Sep 2, 2022 21:18

Ajaxify: May 6, 2009

calusari posted:

I am trying to update a CloudFormation template to whitelist some countries in a CloudFront distribution:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cloudfront-distribution-georestriction.html

How can I set a default list of strings like ['US', 'CA'] for my Ref?
code:
Params:
  GeoRestrictionLocations:
    Type: List<String>
    Default: [what goes here]

CloudFrontDistribution:
  Type: AWS::CloudFront::Distribution
  Properties:
    DistributionConfig:
      Restrictions:
        GeoRestriction:
          Locations: !Ref GeoRestrictionLocations
          RestrictionType: whitelist

code:

List<Number>

Is an invalid param type.

I think you want this instead:

code:

CommaDelimitedList

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html

# ? Sep 2, 2022 22:27

Hughmoris: Apr 21, 2007; Let's go to the abyss!

A question about how I should read AWS compensation information in job postings:

Looking at this Cloud Devops I position in Colorado, they list compensation as $82k - $144k but that includes stock units. Is there a rule of thumb I can apply when looking at their job postings, to figure out my actual take-home salary?

# ? Sep 4, 2022 16:44

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

RSUs are separate from pay. Also just assume stock is going to be worthless if it�s a startup.

# ? Sep 4, 2022 16:47

Happiness Commando: Feb 1, 2002; $$ joy at gunpoint $$

As the person above said, that's base pay only. The only way to figure out actual total compensation is to look at levels.fyi and do some fuzzy math. Also depending on the market your total compensation may exceed or trail your comp target, which may get made up for with additional grants, which will be on a (2 year?) vesting schedule

# ? Sep 4, 2022 16:50

Hughmoris: Apr 21, 2007; Let's go to the abyss!

deedee megadoodoo posted:

RSUs are separate from pay. Also just assume stock is going to be worthless if it�s a startup.

Happiness Commando posted:

As the person above said, that's base pay only. The only way to figure out actual total compensation is to look at levels.fyi and do some fuzzy math. Also depending on the market your total compensation may exceed or trail your comp target, which may get made up for with additional grants, which will be on a (2 year?) vesting schedule

Thanks!

# ? Sep 4, 2022 18:16

Arzakon: Nov 24, 2002; "I hereby retire from Mafia"
Please turbo me if you catch me in a game.

Happiness Commando posted:

As the person above said, that's base pay only. The only way to figure out actual total compensation is to look at levels.fyi and do some fuzzy math. Also depending on the market your total compensation may exceed or trail your comp target, which may get made up for with additional grants, which will be on a (2 year?) vesting schedule

It will not trail, it will exceed in certain markets (NYC/SFO). RSU vesting schedule is 4 years (5/15/40/40%) but first two years will have a signing bonus.

# ? Sep 4, 2022 18:26

Ajaxify: May 6, 2009

Arzakon posted:

It will not trail, it will exceed in certain markets (NYC/SFO). RSU vesting schedule is 4 years (5/15/40/40%) but first two years will have a signing bonus.

Happiness Commando posted:

As the person above said, that's base pay only. The only way to figure out actual total compensation is to look at levels.fyi and do some fuzzy math. Also depending on the market your total compensation may exceed or trail your comp target, which may get made up for with additional grants, which will be on a (2 year?) vesting schedule

With regards to RSUs, you're both right. The first grant is 4-years with a signing bonus in the first two years, but subsequent refresh grants are 2-years, starting in the next calendar year. You can get an in year refresh of your RSUs if you are promoted.

# ? Sep 4, 2022 18:56

Happiness Commando: Feb 1, 2002; $$ joy at gunpoint $$

These numbers are totally made up. OP gets hired when AMZN is at $100/share. If OPs total comp target is 200k, comprised of 150k base and 50k RSUs and then the share price takes a dive to $80/share, OP is not guaranteed an in-year vesting grant of RSUs to make up the shortfall. OP might be required to take it on the chin, because "we're all builders and when we build well, the market does well, and then your actual comp will exceed the TCT"

If the market tanks right before a vesting date, OPs compensation may very well trail their TCT. If it stays down, they wont get a top off of RSUs to get them to hit their TCT until the next annual review (edit: or promo), and those RSUs will likely vest on a 2 year schedule.

Happiness Commando fucked around with this message at 19:03 on Sep 4, 2022

# ? Sep 4, 2022 18:59

Docjowles: Apr 9, 2009

Anecdotally from a couple people I know at AWS, Amazon has been at least doing some stuff lately to make people whole if they get turbo hosed on RSU value (mostly granting even more stock, lol). Because lord knows if those golden handcuffs get loose, the other FAANG companies will happily poach talent.

Historically this has not been a problem because AMZN stock only went up. You�d be insane to leave when you had a large vesting coming up that had doubled in value since your start date. But in the face of a downturn or recession, being so heavy on stock comp isn�t the guaranteed jackpot it has been.

Not to make this sound like all doom and gloom. You�re still probably making vastly more at Amazon than most non-FAANG places.

# ? Sep 4, 2022 19:58

jiffypop45: Dec 30, 2011

Docjowles posted:

Anecdotally from a couple people I know at AWS, Amazon has been at least doing some stuff lately to make people whole if they get turbo hosed on RSU value (mostly granting even more stock, lol). Because lord knows if those golden handcuffs get loose, the other FAANG companies will happily poach talent.

Historically this has not been a problem because AMZN stock only went up. You’d be insane to leave when you had a large vesting coming up that had doubled in value since your start date. But in the face of a downturn or recession, being so heavy on stock comp isn’t the guaranteed jackpot it has been.

Not to make this sound like all doom and gloom. You’re still probably making vastly more at Amazon than most non-FAANG places.

Former AWS worker here. AWS pay was absolutely poo poo for the longest time I started as an L4 at 95k base with 25 RSU's. Their salary hard cap was 165k for literally *everyone* Padre Jeffe included. Everything else was bonus and stock golden hand cuffs. 165k is on par with high mid low senior level at other companies.

Amazon only recently in the past year realized they were losing people left and right because its hard to suffer a burn out culture where you're told it's your privilege to work there and deal with endless casual bigotry while being paid poo poo relative to market rate for the same role.

They got rid of their cap and are now paying market but its still a bad company otherwise and you're basically suffering employment there just because it looks good on your resume.

# ? Sep 4, 2022 20:27

Docjowles: Apr 9, 2009

I don�t think I�ve ever seen more divisive opinions on what it�s like to work at a place than Amazon. There are plenty of takes like yours, that it�s a low paying burnout factory. Then I talk to people who have been there going on 5-10 years (so way after the initial churn and burn phase and grant cliff) and think it is the greatest job going.

From the outside it�s pretty interesting.

# ? Sep 4, 2022 20:48

StumblyWumbly: Sep 12, 2007; Batmanticore!

Docjowles posted:

I don�t think I�ve ever seen more divisive opinions on what it�s like to work at a place than Amazon. There are plenty of takes like yours, that it�s a low paying burnout factory. Then I talk to people who have been there going on 5-10 years (so way after the initial churn and burn phase and grant cliff) and think it is the greatest job going.

From the outside it�s pretty interesting.

I interviewed one candidate who came from the business side of AWS, and he said that the separate departments were entirely different kingdoms and the folks in charge had a lot of control and really set the culture (as long as things went well).

Question: Anyone have opinions on working on the VR/device side of Facebook? I'm pretty happy with where I am now, but I know I'm not making Facebook money.

# ? Sep 4, 2022 20:58

Arzakon: Nov 24, 2002; "I hereby retire from Mafia"
Please turbo me if you catch me in a game.

Docjowles posted:

I don�t think I�ve ever seen more divisive opinions on what it�s like to work at a place than Amazon.

In a company with millions of employees it comes down to whether you end up in a good org or a poo poo one.

# ? Sep 4, 2022 21:01

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 16:37

jiffypop45: Dec 30, 2011

StumblyWumbly posted:

I interviewed one candidate who came from the business side of AWS, and he said that the separate departments were entirely different kingdoms and the folks in charge had a lot of control and really set the culture (as long as things went well).

Question: Anyone have opinions on working on the VR/device side of Facebook? I'm pretty happy with where I am now, but I know I'm not making Facebook money.

I know someone who does the audio engineering side of echo. Other than awful vendors she likes it but shes also trapped because its basically unheard of to get software dev pay with an audio engineering degree.

jiffypop45 fucked around with this message at 22:46 on Sep 4, 2022

# ? Sep 4, 2022 21:01

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Amazon Web Services - Cloud Giant Hits Hard

«‹›61 »