Amazon Web Services - Cloud Giant Hits Hard - The Something Awful Forums

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Amazon Web Services - Cloud Giant Hits Hard

«‹›61 »

Blinkz0rz: May 27, 2001; MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Agrikk posted:

Or you can do what a customer of mine did:

Be really clever and buy an iPhone and put all eighty of their accounts� root 2FA on an instance of google Authenticator and keep the iPhone in a bombproof safe.

They were all kinds of until someone dropped the phone.

I had to fly down there and get on a video call with our legal department and me sitting next to their leadership and vouch that their leadership was actually their leadership and we all had to present IDs and say who we were and that we were authorized to remove MFA from the account.

We got to do this eighty times.

We were about to do this for our root accounts but decided against phone-in-safe because it forced recovery responsibility on only one office when our platform engineering org is geographically distributed.

I asked our TAM and he never got back to me with suggestions so we ended up getting 1password solely for this scenario but I can't help but think there are better solutions out there.

# ? Aug 19, 2019 13:59

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 03:16

crazysim: May 23, 2004; I AM SOOOOO GAY

https://en.m.wikipedia.org/wiki/Shamir%27s_Secret_Sharing ?

# ? Aug 19, 2019 14:25

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

FamDav posted:

You want permissions boundaries. Specifically, you can require create-user/role to include a permissions boundary.

Thanks Ants posted:

An explicit deny rule will always take precedence over anything else, maybe you can get creative with those.

Just an FYI: neither of these options does what I'm looking for.

Permissions Boundaries set a boundary on what a user can do. And explicit deny rules only prevent a user from performing certain types of actions. The issue is that if a user has iam:CreatePolicy they can still build a policy that contains higher privileges and use the policy via an alternate method like assuming a role or assigning it to an ec2 instance. Nothing actually prevents a user from creating a policy that contains specific statements and there are enough other ways to use that policy that the only real way ends up being inspecting the policy on creation.

# ? Aug 19, 2019 14:58

Blinkz0rz: May 27, 2001; MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

crazysim posted:

https://en.m.wikipedia.org/wiki/Shamir%27s_Secret_Sharing ?

For 2fa tokens?

# ? Aug 19, 2019 15:02

crazysim: May 23, 2004; I AM SOOOOO GAY

Blinkz0rz posted:

For 2fa tokens?

Maybe? I hate to do the equivalent of block chain this or that but maybe something using Shamir's algorithm might be useful for distributed responsibility and availability of the 2FA tokens.

# ? Aug 19, 2019 15:22

freeasinbeer: Mar 26, 2015; by Fluffdaddy

Vault does support being a virtual 2fa. If you really wanted to distribute it.

Edit: https://www.vaultproject.io/docs/secrets/totp/index.html

# ? Aug 19, 2019 15:28

Blinkz0rz: May 27, 2001; MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Running vault on prem means we rely on IT to keep it running, always on, and always available while the folks who would actually need the service work in a different org.

If they ran it, it'd be in AWS which feels not awesome to run a recovery system in an environment that could need to be recovered.

We weren't able to find anything that split the uprights between always available, low touch, and secure.

# ? Aug 19, 2019 15:33

Docjowles: Apr 9, 2009

We run 1password for our root MFAs for the same reason. It seems like the least terrible choice. Plus we already have a corporate subscription anyway.

# ? Aug 19, 2019 16:48

xpander: Sep 2, 2004

Docjowles posted:

holy poo poo I'm starting to see why you tout TAM as a fun and cool job so much.

For anyone struggling with 2FA, I strongly recommend ditching individual IAM accounts and just using your corporate SSO solution. Because yeah, dealing with 2FA loving sucks. If you are at a company of any size you hopefully already have some sort of SSO backed by 2FA and you can just reuse that instead of making every AWS user set up a second solution. And not hate your life twice as much every time someone drops their phone in the toilet.

This has the added benefit that engineers do not have permanent access keys. Can't upload your god-mode key to GitHub if you don't have a key You can request temporary keys once you authenticate via SSO, and we make users do this. I wrote a lovely script that makes it very easy to authenticate to our SSO, pick which AWS account you want to work in (filtered to the set this user can access based on their Active Directory groups), and then dump the temp creds to their local environment. Some of the SSO vendors even provide this out of the box. Doing this has already paid un(?)expected dividends like devs coming to us saying "hey I run this production critical job from my laptop every day under my user, and now that's not possible, what gives?" and we can gently repoint them toward not loving running critical jobs from their laptops with admin access.

Apps running on EC2 instances should use IAM instance profiles to assume a role that can do what they need. There will always be service accounts that need an actual IAM user with a long-lived key. But that should be the last resort choice, IMO.

Actual human using AWS? Access via SSO with 2FA, get temp API keys if needed
Application running in AWS? Use IAM roles
App running elsewhere that needs to access AWS resources? OK fine, you get a key but it's restricted to the minimal set of features said app requires. And it's expiring on a set schedule.

That's also what we do at the MSO I work at now, and it's pretty much the One True Way if you have the resources to set it up. Long-running credentials, especially for your admin users, are a ticking time bomb.

# ? Aug 19, 2019 16:56

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

So apparently an issue I was having with Batch was related to how it parses variables. I was expecting the job to pass the command literally to the EC2 container, but apparently it wraps your commands in hidden quotes, so if you use a variable with spaces you don�t need to use enclosing quotes even when the command displays it like you should. If you do try to enclose them yourself, the job assumes you want to convert that to URI code and include it in the variable, so ��hello world�� is passed as �%27hello world%27� and you get a bad request.

So

code:

java -jar code.jar �parameter Ref::Var

Where Var is �hello world� displays as

code:

java -jar code.jar �parameter hello world

But what it really means is

code:

java -jar code.jar �parameter �hello world�

In the job submission info, it behaves as if the variable was one string. I mean, that�s correct, but considering if you copy/paste that into a command line it blows up it made for a ton of confusion. It does not help that is this not documented anywhere.

PierreTheMime fucked around with this message at 17:12 on Aug 19, 2019

# ? Aug 19, 2019 17:09

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

PierreTheMime posted:

So apparently an issue I was having with Batch was related to how it parses variables. I was expecting the job to pass the command literally to the EC2 container, but apparently it wraps your commands in hidden quotes, so if you use a variable with spaces you don�t need to use enclosing quotes even when the command displays it like you should. If you do try to enclose them yourself, the job assumes you want to convert that to URI code and include it in the variable, so ��hello world�� is passed as �%27hello world%27� and you get a bad request.

So
code:
java -jar code.jar �parameter Ref::Var
Where Var is �hello world� displays as
code:
java -jar code.jar �parameter hello world
But what it really means is
code:
java -jar code.jar �parameter �hello world�
In the job submission info, it behaves as if the variable was one string. I mean, that�s correct, but considering if you copy/paste that into a command line it blows up it made for a ton of confusion. It does not help that is this not documented anywhere.

new thread title please Amazon Web Services - It does not help that is this not documented anywhere

# ? Aug 19, 2019 17:15

xpander: Sep 2, 2004

deedee megadoodoo posted:

new thread title please Amazon Web Services - It does not help that is this not documented anywhere

mods

# ? Aug 19, 2019 17:17

Startyde: Apr 19, 2007; come post with us, forever and ever and ever

It�s �exec� style arg passing. Annoying but yea

# ? Aug 19, 2019 17:37

FamDav: Mar 29, 2008

deedee megadoodoo posted:

Just an FYI: neither of these options does what I'm looking for.

Permissions Boundaries set a boundary on what a user can do.

And you can additionally refer to permission boundaries in your own policies context keys. If you read through that, you can write a policy that ensures:

1. Every new user or role has a specific permission boundary
2. Every policy attached to a user or role must only be attached to a user or role that has a particular permission boundary set

So your scenario of a user �creating a role, attaching it to an ec2 instance/lambda/ex�s task� is no longer an issue, as you can ensure that any role user or policy they created has to have a permission boundary on it, and so can�t exceed the privileges you wanted to give out in the first place.

# ? Aug 19, 2019 19:16

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

FamDav posted:

And you can additionally refer to permission boundaries in your own policies context keys. If you read through that, you can write a policy that ensures:

1. Every new user or role has a specific permission boundary
2. Every policy attached to a user or role must only be attached to a user or role that has a particular permission boundary set

So your scenario of a user �creating a role, attaching it to an ec2 instance/lambda/ex�s task� is no longer an issue, as you can ensure that any role user or policy they created has to have a permission boundary on it, and so can�t exceed the privileges you wanted to give out in the first place.

This sounds good in theory and might work, but I do not see how it is possible. The documentation for this incredibly dense. And I have the sneaking suspicion that like a lot of things, it's not something that can be done through the console, which makes it that much harder to implement due to the way our teams work.

# ? Aug 19, 2019 19:43

12 rats tied together: Sep 7, 2006

Permissions boundaries should work for you, unless I'm really missing something here?

You would basically:

Create a managed policy that blocks all of the stuff you want blocked (any action, not just iam:Create*), these have to be Explicit Deny statements though
Create a permissions boundary using/bound to that managed policy
Apply a policy to all of your applicable principals that allows iam:Create* with a Condition block that specifies that a permissions boundary must be attached, and it must be the boundary you created earlier

Your users can't create users or roles that do not have your permissions boundary attached. The boundary contains your explicit deny blacklist, which supercedes any explicit allow statements that your users include. They can't detach the boundary, and they can only create entities with the boundary attached, so your explicit deny blacklist is always present and always takes precedence over any policy your users configure.

They can still create whatever policy they want, with any content they want, they just will never actually get permissions that exceed what you configure for them initially. This should all be doable in the console.

# ? Aug 19, 2019 20:09

FamDav: Mar 29, 2008

The important bit is you can define the permission boundary condition in the permission boundary policy itself. That�s what forces it to be infectious.

They also don�t have to be explicit denys; when permissions are evaluated both the entity policies and the permission boundary policies have to evaluate to an explicit allow.

The first doc I linked also has a good walkthrough with examples of doing all of this.

# ? Aug 19, 2019 20:27

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

The closest thing I think to what you're looking for is to only allow use of specific managed policies (can only create IAM users and roles with certain managed policies and that there is no inline policy for the user or role allowed) and to make it impossible for an IAM user that creates other IAM users to be able to even read their credentials in the process. The ability to create an IAM user or role or such is extremely powerful and their specific role is to create users not to be able to read anything about them such that I think such a system should be fully automated and not possible to get credentials to run as by a human in a large scale AWS account schema.

# ? Aug 19, 2019 21:42

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

Is there a functional AWS automation platform or is it just a Wild West of scripts and bespoke code? After having written some Lambda and Batch stuff, what�s the preferred method to monitor and control them from a central point, or does such a thing even exist?

I�m coming from enterprise scheduling tools, so jumping from tab to tab to work with things in the console just seems untenable. Am I suppose to write my own?

# ? Aug 20, 2019 01:02

Adhemar: Jan 21, 2004; Kellner, da ist ein scheussliches Biest in meiner Suppe.

Are you looking for something like Step Functions? If not, can you give a more specific example of what you�re trying to do?

# ? Aug 20, 2019 02:16

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

Adhemar posted:

Are you looking for something like Step Functions? If not, can you give a more specific example of what you�re trying to do?

That covers a good amount of it. I suppose I'm still at the point where I just haven't acclimated to how AWS manages resources and will need to do some more work with it before bugging people about it.

# ? Aug 20, 2019 14:05

SnatchRabbit: Feb 23, 2006; by sebmojo

Does anoyone know how to setup sns notifications for Config on a per rule basis? I know that I can remediate rules with PublishSNSTopic SSM document, but the remediation seems like a manual step unless I'm missing something. Basically I need some way to either query the config rule on a schedule and then publish to SNS or kick off the remediation on a schedule?

edit: maybe a python lambda for start_remediation_execution() on the particular rule running on a cloudwatch schedule? Or am I overthinking?

SnatchRabbit fucked around with this message at 19:26 on Aug 21, 2019

# ? Aug 21, 2019 19:23

bergeoisie: Aug 29, 2004

Why is it that when I click Request Increase from the EC2 Limits page does it not automatically fill in the region and instance type. It just feels like such an obvious UX improvement.

# ? Aug 23, 2019 19:35

Startyde: Apr 19, 2007; come post with us, forever and ever and ever

The first rule of AWS is Amazon hates you
The second rule is never forget Rule Number One
The cloudwatch logs interface alone is proof enough that Bezos hates you and wants you to suffer.

# ? Aug 24, 2019 22:45

PierreTheMime: Dec 9, 2004; Hero of hormagaunts everywhere!; Buglord

Startyde posted:

The first rule of AWS is Amazon hates you
The second rule is never forget Rule Number One
The cloudwatch logs interface alone is proof enough that Bezos hates you and wants you to suffer.

I had my first real foray into this the other day and good lord it is the worst. I found putting the log from the cli to a converted PowerShell object and just grabbing the message portion helps but it can only handle so much.

# ? Aug 24, 2019 23:13

Agrikk: Oct 17, 2003; Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

Startyde posted:

The first rule of AWS is Amazon hates you

Hah hah hah! I�m getting this made into a t-shirt and am going to wear it in the office.

# ? Aug 24, 2019 23:48

Vanadium: Jan 8, 2005

Why are you using the Cloudwatch Logs interface when you could be using Cloudwatch Logs Insights!!

# ? Aug 25, 2019 15:01

fluppet: Feb 10, 2009

Vanadium posted:

Why are you using the Cloudwatch Logs interface when you could be using Cloudwatch Logs Insights!!

Both are awful

# ? Aug 25, 2019 15:08

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

I am of the opinion AWS cares little for that interface knowing full well most customers would start exporting the logs to another system and they�d make their money one way or another.

It really is odd how some features have nothing for years and show up all of a sudden out of nowhere (EC2 launching instances with encryption from an unencrypted AMI was like 3 years late IMO) while other things like UX options take a seat. A skinnable / brandable AWS console would probably help some for power users but would make a lot of beginners� lives worse searching through docs for screenshots.

I would love to see a future where people don�t typically start from the console but from some form of automated infrastructure as code setup. Azure does this pretty well actually by letting you export your instance launches as code which you can at least drop into Github in retrospect. That saved me a fair bit of time when having to quickly launch and document some work there compared to AWS writing either CF or Terraform first to be able to document it with code. This more or less seems like Dreamweaver for infrastructure people I guess.

# ? Aug 25, 2019 16:17

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

One of my favorite new interface quirks is if you define a cloudwatch rule via cloudformation it will show up in both the cloudwatch and event bridge sections of the console and present the same information but in a different format. You can then edit the rule in event bridge but if you try to save it generates a bunch of console errors. Good stuff. I thought I was going crazy trying to figure out why I couldn�t edit my rules.

# ? Aug 25, 2019 17:48

vanity slug: Jul 20, 2010

AWS Systems Manager is easily the worst part of the AWS console. It's like it's trying to be actively hostile to users.

# ? Aug 25, 2019 20:53

freeasinbeer: Mar 26, 2015; by Fluffdaddy

The most hostile ones I�ve had to deal with recently were EKS and MKS.

# ? Aug 25, 2019 21:31

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

Yeah the SSM console is so bad that we are considering writing our own for dealing with parameter store.

# ? Aug 25, 2019 21:34

vanity slug: Jul 20, 2010

deedee megadoodoo posted:

Yeah the SSM console is so bad that we are considering writing our own for dealing with parameter store.

i'd pay for a usable version tbh

# ? Aug 25, 2019 21:39

12 rats tied together: Sep 7, 2006

I barely know how to use the web ui for AWS because my primary interface to it is text editor and terminal. The services where the UI is part of the value add like logs (and insights), EMR, lambda are all great, except DataPipeline which is garbage.

Maybe this is a bit of a hot take but I would never touch an SSM parameter in the UI -- I'd probably end up explicitly blocking that on our admin users if it ever comes up. Basically I think you should treat your AWS account like a database, every real change should be applied through a tagged migration or some facsimile. The interface is only useful for its ability to colocate bits of relevant information and letting people poke around in object/log storage.

# ? Aug 25, 2019 22:52

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

It must be nice in the fantasy world you live in.

# ? Aug 26, 2019 01:53

Cancelbot: Nov 22, 2006; Canceling spam since 1928

Cross posting here for the Agrikk goodness. I've reached the final stage (on site interview) to become an AWS TAM :woop:

I've had a little help from our current TAM and the Enterprise Support Manager after my second screen interview but i'm still super nervous. I've been writing down as many stories as I could for each of the leadership principles but concerned I'll forget it all and crash and burn when put in front of the 5 or 6 TAMs i'll face during the day.

I can always fall back on "not being an idiot" if I stumble I suppose. But as a TAM yourself Agrikk what should I be demonstrating during this process?

# ? Aug 26, 2019 13:40

vanity slug: Jul 20, 2010

12 rats tied together posted:

I barely know how to use the web ui for AWS because my primary interface to it is text editor and terminal. The services where the UI is part of the value add like logs (and insights), EMR, lambda are all great, except DataPipeline which is garbage.

Maybe this is a bit of a hot take but I would never touch an SSM parameter in the UI -- I'd probably end up explicitly blocking that on our admin users if it ever comes up. Basically I think you should treat your AWS account like a database, every real change should be applied through a tagged migration or some facsimile. The interface is only useful for its ability to colocate bits of relevant information and letting people poke around in object/log storage.

you realize ssm is more than just the parameter store?

# ? Aug 26, 2019 17:48

12 rats tied together: Sep 7, 2006

I do, yes, but I would only specifically take issue with modifying a parameter in the UI.

I see the issue with systems manager (no longer "simple", iirc) as basically the issues with the wider industry going really hard on Chef/Puppet, AWS expanding their offerings to include OpsWorks, and then everyone remembering that managing server config is only like a third of the problem space for infrastructure engineers (and it's not really the hard part either). It's probably a serviceable orchestration tool but I wouldn't know, I've never needed to use it.

Ansible Tower is the closest thing to a paid product that replaces systems manager that I'm aware of, but I would describe it as worse or at least equally bad. Similar to EMR/DataPipeline, I think this is just one of those areas that is fundamentally difficult to communicate through a UI.

# ? Aug 26, 2019 20:30

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 03:16

Agrikk: Oct 17, 2003; Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

Cancelbot posted:

Cross posting here for the Agrikk goodness. I've reached the final stage (on site interview) to become an AWS TAM I've had a little help from our current TAM and the Enterprise Support Manager after my second screen interview but i'm still super nervous. I've been writing down as many stories as I could for each of the leadership principles but concerned I'll forget it all and crash and burn when put in front of the 5 or 6 TAMs i'll face during the day.

I can always fall back on "not being an idiot" if I stumble I suppose. But as a TAM yourself Agrikk what should I be demonstrating during this process?

Be honest and forthright about what you know and don�t know. You got this far because you are demonstrating a keen mind, a personable demeanor and good judgement. Keep doing that.

It�s totally okay to say, �I don�t know the answer to that, but here is how I�d find out. Also, would you mind if I took a minute or two to take a guess at the answer and try to reason it out?�

This process demonstrates that you have a analytical mind even if you don�t know an answer and it shows you are comfortable with not knowing.

With 150+ service offerings these days, it is impossible to know everything about everything, and we need you to be comfortable with that ignorance.

Also: get your sleep and eat well the day before and have stuff to snack on during your on-site loop. The five hours [or whatever it is these days] can be a grueling affair and you are best to be well-rested and well nourished.

# ? Aug 26, 2019 22:01

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Amazon Web Services - Cloud Giant Hits Hard

«‹›61 »