Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
JHVH-1
Jun 28, 2002

Pollyanna posted:

Our AMI bake times have ballooned to 40-50 minutes, and I really want to dive and debug why that's happening. I've tried reaching out to my coworker that's in charge of the pipeline/AWS stuff, but he's unhelpful and reluctant to walk me through the process, and what we're doing and why. I want to just bypass him and do some of my own digging to figure out how to reduce the amount of time it takes to bake. What're the common reasons why baking an AMI might take so long? Something about the files involved to do so? Is there a way to debug/step through the process?

I never got an answer re: why we're baking an AMI for each new commit to a branch, besides "that's the commonly accepted pattern". I get that it's technically correct, but it's also bullshit slow, and I question whether or not it's worth it given that we commit early and often and therefore deploy to ticket-specific servers early and often, and this time fuckin' adds up man. We're behind schedule as-is and this process is making it so much worse.

If you go into the dashboard you can see the actual process of volumes being snapshotted to create the AMI. That is usually what takes the time up. Volume snapshots are supposed to be deltas, so if you snapshot the same volume it shouldn't take as long after repeated imaging assuming you are doing it from the same instance each time. If a new instance is spun up each time and you keep adding more data then that would add to the time.

Oh also the type of volumes you are attaching could impact it as well, so if you aren't using EBS backed instances it would take longer.

Adbot
ADBOT LOVES YOU

JHVH-1
Jun 28, 2002

jiffypop45 posted:

Is that documented somewhere? I didn't see it. That definitely makes sense though.

Edit:

Found it

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/t2-instances.html

Its actually not cutting down that much. I'll just let it run as is for a bit and see if anything interesting happens from it.

Was going to point out, but I see it is mentioned on that page as well that you can view CPUCreditUsage and CPUCreditBalance in CloudWatch.

JHVH-1
Jun 28, 2002

Vanadium posted:

My data is in redshift already and I don't wanna gently caress with the setup in general too much. I guess it's gonna be a cronjob on some random host to do the queries and post the results to S3 and the lambda then just verifies that everything went ok.

Is there a standard way to hack up ssh or dns so you can ssh to instances by instance id or stuff like that without having to do lookups yourself, by hand?

If you have elastic IPs attached you can use CNAMEs pointing to the Public DNS name. If it is internal it should still route it privately (Like requests inside your VPC)

JHVH-1
Jun 28, 2002

a hot gujju bhabhi posted:

I'm fairly new to AWS so I apologise for the super basic question, but what service(s) would I use if I wanted to make a website that could compile less into CSS for a user to download? I figure that I should to do this in a Node.js Lambda and then send the result to S3 and publish to an SNS to say that the download is ready, which my webpage can then react to. Am I on the right track?

I think you could do something with the new code pipeline type services, but lambda could be fine from the looks of it. I am thinking about using something similar myself (using our existing bamboo server to build/deploy instead).

A whole bunch of options here using different languages https://www.staticgen.com
Like here is a blog post example using hexo https://medium.com/@TedYav/using-hexo-and-aws-to-build-a-fast-massively-scalable-website-for-pennies-ea3c0f1115a
It has its own built in s3 publisher.

JHVH-1
Jun 28, 2002

jiffypop45 posted:

Correct me if I'm wrong but from my understanding pipelines sends packaged code to an ec2 host or autoscaling group. I don't think you can send it to an s3 bucket from there.

Yeah for the most part, but it sounds like you could use the pipeline to do the build and put the zip on s3 and then trigger deploying that to a public bucket with lambda. Kinda a round about way to replace having a dedicated build server or just pushing from a machine that has s3 access to the bucket where the site lives.

JHVH-1
Jun 28, 2002

Virigoth posted:

Ah ok I can see that. I'm having the political fight with Docker right now. We're getting ready to fix a problem with #3 that has been a big security bug for awhile so we'll see what that does to our multi-region deployments and time. For #2 I'm not a Chef guy but is there no way to setup your playbooks (We use ansible) so that when your service does the "configure" playbook you can just run a quick set of scripts or invoke something you baked on there? I'm looking at this from a amazon linux AMI perspective we bake on top of.

If I was you I wouldn't put any major cycles into Kubernetes until after reInvent. Like I'd go full stop if you were thinking of starting right now. It just seems like Kubernetes is ripe enough that AWS might pick it up for some sort of support.

Guess what, they announced it. Also containers without having to manage the hosts.

Also re:invent was pretty nuts. Next time I’m going to try and plan everything out earlier.

JHVH-1
Jun 28, 2002

a hot gujju bhabhi posted:

I have a question - over the Christmas break one of my coworkers decided to wind down some EC2 instances that are only used for testing etc (i.e. not needed outside business hours). This is fine for most of our stuff, but some of the instances he terminated were actually belonging to Elastic Beanstalk stacks. Since they've been started back up, they've been having funny behaviour (constant cycle of scaling up and down, CPU strain etc). In each case, rebuilding the environment fixed the problem. I'm just wondering three things:

1) Is manually terminating the instances definitely what caused this issue or am I suffering confirmation bias? Has anyone experienced this before?
2) Is there a simpler, lower impact way to fix this problem other than a complete environment rebuild?
3) Is there a recommended way to do what my coworker was trying to do (i.e. pause the environment's underlying instances while they were not required)?

My experience with EB has been that sometimes its an annoying black box and you don't know why its failing, so spinning up a replacement environment is easier.
Its really just there to prototype things and make it easier for developers, and not meant for a production environment. My last company we eventually got rid of it and just ran the same deployment code everywhere, we just scaled back the instances (smaller types, scheduled shutoffs, or spin up environments on demand to test and then destroy when done.

If you stick with EB then I think you have to manage it through the cli or its own dashboard. If you start screwing with the EC2 instances it is using it tends to get confused.

JHVH-1
Jun 28, 2002

Rapner posted:

So the security exam is back in beta - first sessions to sit are on Monday. https://aws.amazon.com/certification/beta-exam/

Free online training for it too. Couldn't hurt to go through that.

JHVH-1
Jun 28, 2002

a hot gujju bhabhi posted:

Is this really the case? I mean I can see that it would be the case in practice perhaps, if it's unreliable or a bit lovely or whatever. But is this truly AWS's intent for the service?

They have improved it like everything else. Originally they didn't sell it as something you run production applications on, but now it scales out etc. From an operations perspective I found it annoying so it was originally designed so developers can throw an app at it and not think about the rest. Problem is developers seem to get used to it and then expect it so its more fleshed out now than it used to be.

This is from their FAQ

quote:

Q: What is AWS Elastic Beanstalk?
AWS Elastic Beanstalk makes it even easier for developers to quickly deploy and manage applications in the AWS Cloud. Developers simply upload their application, and Elastic Beanstalk automatically handles the deployment details of capacity provisioning, load balancing, auto-scaling, and application health monitoring.

DEVELOPERS DEVELOPERS DEVELOPERS

JHVH-1
Jun 28, 2002
Seems like it would make more sense just to have IAM policies you can disable so access gets cut off. Then the content can get updated (if it even changes) before putting the policy back again before next semester.

Boto would work fine, and probably good to try as an exercise though. The CLI is just more straight forward is all, like you could probably do it all from a single line if its in the same bucket using the flags rm provides.

JHVH-1
Jun 28, 2002

Seventh Arrow posted:

Also, my exam is on Monday but I'm still fuzzy on encryption of volumes. How can I lock this down better?

I actually have been going through the training for the beta security exam since its free and they cover it in the second section of the course:

https://aws.amazon.com/training/course-descriptions/security-fundamentals/

Though you might not want to go through the whole class till after your exam and just find that part so you don't overload yourself with info. There is also a corresponding whitepaper: https://d0.awsstatic.com/whitepapers/AWS_Securing_Data_at_Rest_with_Encryption.pdf

AWS has whitepapers for like EVERYTHING it seems.

JHVH-1
Jun 28, 2002

SnatchRabbit posted:

Has anyone messed around with Lambda functions? My boss asked me to come up with a function to do some penetration testing on our instances, essentially, check to see whether a certain port on a certain instance is open. I'm thinking about using the API gateway and maybe a simple webpage front end that will run the lambda function but I'm open to ideas. I kind of suck with coding in python but this should be a pretty simple. Anyone have any suggestions?

Sounds pretty complicated, could just run nc or something. I had a script like this I wrote in python though a couple jobs ago to make sure a port was open so the app was running just using sockets I think.

Also a heads up, technically if you are doing penetration testing you are supposed to notify them https://aws.amazon.com/security/penetration-testing/
If you have your rules set up properly and only allow what you need to though you shouldn't be needing to poll this kind of thing frequently. Based on the network rules you should be able to tell if something is allowed or not. Just stumbled upon this while googling a second ago: https://nccgroup.github.io/Scout2/
You could also enable flow logs on a VPC: https://aws.amazon.com/blogs/security/how-to-optimize-and-visualize-your-security-groups/

JHVH-1
Jun 28, 2002

Less Fat Luke posted:

Transmit is great (especially if you can expense it).

Does it let you set encryption options these days? I think I was using it maybe 3 years ago before I started my last job, but I found I couldn't set the default encryption options for s3.
Like now you can at least have bucket policies and force it, but at the time it was annoying. Cyberduck was the only client I found that had this as an option so I stuck with it. Plus it can generate temporary signed URLs easily.
If I am just syncing a directory even I use cyberduck to grab the s3:// url and then jump on command like and use 'aws s3 sync'

JHVH-1
Jun 28, 2002

Thanks Ants posted:

The region thing in AWS being a global setting is fairly annoying - I’d like to see all instances returned and then a column for region that can be filtered.

I assume there’s a good technical reason for this and presumably it ensures that each region is separated from another so you don’t have issues with your local region meaning you also lose management of other regions but I’ve not read anything that explains why it’s the way it is.

You can't move stuff between regions though, and then you would have to pick region every time you create anything. If it just uses a default you would have to remember to switch it or re-create things all the time.

JHVH-1
Jun 28, 2002
You can trigger based on scheduled events with lambda https://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html
Lambda has other triggering options as well there is SNS for messaging, SQS for queuing, S3 can be used as hotfolder.

Also didn't know this was a thing till now https://aws.amazon.com/batch/

JHVH-1
Jun 28, 2002

very stable genius posted:

Lambda has a short maximum execution time and since he mentioned using compute optimized instances I’m guessing that won’t work. Have you looked at spot instances though?

Edit - also look at using SNS for your messaging. If lambda will work for you, it’s dead simple to trigger lambda functions off of an SNS topic.

Well Lambda can be used to trigger a job, like run something in fargate and then it does its thing and generates what you need and then exits so nothing is left running.

Example: https://serverless.com/blog/serverless-application-for-long-running-process-fargate-lambda/

JHVH-1
Jun 28, 2002
I'm a devop you nerds.

JHVH-1
Jun 28, 2002

Orkiec posted:

I'd rather not talk in too much detail about this for obvious reasons :), but Aurora Postgres is absolutely a Postgres fork and not a MariaDB one (https://www.youtube.com/watch?v=nd_BT_H-vsM). What Aurora Postgres and Aurora MySQL do share a lot of is the storage layer, which is why they are sharing the same name.

I was excited with I found Aurora MySQL added smaller instance types so I could give it a try. I had set up a dev environment for a project and got it working. Then the devs come back to me and decide they want Postgres. So I thought I would just spin up a replacement but I find out the instance types didn’t exist yet there. Was quite a bummer and ended up just installing it on ec2 for them.

JHVH-1
Jun 28, 2002

Thanks Ants posted:

is there a best-practices anywhere for using SAML with AWS Cognito as well as the AWS control panel? Presumably I just create one app for Cognito and one for the other stuff, or is there a more elegant way to deal with this?

I was just starting to set this up on Friday. Added our Azure AD as identity provider. It basically uses assumed roles to give access to the dashboard or programmatic access. So you can map groups to the role to give access to whatever.

Currently logging in is initiated from our microsoft stuff, I haven't gotten around to initiating it from AWS. It doesn't actually create the users either, as its assuming a role. The login gets tracked in cloudtrail though.

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml.html I started going through this on my end and the admin of the Azure side had the typical microsoft instructions that tell you what to click where and what to type blah blah blah.

JHVH-1
Jun 28, 2002

very stable genius posted:

CloudFormation is actually garbage and the last page of this thread has been hilarious watching people defend it.

Wow you really convinced me not to use CloudFormation by saying "its garbage"

JHVH-1
Jun 28, 2002

Thanks Ants posted:

The application is supplied as an ISO or an OVA, and upgrades involve mounting the ISO and using the (virtual) console. While I could deploy the thing into Hyper-V and then migrate it into Azure it basically paints me into a corner as far as future upgrades go.

This process doesn't seem too convoluted in AWS if you have an OVA:
https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#import-image-prereqs

JHVH-1
Jun 28, 2002

MJP posted:

How useful is the AWS Certified Cloud Practitioner, and in the future, AWS Certified Sysops Administrator, to someone whose career focus is Windows infrastructure support/administration? My present position is secure but I'd like to at least be prepared for a meteor destroying the entire industry where I work (terrestrial radio D: ). It seems like I'd be better served with Office 365 but a lot of job listings mention that AWS familiarity is a good thing. I have never touched AWS, I am not a developer nor do I wish to join them, and I'd only want to be able to understand how an AWS environment would work alongside a Windows one.

AWS has a special lounge you can hang out in at their events if you are certified. Last years re:Invent they had arcade machines set up in it to match the RPG role based certification theme.

JHVH-1
Jun 28, 2002

jiffypop45 posted:

If your company is paying for that. Apparently tickets are in the thousands. I had no idea it was so expensive.

Yeah it seems kinda pricey but it does include the whole week of event, free meals if you stick to their schedule, multiple free parties with booze. Plus any extra vendor parties you get into.

Honestly Vegas is tiring after 3 days though so I don’t plan on going again this year. We are going to try and do Ansiblefest in Austin instead cause I’d probably learn more and it’s only 2 days.

There are one day expos though at various cities that are completely free. It only costs money if you do the certification readiness courses or the other various boot camp stuff they have the day before.

If you are new then AWS though the sessions at those events will help learn a fair bit. The hands on ones are fun to, like where you use lambda to make an alexa skill and whatnot.

JHVH-1
Jun 28, 2002



Votlook posted:

What is a good way to manage ssh access to ec2 servers?
We currently have the public keys of the dev team baked into all our AMI's, but this creates a lot of work whenever a new team member leaves or joins, as we have to
rebuilt all the ami's and update all our services.
We have tried using bastion server, but it complicates some of out tooling.
I'm looking for a solution that where it is easy to add and remove access to each machine on-the-fly; preferably something really simple and robust.


I don’t even like having devs on machines if I can help it. It kind of encourages the whole keeping servers as pets instead of cattle thing and makes things harder later on when you have to figure out what random config change some dev did that isn’t there in your automation.

I’ve contemplated setting up a directory service and using that. Right now we started using ansible as a quick and dirty way. You could probably do something with the ec2 config service as well.

This was one of the things I liked about OpsWorks was it could generate temporary keys for you to get in and do some work. The privs were by AWS users and on a per stack basis so easy to dole out.

JHVH-1
Jun 28, 2002
If you attend one of the AWS Summits you can often snag some credits to add to an account and combine it with free tier for 1 year.

I don't think DynamoDB costs as much as that Azure offering. Lambda (the serverless option) has quite a high threshold before it costs anything but you still pay for things like data storage if you need it.
Mainly just have to check the pricing either per each service or use their pricing calculator to add things up. I think you can also set up alarms for pricing to let you know when you might hit a personal limit.

For my personal site I originally had EC2 with a local mysql db, but it kept running out of memory. It was double the cost to move to managed service so I just ended up getting a dedicated VPC from digital ocean for flat $5 month.

JHVH-1
Jun 28, 2002

Wish this was out for PostgresSQL. Those workloads end up being more expensive and aurora version has a higher minimum instance type.

Might try it out for something else though.

JHVH-1
Jun 28, 2002
They have online training, some of which is free as well. Also a ton of their seminars on different tech are archived on youtube.

JHVH-1
Jun 28, 2002
Anyone mentioned VPC peering yet? https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html

JHVH-1
Jun 28, 2002

SnatchRabbit posted:

Athena sounds like it might be a good fit, but alternatively you could run a managed Postgres database in RDS and query that with say Lambda using Python, although, the timeout on Lambda queries are five minutes so you might need to break up the operations you're doing. Lambda might be a nice fit because assuming the queries run in a reasonable timeframe you could write the results directly to S3 or dynamodb using the boto3 library in Python.

FYI they increased the lambda timeouts not too long ago, and it’s 30 minutes now.

I haven’t used it yet, but step functions is kinda cool for some of this stuff. It lets you mix different services, even human interaction on into a process flow.

JHVH-1
Jun 28, 2002

Doh004 posted:

Hey thread, I host MY FIANCE'S WordPress blog right now on an EC2 instance that I configured myself using Centos and NGINX (what I was already familiar with). I did this primarily to start teaching myself more about AWS and I'm liking it so far. I'm still on my 12 month free tier for a lot of things, but I had to upgrade the EC2 instance to something with more memory (kept running out of memory when dealing with a lot of her assets and plugins).

Right now, it's about ~$18 a month in charges to run her site, which is absolutely A-OK with me and well worth it, money isn't the issue. But, I just saw Lightsail and it looks like I could be running her site on a Lightsail instance for ~$10 a month.

Would it be worth migrating it over? I have the domain hosted on Google Domains (before I was over on AWS and had done it with Route53) and it's using a Let's Encrypt SSL cert, and I'm hosting all of her assets on S3 on a Cloudfront distribution. Would it end up costing about the same for all of that added up?

I of course could just do this myself now, but I'd rather not have to migrate her site if I don't need to.

Wordpress main issue is the database and its memory usage. I used to run my personal site on EC2 and then come back a week later and realize it ran out of memory and killed mysql. Then I added a swap file and it seemed ok for the most part but still was slow and occasionally had issues.

I ended up switching to a $5/mo digital ocean instance which worked out better. Lightsail didn't exist then, but its the same idea... more of a dedicated virtual server than elastic. Wordpress runs fine in AWS if you just use RDS for the DB.

Thinking about it, another option now is Aurora serverless. If that cost is low enough you could stick with a small t3 instances in EC2.

JHVH-1
Jun 28, 2002
Maybe I’m wrong but I think the bucket might not need have to have the same name as the domain if cloudfront is there. For static sites with http it does cause the s3 endpoint forwards it but for cloudfront it just uses the bucket as the origin.

Well it can’t hurt anyway to keep it matching just for organization sake.

Also for the cloudfront distro you can often start using it right away while it deploys to all their endpoints, but every time you tweak settings it will update again anyway. Might as well get it all right and then wait till it no longer says InProgress.

JHVH-1
Jun 28, 2002

Scrapez posted:

Is there a way from the command line on an EC2 instance to retrieve just the public IP address associated with that instance based on the private IP?

I can do `aws ec2 --region us-east-1 describe-addresses` which returns a list of all addresses and I could parse out the PublicIP of the instance I'm looking for with a combination of grep and awk but is there a better way of doing this?

I would be putting the private IP in a variable as I can obtain that via ifconfig and then I'd like to return the public IP based on the private IP.

I'm writing a bootstrap script that will update a config file on the instance with the public IP of that instance. Thoughts?

Also be aware if you use public-hostname it will resolve to the public IP or private IP depending on where it is resolved. That way you can do things like route the traffic internally for some systems with the same hostname.

JHVH-1
Jun 28, 2002
Can you manually run the SSM command to make sure it works? You have the instances set up with the agent and everything right? (Depending on what your base image is I think there is a chance its not installed already).

JHVH-1
Jun 28, 2002

Scrapez posted:

What is the best method for triggering an autoscaling event based on output into a log file on the EC2 instances in the group? Use case is a SIP platform and I'd like to be able to trigger a scale out event when number of calls on any given instance reaches X.

You can create a custom cloudwatch metric and then use it as your scaling criteria

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html

JHVH-1
Jun 28, 2002

Scrapez posted:

Thank you. That is exactly what I was looking for but google searches had not gotten me to that.

Probably have to play around with the alarms and getting the right metrics so you have both scale up and scale down criteria based on something that covers the whole cluster.
My last company we had a developer that was populating a metric in their code and never thought about creating a scale down one, so the thing would get busy or a bug would scale it out like crazy and then never reduce it.

JHVH-1
Jun 28, 2002
You can do some tweaking based on time at least. I've done that before reducing the capacity over the weekends when I knew it wouldn't be that important.

At least you can make the metrics and create alarms and see what they do before making them the scaling criteria. Like if you know you have 0 calls during long periods on a regular basis you could set an alarm to scale down then or something.

JHVH-1
Jun 28, 2002
Ya know I just remembered, I think you can also send your logs to cloudwatch and then create metric filters based on that.
The newer cloudwatch agent also collects some other stats for you too. I haven't yet played with it, just the older cloudwatch logs one https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html

JHVH-1
Jun 28, 2002
I use cyberduck on the mac, but no idea if its any good on windows. Also versioned buckets confuse the hell out of it if you delete something so stuff still shows up because of the way s3 handles "directories"

Cloudberry Explorer is supposed to be pretty good. Sometimes I wish there was an easy web based front end that I could hand out creds for and not have to manage keys or have people go on the dashboard. We sometimes have the problem that people were sharing keys. The last time I got a request related to that I ended up adding a role we mapped to an AD group and let them log in via an application in office.com. That way they just click an icon and their role only allows the s3 bucket they have permissions to access to in the aws dashboard.

JHVH-1
Jun 28, 2002
Though elasticwolf sounded interesting but the last release was in 2017. It probably does basic stuff fine. The problem I had a few years ago when evaluating s3 clients is that so little of them at the time supported modern s3 features like server side encryption. Though now you can set buckets to force that by default, so it wouldn’t matter.

Adbot
ADBOT LOVES YOU

JHVH-1
Jun 28, 2002

Docjowles posted:

This isn't really anything related to AWS. The requirements.txt file doesn't just magically do anything on its own. You need to do something like "pip install -r requirements.txt" first to actually download and install the dependencies. Then your app should work.

I think it does install the packages if you format it correctly and put it in your application bundle:
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html

Also says if it fails validation it won't work and you have to check the logs for that as it shows up as a warning.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply