Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Thanks Ants
May 21, 2004

#essereFerrari


I've done SQL installed on a VM before where it was hosting a really badly written app that the developer weren't interested in making any changes to in order for it to be compatible with :yaycloud: database products. If it's something being developed in-house then you need to make it in a way that is compatible with RDS etc.

Adbot
ADBOT LOVES YOU

22 Eargesplitten
Oct 10, 2010



Just-In-Timeberlake posted:

Go to CloudFront > Functions

Create a new function:

code:
function handler(event) {
    var request = event.request;
    var uri = request.uri;
    
    // Check whether the URI is missing a file name.
    if (uri.endsWith('/')) {
        request.uri += 'index.html';
    } 
    // Check whether the URI is missing a file extension.
    else if (!uri.includes('.')) {
        request.uri += '/index.html';
    }

    return request;
}

For some reason this didn't work, but I think I found something.

While using the website origin you can't use OAI but you can add a custom header. I added a referer header and an S3 bucket policy to only allow access with that header, I'll put in a massive randomly generated key to effectively lock down access to only the distribution. Does anyone see a flaw in this? Normally the referer header specifies the website it's coming from and can be easily spoofed but in this case it seems like it should have a similar effect to requiring an SSH key or something. Right?

Internet Explorer
Jun 1, 2005





There's a bunch of off the shelf software companies that won't work with / support anything other than a database on a VM.

Docjowles
Apr 9, 2009

It's also cheaper to run bare EC2 instances than run RDS. But this argument falls apart pretty quickly if you put any value at all on your own time spent managing those instances, because operating databases loving sucks.

And while it's true that certain things work differently or are disallowed in RDS, the vast majority of apps will be just fine. As you branch out from standard mysql/postgres/mssql in RDS to Aurora, or Aurora serverless, etc. then the number of caveats grow.

Docjowles fucked around with this message at 21:31 on Jun 14, 2022

Happiness Commando
Feb 1, 2002
$$ joy at gunpoint $$

If you're a megacorp with a sweet private licensing agreement with Microsoft

The Fool
Oct 16, 2003


Happiness Commando posted:

If you're a megacorp with a sweet private licensing agreement with Microsoft

We're ending SA on a bunch of stuff and teams are scrambling to migrate from MSSQL run on-prem or on azure vm's to cosmos or azure sql


It's fun to watch, super glad I'm not in the trenches for any of those teams

Thanks Ants
May 21, 2004

#essereFerrari


Ending SA shouldn't automatically result in having to scramble to do anything, it just means they have to hold on the release they're running

kalel
Jun 19, 2012

system administration? software automation?

...something awful?

Internet Explorer
Jun 1, 2005





yes.

(software assurance)

Agrikk
Oct 17, 2003

Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

kalel posted:

Out of curiosity, would there ever be a reason to use an EC2 with a database image instead of RDS? I have some microservices inside fargate tasks which connect to a MySQL RDS (or at least, I'm trying to and am currently in the process of debugging). My understanding is that the typical industry standard way to manage a database is through RDS for scalability and convenience, but is there ever a motivation not to do that?

There are plenty of reasons, most of them edge cases.

As mentioned above, the most common reason is regarding RDS MSSQL and it's cost per hour of operation (but like mentioned upthread, the cost is purely for it's operation and does not take into account maintenance operations, cost of management, etc). Backup/restore in the classic sense is wonky if you don't use native AWS capabilities, and using special procedures and functions can be tricky. For RDS you still need an EC2 instance if you are doing any kind of ETL (if you aren't using native AWS ETL services and SSIS isn't available as a SaaS offering).

But mostly it's that some folks can't give up control and delight/need to have access "under the hood". These same folks still name their servers after planets/cartoon characters/SciFi references and have obsessively nursed them along for years.

I have two MSSQL database clusters running on EC2 and they run great and are very performant (especially with FSx ONTAP!). But I also have a couple of Aurora MySQL and Aurora Postgres instances that I haven't touched in ages and couldn't imagine running those workloads on an EC2 instance.

vanity slug
Jul 20, 2010

In our case (also MSSQL) we needed to fine tune storage performance without breaking the bank. Tempdb on ephemeral storage, databases on their own disks because their performance requirements were predictable, stuff like that. And at the time AOAG wasn't properly supported on RDS either.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
There's plenty of other reasons usually having to do with legacy or wanting to embed the database under full control of an application's database (you're creating your own janky version of RDS, for example). When we talk about microservices it really just means "take care of your own hard dependencies like databases and keep your surface area judiciously small." Things start to get awkward when you provision a new database as part of your application v an application is started and presumed to have a database already provisioned for it because these are very different workflows with vastly different sets of requirements and permissions for said application.

IMO, if one is cost sensitive much at all, has relatively small scale (a scale where people can get by doing classic sysadmin and still be business viable), growth is not super duper fast, and doesn't have to run stuff in AWS due to customer requirements sticking with a colo and being careful about which cloud services to use is plenty good enough. At previous companies that sucked donkey balls at scale and didn't really need to the systems we always had the least trouble with in terms of time spent in meetings were our bare metal hosted machines. Of course they couldn't scale anything up or down, but we blew so much time on cloud services like AWS and Azure that we kept having political problems over in our dysfunctions that it sunk the company and wasted more money for no good reason, which only hastens the inevitable demise of the company while enriching the hyperscalers.

Absolutely go with Big Cloud if you know you need them though and have the budget to afford several people like the pros in the thread at $250k+ total comp in the US. Most businesses doing cloud anything are basically doing it for FOMO and / or can't retain people without having cloud in their job descriptions. I say this as a former hiring manager some years ago.

Agrikk
Oct 17, 2003

Take care with that! We have not fully ascertained its function, and the ticking is accelerating.
“I have three domain controllers sitting in three regions on AWS. Two database servers and a file server in my corp VPC. I have four web servers and two app servers (both in fixed pools )and an RDS MSSQL instance in My Prod VPC. How come it’s sooo expensive? The cloud was supposed to SaVe Me mOnEy!”

BaseballPCHiker
Jan 16, 2006

Agrikk posted:

“I have three domain controllers sitting in three regions on AWS. Two database servers and a file server in my corp VPC. I have four web servers and two app servers (both in fixed pools )and an RDS MSSQL instance in My Prod VPC. How come it’s sooo expensive? The cloud was supposed to SaVe Me mOnEy!”

Oh god this is my company to a T. I cant wait to hit a year and bounce so I dont have to pay back my sign on bonus.

Why we as a 99% windows shop decided to go with AWS instead of Azure, and then just do a lift of shift of everything is beyond me.

Docjowles
Apr 9, 2009

BaseballPCHiker posted:

Oh god this is my company to a T. I cant wait to hit a year and bounce so I dont have to pay back my sign on bonus.

Why we as a 99% windows shop decided to go with AWS instead of Azure, and then just do a lift of shift of everything is beyond me.

Management: it’s ok we just need to get the move done asap so we can terminate our data center lease and avoid paying for the next hardware refresh. We’ll optimize later

Also management: what the gently caress do you mean you want another 9 months to rewrite the app to use s3 and containers vs oversized EC2 instances with a petabyte of high iops EFS volumes. We just spent a year moving to the cloud it’s fine

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
Mods, please enforce trigger warnings kthx

Scrapez
Feb 27, 2004

Docjowles posted:

Management: it’s ok we just need to get the move done asap so we can terminate our data center lease and avoid paying for the next hardware refresh. receive our bonuses for moving to the cloud. We’ll optimize later

Also management: what the gently caress do you mean you want another 9 months to rewrite the app to use s3 and containers vs oversized EC2 instances with a petabyte of high iops EFS volumes. We just spent a year moving to the cloud it’s fine

My yearly metrics just said A cloud...not WHICH cloud.

deedee megadoodoo
Sep 28, 2000
Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.


welcome to the oracle cloud

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug
Reasonably new to cloud native design stuff - wondering about a proposed setup.

We have a tool that's going to be used sporadically (during certain types of events only) - we wanted to design it as a webapp as a reasonably straightforward DDB backend, with the frontend basically delivered as a static file that calls to various lambdas for dynamic content, using Flask on Lambda to handle calls to the DDB backend for reads/writes/etc.

Is this setup going to be able to deliver anything near acceptable performance, or is the startup/ephemeral nature of lambdas going to be a problem? My fallback plan is going to Fargate or something for a container, or otherwise just setting up some sort of containerized server.

Falcon2001 fucked around with this message at 19:17 on Jun 22, 2022

12 rats tied together
Sep 7, 2006

it's hard to say without benchmarking, especially if your flask app does a lot of lazy loading. my intuition is that most of your fears can be allayed by lambda's provisioned concurrency feature, which basically prewarms a bunch of executors for you.

since you'd be running in flask on lambda, it shouldn't be too complicated to switch towards flask in a container later, if you find that it's not working

Agrikk
Oct 17, 2003

Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

Falcon2001 posted:

We have a tool that's going to be used sporadically

Sometimes the obvious solution can also be the cheapest in the right circumstances. If you have times when the tool is active and times when it isn’t used at all, you can save a lot of money and complexity by building a container or even an EC2 instance that points to your back end.

Then write a script that turns everything on when you need it, and another script for turning everything off when you are done.

Of course if your utilization is not zero during the downtimes then this won’t work, obviously.

crazypenguin
Mar 9, 2005
nothing witty here, move along

Falcon2001 posted:

Is this setup going to be able to deliver anything near acceptable performance, or is the startup/ephemeral nature of lambdas going to be a problem?

It will likely work fine. I’ve built plenty of internal apps on lambda.

You havent defined what acceptable performance means, but a python cold start should be short. Well under a second. (idk about Flask, but the answer is just: try it)

Once a lambda instance is warmed, it sticks around for awhile and serves quickly

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug

Agrikk posted:

Sometimes the obvious solution can also be the cheapest in the right circumstances. If you have times when the tool is active and times when it isn’t used at all, you can save a lot of money and complexity by building a container or even an EC2 instance that points to your back end.

Then write a script that turns everything on when you need it, and another script for turning everything off when you are done.

Of course if your utilization is not zero during the downtimes then this won’t work, obviously.

In this case, the problem would be that it's a service that needs to work very quickly when we need it to - basically for realtime response stuff, otherwise I'd agree that's a pretty good approach.

For the others, it sounds like the idea is at least sane enough to get up to the 'testing' phase. 'Acceptable performance' mostly meant 'Is the latency going to be high enough that a user would find the delay irritating' and it doesn't sound like there's a significant problem here.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

crazypenguin posted:

It will likely work fine. I’ve built plenty of internal apps on lambda.

You havent defined what acceptable performance means, but a python cold start should be short. Well under a second. (idk about Flask, but the answer is just: try it)

Once a lambda instance is warmed, it sticks around for awhile and serves quickly

Just schedule a keep warm function for every 4 minutes or so

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Falcon2001 posted:

In this case, the problem would be that it's a service that needs to work very quickly when we need it to - basically for realtime response stuff, otherwise I'd agree that's a pretty good approach.

For the others, it sounds like the idea is at least sane enough to get up to the 'testing' phase. 'Acceptable performance' mostly meant 'Is the latency going to be high enough that a user would find the delay irritating' and it doesn't sound like there's a significant problem here.

I just tested a Django site served by Lambda that’s got good backend features (DRF, some analytics) and a cheap RDS instance behind it but almost no content on the page I loaded. GTmetrix gave it a 100% A on us-east-1 from Vancouver. So Lambda will not be what makes it slow.

MightyBigMinus
Jan 26, 2020

its not engineering if you're not using numbers

"sporadic" doesn't mean anything. neither does "acceptable".

kalel
Jun 19, 2012

accessing my load balancer's DNS yields a 503 service unavailable error. I have an ECS cluster, my fargate containers are "running," I have health checks defined... so why is my target group not allowing me to register any targets. why are there no available instances in the resource list. wtf is going on fuuuck

kalel
Jun 19, 2012

aws sucks, dev ops sucks. Lol

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug

MightyBigMinus posted:

its not engineering if you're not using numbers

"sporadic" doesn't mean anything. neither does "acceptable".

This is totally fair, but I'm also describing things that I don't have clear measurements on.

This is a tool used during response to certain types of incidents; based on past experience I expect to use it 6-12 times a year for a few days at a time, but you can't predict incidents, so I'm trying to make sure it's at least somewhat scalable.

By 'acceptable', I'm trying to ask 'is this going to deliver a level of responsiveness that humans won't find to be weirdly slow or stilted'; I don't have a lot of frontend experience so web user experience stuff is something I don't have a ton of experience on yet, so I'm not sure what the right term would be. CarForumPoster's response covers it though; basically I just wanted to check if my entire design was fundamentally flawed from the get-go, and it doesn't really sound like it.

Woodsy Owl
Oct 27, 2004

kalel posted:

aws sucks

Drake nah

kalel posted:

dev ops sucks. Lol

Drake yeah

22 Eargesplitten
Oct 10, 2010



I'm trying to get our ECS containers tagged so we can see what particular task definitions/services are costing us the most. The problem is that when the services were set up while they had AWS-managed tags enabled they weren't set to make the tasks inherit tags from the service or task definition. Is there a way to do this after the fact?

E: Found it, I can do it through the CLI. Except now I have to wait for our security guy to give me access to create an access key for myself.

22 Eargesplitten fucked around with this message at 20:21 on Jun 29, 2022

Startyde
Apr 19, 2007

come post with us, forever and ever and ever
Cloudshell sib, if your dash creds do it, you can do it. CLI’s installed, you also get a small per-region persistent store.

StumblyWumbly
Sep 12, 2007

Batmanticore!
Has anyone used Azure for IoT applications, like sensors uploading recordings? How is it, especially compared to AWS?

My team has some applications that mainly use AWS for managing our sensors. The company that owns my division (bought us a year after we started on our application) favors Microsoft and Azure for everything, for no particular reason. I'm worried that bringing everything under one umbrella will be a priority at one point, and I don't have any feel for how hard that will be, but my guess is a big pain.

jaegerx
Sep 10, 2012

Maybe this post will get me on your ignore list!


You need to elaborate more. Sensors like you’re reading camera data?

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
Uh... sensors mean a lot of different things across different industries. Should probably clarify that in terms of kinds of data collected, bandwidth per sensor, etc.

StumblyWumbly
Sep 12, 2007

Batmanticore!
Yeah, sorry, it's not video but it takes moderately high rate measurements, maybe 240-1000 kb/s, but the device is used for engineering tests, not continuous monitoring, so we upload recordings of a few seconds to a few minutes up to S3, process it into some characteristics we put into the database, and let the user download it if they want.

I feel like the big questions might be how good is Azures IoT support, and how much would everything need to restructure if we're DynamoDB based now and have to move to Cosmos or whatever Azure has.

Also, sorry if Azure is too off topic. I thought this was a general Web Service thread. It doesn't look like there's an Azure thread, so maybe that says something about Azure vs AWS?

The Fool
Oct 16, 2003


StumblyWumbly posted:

Yeah, sorry, it's not video but it takes moderately high rate measurements, maybe 240-1000 kb/s, but the device is used for engineering tests, not continuous monitoring, so we upload recordings of a few seconds to a few minutes up to S3, process it into some characteristics we put into the database, and let the user download it if they want.

I feel like the big questions might be how good is Azures IoT support, and how much would everything need to restructure if we're DynamoDB based now and have to move to Cosmos or whatever Azure has.

Also, sorry if Azure is too off topic. I thought this was a general Web Service thread. It doesn't look like there's an Azure thread, so maybe that says something about Azure vs AWS?

There's a bunch of Azure guys around, we also post in the other IT threads.

I'm not doing anything with Azures IoT stuff, but I have soooo much cosmos db in my environment.

22 Eargesplitten
Oct 10, 2010



I've been assigned to make some kind of short presentation and KB article for my new job about an AWS topic that would be useful for our developers. What topic do you all wish that developers understood better in AWS? We use a mix of EC2 VMs and Fargate EC2 containers, I'm thinking maybe something about how auto-scaling works so that if developers make something that causes a huge resource-intensive spike they can understand why everything moves slow for a little bit until autoscaling catches up.

The Fool
Oct 16, 2003


22 Eargesplitten posted:

I've been assigned to make some kind of short presentation and KB article for my new job about an AWS topic that would be useful for our developers. What topic do you all wish that developers understood better in AWS? We use a mix of EC2 VMs and Fargate EC2 containers, I'm thinking maybe something about how auto-scaling works so that if developers make something that causes a huge resource-intensive spike they can understand why everything moves slow for a little bit until autoscaling catches up.

the devs I support are all hilariously bad at secret management

Adbot
ADBOT LOVES YOU

Docjowles
Apr 9, 2009

For my money it’s some very basic info on how networking and/or dns work. I’ve had some absolutely :stare: conversations with senior devs where you suddenly realize that what they’re trying to explain only makes sense if their world view about how two computers communicate is totally and fundamentally broken.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply