Cloud Computing: Mostly fog machines in other people's datacenters

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Cloud Computing: Mostly fog machines in other people's datacenters

«‹›10 »

adorai: Nov 2, 2002; 10/27/04 Never forget; Grimey Drawer

flosofl posted:

I remember when Altavista was the best search engine and Yahoo was an actual index of sites grouped by category.

I also remember having a SLIP account to play on a MUD and download an 0.99 kernel version of Slackware to put on floppies. I'd get a floppy image via FTP to my SLIP account, use Z-Modem to pull it down to my computer and then get the next one. It took about a week.

When my ISP started offering PPP, it blew my mind.

oh yeah? i remember trying to download porn from a bbs via zmodem and being pissed when the girl had a swimsuit on.

# ? Dec 22, 2016 01:49

Adbot: ADBOT LOVES YOU

# ? May 5, 2024 13:22

YOLOsubmarine: Oct 19, 2004; When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

adorai posted:

oh yeah? i remember trying to download porn from a bbs via zmodem and being pissed when the girl had a swimsuit on.

I remember doing this and waiting ten minutes to be pissed because it took like twenty minutes to load the whole picture and it loaded from the top down. I had a Prodigy account.

# ? Dec 22, 2016 02:18

Docjowles: Apr 9, 2009

My main memory of Alta Vista was when my buddy in high school typoed it in the computer lab, ended up on a porn site with the lab manager directly over his shoulder, and got suspended over it :lol:

(He was constantly in trouble and so did not get the seemingly obvious benefit of the doubt in this case)

Also, yeah, the "20 minutes to download one lousy naked pic and it wasn't even good" struggle was real.

# ? Dec 22, 2016 02:27

Aunt Beth: Feb 24, 2006; Baby, you're ready!; Grimey Drawer

Docjowles posted:

My main memory of Alta Vista was when my buddy in high school typoed it in the computer lab, ended up on a porn site with the lab manager directly over his shoulder, and got suspended over it

(He was constantly in trouble and so did not get the seemingly obvious benefit of the doubt in this case)

Also, yeah, the "20 minutes to download one lousy naked pic and it wasn't even good" struggle was real.

This reminds me how in elementary and middle school we would routinely trick teachers and other students into going to whitehouse.com. :airquote:

Accidental :airquote:

porn!

# ? Dec 22, 2016 04:14

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

adorai posted:

oh yeah? i remember trying to download porn from a bbs via zmodem and being pissed when the girl had a swimsuit on.

The struggle was real. (I wonder if Sepist remembers America's Suggestion Box in Ronkonkoma.)

# ? Dec 23, 2016 04:57

Walked: Apr 14, 2003

Anyone dealt with Azure RM in depth?

I'm trying to figure out how to setup a template that deploys a two VM configuration (just IIS and SQL).

Where I'm hanging up is passing the SQL endpoint (IP or other means to resolve) back to the IIS VM at time of deployment.
The best I can come up with is an output on the SQL nic; and reference that IP value in the DSC extension parameters of the IIS server.

To me this seems really roundabout and not ideal. Is there something I'm missing? I really just want to be able to say "hey, your new SQL server VM is at this IP address", but without involving using the domain join extension as this is envisioned to be short lived/ephemeral environments that dont need to be tied into AD.

# ? Dec 28, 2016 16:36

Gucci Loafers: May 20, 2006; Ask yourself, do you really want to talk to pair of really nice gaudy shoes?

Walked posted:

Anyone dealt with Azure RM in depth?

I'm trying to figure out how to setup a template that deploys a two VM configuration (just IIS and SQL).

Where I'm hanging up is passing the SQL endpoint (IP or other means to resolve) back to the IIS VM at time of deployment.
The best I can come up with is an output on the SQL nic; and reference that IP value in the DSC extension parameters of the IIS server.

To me this seems really roundabout and not ideal. Is there something I'm missing? I really just want to be able to say "hey, your new SQL server VM is at this IP address", but without involving using the domain join extension as this is envisioned to be short lived/ephemeral environments that dont need to be tied into AD.

IPs aren't defined in the VM itself only on the virtualized NIC.

# ? Dec 29, 2016 08:35

Walked: Apr 14, 2003

Tab8715 posted:

IPs aren't defined in the VM itself only on the virtualized NIC.

Yes; I get that. I was looking for a method to pass information about the SQL Server instantiated in the same deployment to the other VM; IP was an example, but things that can be parameterized during the template deployment I'd also like to be able to pass to the new instance.

I ultimately solved this with CustomData, which is documented for Azure Classic, but basically documented nowhere for AzureRM.

what ended up working well:

code:

#...
        "osProfile": {
          "computerName": "[variables('vmName')]",
          "adminUsername": "[parameters('adminUsername')]",
          "adminPassword": "[parameters('adminPassword')]",
          "customData": "[base64('testingInfo')]"
        }

#...

Which can be read from inside the new VM:

code:

PS C:\AzureData\> $data = Get-Content C:\AzureData\CustomData.bin
PS C:\AzureData\> Write-Host $data
testingInfo

You can use output values from other resources rather than the testingInfo (oversimplified here intentionally), and essentially create a CSV which can be processed as desired on the VM with the injected info.

# ? Dec 29, 2016 15:02

Ashex: Jun 25, 2007; These pipes are cleeeean!!!

What log analysis tool are the cool kids using these days? Recently returned to my company after working as a consultant, been placed with a different cloud ops group that made some ....peculiar... decisions.

Right now I'm evaluating graylog for consuming cloudtrail (AWS shop) and I like that it's easy to setup but the aws plugin has some limits. I looked at ELK as I know of another group that is using it but its cloudtrail plugin has no documentation.

My pipe dream is to have all the apps push logs to cloudwatch and I'll just consume that with whatever I want. Graylog unfortunately doesn't support consuming cloudwatch anything.

# ? Jan 2, 2017 15:23

Less Fat Luke: May 23, 2003; Exciting Lemon

To me Splunk is the gold standard for that provided you can afford it. SumoLogic is a distant second (and much cheaper). I think also AWS has a CloudWatch for Logs offering now that's kind of like ELK, though I've never used it.

# ? Jan 2, 2017 16:49

Ashex: Jun 25, 2007; These pipes are cleeeean!!!

I've heard of the two but haven't touched them, I will check them out but I imagine a lower cost solution will be easier to get implemented (politically speaking). AWS has a managed ElasticSearch with kibana built-in, you can setup a CloudWatch Log group to stream directly into it via Lambda.

These guys are ridiculously cost conscious to the point where it's kinda hurting them, I've been reviewing events and one account gets 20,000+ API requests a day by some billing bot, there's no other activity.

# ? Jan 2, 2017 18:37

Internet Explorer: Jun 1, 2005

Splunk is great but their pricing is ridiculous. And their support has been really hit or miss.

# ? Jan 2, 2017 19:37

Docjowles: Apr 9, 2009

ELK is probably still the best free option in this space.

# ? Jan 2, 2017 21:57

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Docjowles posted:

ELK is probably still the best free option in this space.

Agreed, with the caveat that Graylog is a lot easier to get up and running with if you're starting off with syslog.

# ? Jan 2, 2017 22:16

good jovi: Dec 11, 2000; 'm pro-dickgirl, and I VOTE!

Watch out for AWS' managed ElasticSearch, it's a couple versions behind, so you'll have to use an older version of Kibana with it.

# ? Jan 2, 2017 22:37

Ashex: Jun 25, 2007; These pipes are cleeeean!!!

Vulture Culture posted:

Agreed, with the caveat that Graylog is a lot easier to get up and running with if you're starting off with syslog.

I'm really liking Graylog but it feels like the community around it is really small. I'm going to continue tinkering with it but I need to move off the docker environment I setup as graylog is doing something to cause dockers dns resolver to fall over every 30 minutes.

good jovi posted:

Watch out for AWS' managed ElasticSearch, it's a couple versions behind, so you'll have to use an older version of Kibana with it.

They finally released 2.3 support which has helped adoption but there's a few annoying design decisions such as no VPC support so you either do IP restriction or leave it open unless you want to figure out how to do signed requests.

Docjowles posted:

ELK is probably still the best free option in this space.

I may have to go this route later on but I'm not looking forward to securing Kibana due to the total lack of user management unless you pony up cash for ~~Shield~~ X-Pack. I just discovered search guard though so maybe I should look at this closer.

Ashex fucked around with this message at 02:33 on Jan 3, 2017

# ? Jan 3, 2017 02:10

Docjowles: Apr 9, 2009

Yeah that is annoying. One quick and dirty workaround is to just stick apache or nginx in front of Kibana and have it do basic auth for you. This seems less painful than dicking around with Elastic's auth mess.

# ? Jan 3, 2017 16:17

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

If you're in an enterprise house of horrors, you can use Shibboleth with Nginx to perform Single Sign On (specifically via FastCGI), too. https://github.com/nginx-shib/nginx-http-shibboleth The primary advantage of using something like Shibboleth is that you can potentially hand off authn/authz headaches to teams of people that eat it for breakfast while you can maintain your apps and services separately from that layer.

At a previous place in 2014, even the enterprise security offerings for an ELK stack wasn't good enough and we wound up deferring everything to AD / LDAP with ACL mappings. But of course, this was before they released X-Pack and such, but with how rudimentary the roadmap looked for enterprise-gently caress-cool-things shops it wasn't going to pan out.

# ? Jan 3, 2017 17:39

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Docjowles posted:

Yeah that is annoying. One quick and dirty workaround is to just stick apache or nginx in front of Kibana and have it do basic auth for you. This seems less painful than dicking around with Elastic's auth mess.

Been doing this for years because of the no-VPC mess on Amazon's hosted offering, works fine. There's an Nginx Lua mod to do OAuth2, if that's more your thing than HTTP Basic.

# ? Jan 3, 2017 18:33

firebeats: May 8, 2016

Does anyone here have any experience with Vultr, RootBSD or CloudSigma? I'm looking for a provider that does OpenBSD images, and those are the only 3 that I've found at the moment.

# ? Feb 5, 2017 21:52

netcat: Apr 29, 2008

We're having this strange issue where instances created in openstack only gets an IP assigned like 50% of the time. What makes it even more strange is that if I grep for ERROR printouts in the neutron logs I get errors printed when -it works-, but when it fails I see nothing at all. Anyone knows what might cause this or how I should go about troubleshooting? I have googled around for this but not found anything that seems to match my problem

e: I found this page and it seemed to have the solution. The neutron hosts files weren't set properly for whatever reason.

netcat fucked around with this message at 17:12 on Feb 13, 2017

# ? Feb 13, 2017 11:52

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

netcat posted:

We're having this strange issue where instances created in openstack only gets an IP assigned like 50% of the time. What makes it even more strange is that if I grep for ERROR printouts in the neutron logs I get errors printed when -it works-, but when it fails I see nothing at all. Anyone knows what might cause this or how I should go about troubleshooting? I have googled around for this but not found anything that seems to match my problem

e: I found this page and it seemed to have the solution. The neutron hosts files weren't set properly for whatever reason.

Glad you found it! I had this problem once and the issue was that we were running a multi-master Galera cluster, but there are parts of Nova that don't deal correctly with multiple active writers. We had to create a special active/passive load balancer frontend for MySQL connections that did writes.

# ? Feb 13, 2017 17:35

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

A while ago I think someone here wanted to know how to start AWS instances in a stopped state and while researching an unrelated issue found that you can launch an instance with userdata for the cloud-init boot script to start-up and immediately shutdown. You cannot start an instance without incurring some form of a cost in one way or another, but using something like an instance store (provided you don't mind the first-write penalty or are ok with using one of the more expensive SSDs that's pre-warmed to avoid the first 10+ min of initialization) you can minimize the cost of pre-baked and lukewarm AWS instances.

Not sure if anyone else's place has horrific habits of blowing crazy amounts of money on instances that sit doing nothing for the most part using the most expensive setups possible but I'm still amazed at how so many places can blow through $1MM / mo with maybe 10% utilization of resources and hardly blink even though they're typically cost center orgs.

# ? Feb 24, 2017 14:50

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

necrobobsledder posted:

A while ago I think someone here wanted to know how to start AWS instances in a stopped state and while researching an unrelated issue found that you can launch an instance with userdata for the cloud-init boot script to start-up and immediately shutdown. You cannot start an instance without incurring some form of a cost in one way or another, but using something like an instance store (provided you don't mind the first-write penalty or are ok with using one of the more expensive SSDs that's pre-warmed to avoid the first 10+ min of initialization) you can minimize the cost of pre-baked and lukewarm AWS instances.

That was probably me, and this was what we ended up doing (albeit with a Terraform provisioner rather than a cloud-init whatever).

# ? Feb 24, 2017 15:11

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Vulture Culture posted:

Glad you found it! I had this problem once and the issue was that we were running a multi-master Galera cluster, but there are parts of Nova that don't deal correctly with multiple active writers. We had to create a special active/passive load balancer frontend for MySQL connections that did writes.

It will be a glorious day when the last of the SELECT FOR UPDATE statements are finally killed, which I predict will happen sometime after the Z release.

# ? Feb 24, 2017 16:03

Docjowles: Apr 9, 2009

chutwig posted:

It will be a glorious day when the last of the SELECT FOR UPDATE statements are finally killed, which I predict will happen sometime after the Z release.

Sorry, that's been deprioritized in favor of "rename the networking stack yet again. but just do a s/neutron/butt and nothing deeper. so everything is still prefixed with q_ for no goddam reason. except 2 things, picked at random, which are still named incorrectly but in a different way"

# ? Feb 25, 2017 04:31

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Docjowles posted:

Sorry, that's been deprioritized in favor of "rename the networking stack yet again. but just do a s/neutron/butt and nothing deeper. so everything is still prefixed with q_ for no goddam reason. except 2 things, picked at random, which are still named incorrectly but in a different way"

This is a great suggestion. I'll refer it to the Software-Defined Variable Naming committee

# ? Feb 26, 2017 00:21

incoherent: Apr 24, 2004; 01010100011010000111001
00110100101101100011011
000110010101110010

Please regale me in your stories today of management asking you, personally, to contact Mr. Zon to ask what's up with the cloud.

e: classic

incoherent fucked around with this message at 03:28 on Mar 1, 2017

# ? Mar 1, 2017 02:45

Thanks Ants: May 21, 2004; #essereFerrari

That's great because it implies that a medical devices company doesn't have any sort of support contract.

# ? Mar 1, 2017 08:03

netcat: Apr 29, 2008

Is Heat kind of broken? I have a stack descriptor that's not very complex but heat fails to create the stack maybe 50% of the time because it can't assign a floating IP to one of the instances (but it -can- do it for the other instances on the same subnet which is bizarre) and it also 100% fails to delete the stack due to I'm guessing this bug https://bugs.launchpad.net/heat/+bug/1493894 that has been unassigned for two years.

I also have some autoscaling policies in place that fail to scale up/down because it can't find the scaling group, despite managing to create the first instance when the stack is created.

I'm not sure how to work around these problems either, except not using floating IP's which isn't really an option at the moment. My stack creates a network and connects it to the public net with a router, and two instances + 1 AutoScalingGroup. Servers are defined in separate yaml files, which I have to do if I want to assign floating IP when scaling up.

# ? Mar 15, 2017 16:42

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

netcat posted:

Is Heat kind of broken? I have a stack descriptor that's not very complex but heat fails to create the stack maybe 50% of the time because it can't assign a floating IP to one of the instances (but it -can- do it for the other instances on the same subnet which is bizarre) and it also 100% fails to delete the stack due to I'm guessing this bug https://bugs.launchpad.net/heat/+bug/1493894 that has been unassigned for two years.

I also have some autoscaling policies in place that fail to scale up/down because it can't find the scaling group, despite managing to create the first instance when the stack is created.

I'm not sure how to work around these problems either, except not using floating IP's which isn't really an option at the moment. My stack creates a network and connects it to the public net with a router, and two instances + 1 AutoScalingGroup. Servers are defined in separate yaml files, which I have to do if I want to assign floating IP when scaling up.

Everything in OpenStack is broken all of the time, even when it was working fine ten minutes ago. Sorry!

It sounds like the problem is that your cluster -- either Neutron or Nova -- falls over and has problems when you try assigning too many floating IPs in a short period of time. I'm not entirely sure where that problem lives, but the main problem doesn't sound like a Heat issue at all.

# ? Mar 15, 2017 17:51

netcat: Apr 29, 2008

Vulture Culture posted:

Everything in OpenStack is broken all of the time, even when it was working fine ten minutes ago. Sorry!

It sounds like the problem is that your cluster -- either Neutron or Nova -- falls over and has problems when you try assigning too many floating IPs in a short period of time. I'm not entirely sure where that problem lives, but the main problem doesn't sound like a Heat issue at all.

Nah I guess it's not really a Heat problem in the end, but it's weird since it works OK when assigning floating IP's if I don't have a nested stack (or at least it did before, I haven't tried in a while). I've actually had similar problems when creating servers and allocating floating IP's through a script, and then I did a retry mechanism but that doesn't really seem possible in this case.

Also my scaling problem is I probably something unrelated, but the examples work OK and I haven't really deviated a lot from them... Probably hard to say anything without seeing the files though.

# ? Mar 15, 2017 18:38

necrobobsledder: Mar 21, 2005; Lay down your soul to the gods rock 'n roll; Nap Ghost

Crap like that is why I'm loathing deploying a brand new bare metal production Openstack Newton environment. Given contracts that literally forbid us from putting customer data in AWS and we have hard dependencies upon appliances that are literally hardware-only so the best we can do is to manage some bare metal and deploy Kubernetes, Swarm, or... Openstack. In the end I'll be deploying Kubernetes on top of Openstack and letting Kubernetes figure out how to deal with the unreliable stack below. That'll take about two years I figure and be a good time to quit or die of alcohol poisoning.

# ? Mar 16, 2017 00:07

jaegerx: Sep 10, 2012; Maybe this post will get me on your ignore list!

necrobobsledder posted:

Crap like that is why I'm loathing deploying a brand new bare metal production Openstack Newton environment. Given contracts that literally forbid us from putting customer data in AWS and we have hard dependencies upon appliances that are literally hardware-only so the best we can do is to manage some bare metal and deploy Kubernetes, Swarm, or... Openstack. In the end I'll be deploying Kubernetes on top of Openstack and letting Kubernetes figure out how to deal with the unreliable stack below. That'll take about two years I figure and be a good time to quit or die of alcohol poisoning.

Coreos has openstack running in containers where it should be considering how often poo poo fails. You should look into that. Or just run k8s on top of coreos.

# ? Mar 16, 2017 01:37

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

netcat posted:

Nah I guess it's not really a Heat problem in the end, but it's weird since it works OK when assigning floating IP's if I don't have a nested stack (or at least it did before, I haven't tried in a while). I've actually had similar problems when creating servers and allocating floating IP's through a script, and then I did a retry mechanism but that doesn't really seem possible in this case.

Also my scaling problem is I probably something unrelated, but the examples work OK and I haven't really deviated a lot from them... Probably hard to say anything without seeing the files though.

If you aren't aggregating all your service logs to a single place, like Papertrail or Loggly or Splunk or Logstash, start there. (Make sure your clocks are properly synchronized over NTP or correlating those logs together between all the different systems/services is gonna be a wreck.) The logs do a pretty decent job of telling you what's wrong if you jack up the log levels.

necrobobsledder posted:

Crap like that is why I'm loathing deploying a brand new bare metal production Openstack Newton environment. Given contracts that literally forbid us from putting customer data in AWS and we have hard dependencies upon appliances that are literally hardware-only so the best we can do is to manage some bare metal and deploy Kubernetes, Swarm, or... Openstack. In the end I'll be deploying Kubernetes on top of Openstack and letting Kubernetes figure out how to deal with the unreliable stack below. That'll take about two years I figure and be a good time to quit or die of alcohol poisoning.

The problem with running OpenStack is that you have to already be really, really good at running systems at scale before you can use it to make running your other systems at scale easier.

Vulture Culture fucked around with this message at 05:22 on Mar 16, 2017

# ? Mar 16, 2017 05:20

Internet Explorer: Jun 1, 2005

Vulture Culture posted:

If you aren't aggregating all your service logs to a single place, like Papertrail or Loggly or Splunk or Logstash, start there.

Are those decent alternatives to Splunk? I like Splunk a lot, but the price is hard for a small company like mine to justify and Splunk Light is the worst redheaded step child of a product ever.

# ? Mar 16, 2017 05:31

Proteus Jones: Feb 28, 2013

Internet Explorer posted:

Are those decent alternatives to Splunk? I like Splunk a lot, but the price is hard for a small company like mine to justify and Splunk Light is the worst redheaded step child of a product ever.

ELK? (Elastisearch - Logstash - Kibana)

# ? Mar 16, 2017 05:33

jaegerx: Sep 10, 2012; Maybe this post will get me on your ignore list!

flosofl posted:

ELK? (Elastisearch - Logstash - Kibana)

This

# ? Mar 16, 2017 08:08

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Internet Explorer posted:

Are those decent alternatives to Splunk? I like Splunk a lot, but the price is hard for a small company like mine to justify and Splunk Light is the worst redheaded step child of a product ever.

Papertrail's plans start at $7/month if you don't have a whole ton of logs to ingest and search

flosofl posted:

ELK? (Elastisearch - Logstash - Kibana)

A great option if you already know how to administer Elasticsearch, sort of a bear to run competently otherwise

# ? Mar 16, 2017 16:00

Adbot: ADBOT LOVES YOU

# ? May 5, 2024 13:22

Internet Explorer: Jun 1, 2005

flosofl posted:

ELK? (Elastisearch - Logstash - Kibana)

Thanks for reminding me of this. I think I was looking at it but then put it down and got distracted.

Vulture Culture posted:

Papertrail's plans start at $7/month if you don't have a whole ton of logs to ingest and search

A great option if you already know how to administer Elasticsearch, sort of a bear to run competently otherwise

I'll check out Papertrail. I think my problem with ELK was what you described, a bit too much work to justify right now.

# ? Mar 16, 2017 16:35

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Cloud Computing: Mostly fog machines in other people's datacenters

«‹›10 »