|
Mr. Crow posted:The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices https://www.amazon.com/dp/B01BJ4V66M/ref=cm_sw_r_cp_apa_sQjfzbWBC57FV
|
#
?
May 26, 2017 06:13
|
|
|
|
| # ? May 14, 2024 08:48 |
|
|
necrobobsledder posted:It's depressing to see so many places that will have basically none of the above because they're culturally so far gone that even the idea of deploying faster makes leadership scared "why would we want to deploy faster? going slower means it's more reliable!" I see so much backlash against anything that fast-moving companies do while people working in really low performing enterprise shops act like their jobs deploying J2EE apps almost exactly like it's 1997 are so hard compared to the things modern software companies do. There are outliers I promise. Company I work for is 1400+ people globally with a main application that is a monolithic beast that's also nightmare to maintain. Yesterday morning a production bug was found, got fixed, artifact built, and deployed around 11am and users never noticed the deploy. Hell, the majority of my team members didn't even know a deploy happened. Of course it took 2+ years of working with devops and developers, pulling up my sleeves, and doing other people's jobs to get things to where they are now, but it's great. Change is slow in big companies. You just gotta find the key players and get buy in from the people who matter. Sometimes you gotta bust out the consultant hat and really sell it for people to budge. My experience is anecdotal, and I 100% agree with your assessment of leadership. That's all I've seen as well no matter where I go. I'll never work somewhere that refuses to make things better out of stubbornness after all the facts are on the table.
|
|
#
?
May 26, 2017 16:24
|
|
|
A buzzword that can make a big difference in trying to drive change is explaining the idea of "mean time to recovery" vs "mean time to failure". Most people have heard of the later and it's what drives most of these archaic companies, less have heard of the former and even old school kludgey bastards have been more malleable to change after focusing and explaining the idea. YMMV of course.
|
|
#
?
May 28, 2017 20:22
|
|
|
I have to disagree, mean time to recovery is very well understood even by old school curmudgeons because the big bureaucratic places have been getting familiar with disaster recovery scenarios where RPO and RTO figures start to matter because it helps determine how much money is at risk and what they're willing to spend on an insurance policy basically. Most of the old school operations management folks in the US come from backgrounds in military or manufacturing in terms of culture and primitives because these are among the most well studied academically. This isn't to say that these aren't applicable to tech in any fashion; Tim Cook's background is manufacturing and supply chain operations and logistics, after all.
|
|
#
?
May 28, 2017 21:23
|
|
|
We use the ELK stack for logging and I ran out of disk space for elasticsearch way sooner than I thought I would. How can I tell which log entries (i.e. by hostname or something) are consuming the most disk space?
|
|
#
?
Jun 2, 2017 01:32
|
|
|
fletcher posted:We use the ELK stack for logging and I ran out of disk space for elasticsearch way sooner than I thought I would. How can I tell which log entries (i.e. by hostname or something) are consuming the most disk space? Probably dumb question, but are you accounting for replicas? However many replica shards you use will multiply your index size, and could cause disk usage to appear crazy if you aren't expecting it. If it's not that, it might be easiest to examine the raw logs, if they're accessible. Which ones are the biggest per day? ES does have a _size field you can query on but it's not enabled by default. You could update your mappings with that and wait for the index to roll over to the next day, and see what shakes out. Failing all that, you can just make a quick visualization of log lines per day by hostname, which should be a decent proxy for size. Obviously that doesn't work if one log is spamming billions of massive Java stack traces and one is just a normal HTTP access log, but it's a start. I'm pretty aggressive about dropping any message fields I don't care about (like _all) to maximize my storage space. You can also tweak the index.codec setting, either globally or per index, to use slightly better compression. And finally, set up a cron with something like curator to drop indices older than N days.
|
|
#
?
Jun 2, 2017 02:06
|
|
|
fletcher posted:We use the ELK stack for logging and I ran out of disk space for elasticsearch way sooner than I thought I would. How can I tell which log entries (i.e. by hostname or something) are consuming the most disk space?
|
|
#
?
Jun 2, 2017 03:20
|
|
|
Vulture Culture posted:Can you post your index template for logstash-* and some better information about what kinds of logs you're ingesting and how? Docjowles' great points beside, you probably have a whole pile of duplicated data and a number of analyzed fields that don't need to be analyzed. Yeah analyzed fields will kill you, I just recently discovered that dates are analyzed by default after we had turned off all the strings. We always keep _all since we're using ES as our database lol
|
|
#
?
Jun 2, 2017 03:46
|
|
|
Does anyone here know anything about the Euromap xx communication protocols? I would like to pull basic information from a nestal injection machine. When the machine goes down and starts up. I haven't able to find any tutorials. I would like to setup an opcua server to pull the information. Could anyone point me in the right direction?
|
|
#
?
Jun 2, 2017 05:09
|
|
|
Thanks for all the replies guys! Much appreciated.Docjowles posted:Probably dumb question, but are you accounting for replicas? However many replica shards you use will multiply your index size, and could cause disk usage to appear crazy if you aren't expecting it. Since mine is only a single machine that hosts elasticsearch, logstash, and kibana I'm guessing the replica shards does not apply? Visualization of log lines per day by hostname helped narrow it down quite a bit. Apparently we had a bug in our CI environment that was logging millions of stack traces. So I think I should be in much better shape once I clean those up. Going to look into the compression and curator as well. I figured we'd need something like that eventually anyways. Vulture Culture posted:Can you post your index template for logstash-* and some better information about what kinds of logs you're ingesting and how? Docjowles' great points beside, you probably have a whole pile of duplicated data and a number of analyzed fields that don't need to be analyzed. This is gonna be a dumb question as I am a total newbie to the whole ELK stack but how do I see what my index template for logstash-* is? As for what kinds of logs I'm ingesting - it's pretty basic nginx & tomcat logs. Janitor Prime posted:Yeah analyzed fields will kill you, I just recently discovered that dates are analyzed by default after we had turned off all the strings. We always keep _all since we're using ES as our database lol What is meant by analyzed fields here?
|
|
#
?
Jun 7, 2017 02:54
|
|
|
I second that question! We are just starting with Elastic over here and any pointers toward "Here is how not to be dumb with Elastic" would be most helpful!
|
|
#
?
Jun 7, 2017 13:05
|
|
|
The String type in ES is analyzed by default, so that you can do full text searches on it. That's not necessary for a lot of strings so you have to manually specify in your index mappings that the fields you don't care about should not be analyzed. https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html
|
|
#
?
Jun 7, 2017 14:22
|
|
|
^ That guide "The ElasticSearch Definitive Guide" is actually a very good read. Even just the intro sections that describe fundamental ES concepts like node, index, cluster, document, shard etc. Regarding replicas, you might still have replica shards even if you only have one physical host involved. I think by default, ES will give you 2 replicas. You can check by doing a "curl localhost:9200/_cat/shards". A p in the third column indicates a primary shard. r indicates replica.
|
|
#
?
Jun 7, 2017 14:42
|
|
|
fletcher posted:This is gonna be a dumb question as I am a total newbie to the whole ELK stack but how do I see what my index template for logstash-* is?
|
|
#
?
Jun 7, 2017 15:14
|
|
|
I'm trying to set up a simple docker setup to learn (container my home fileserver's services). Just to be sure I'm understanding this right: you create a data volume and that's an overlay on the base state of the container that's persisted between runs of the app container? And you can then update the app container independently for point releases without nuking your data? What's the advantage of calling it a "data volume" vs just mounting a host dir in and calling it a day? More ability to scale when you move to a swarm/compose-type system? With a swarm/compose/whatever system, how do you handle the failure of something stateful like a database server where "spin up another" isn't an option? Replication across multiple nodes? One master instance? (Postgres's ability to unfuck itself from the WAL should be handy here) Is there a way for container apps to programmatically signal that they think a service is offline and vote for a reboot of a dead service container in a swarm-type system? For swarm-type setups, what is the best way to accomplish logging for the swarm in a centralized application for debugging/health monitoring? Paul MaudDib fucked around with this message at 02:54 on Jun 8, 2017 |
|
#
?
Jun 8, 2017 02:34
|
|
|
Paul MaudDib posted:I'm trying to set up a simple docker setup to learn (container my home fileserver's services). The answer to most of these questions is why Kubernetes exists. You can add cluster-level services for things like health, logging, etc. I'm not super familiar with what's available for swarm, but you'd probably be looking at deploying instances of your logging service in containers alongside your app containers, and have those feed data to another service (like Prometheus or ELK or w/e). Technically, mapping a local directory is just another type of data volume. The docker volume stuff has drivers for various other bits and bobs, like mounting shared storage directly into your containers. Mostly, creating a data volume allows you to somewhat easily share that volume with multiple separate containers.
|
|
#
?
Jun 8, 2017 03:42
|
|
|
Paul MaudDib posted:With a swarm/compose/whatever system, how do you handle the failure of something stateful like a database server where "spin up another" isn't an option? Replication across multiple nodes? One master instance? (Postgres's ability to unfuck itself from the WAL should be handy here) I've been told don't ever dockerize databases. I haven't explored too much into the why, but i'm just parroting what someone with way more docker experience has told me.
|
|
#
?
Jun 8, 2017 15:56
|
|
|
Paul MaudDib posted:With a swarm/compose/whatever system, how do you handle the failure of something stateful like a database server where "spin up another" isn't an option? Replication across multiple nodes? One master instance? (Postgres's ability to unfuck itself from the WAL should be handy here) poemdexter posted:I've been told don't ever dockerize databases. I haven't explored too much into the why, but i'm just parroting what someone with way more docker experience has told me. Surely it's fine to dockerize the database application, though? If you keep your data folder outside of the container, then the docker container running doesn't hold any long-term state, and stopping the container is equivalent to SIGKILLing a regular database process (which, at least for Postgres, would be a quite fine and safe thing to do).
|
|
#
?
Jun 8, 2017 17:32
|
|
|
NihilCredo posted:Surely it's fine to dockerize the database application, though? If you keep your data folder outside of the container, then the docker container running doesn't hold any long-term state, and stopping the container is equivalent to SIGKILLing a regular database process (which, at least for Postgres, would be a quite fine and safe thing to do). This seems to be the shortest way from A to B for small-scale setups in my (very novice) opinion. Have a postgres application container which points to a data directory which is mounted in from the host (eg /srv/postgres/). Unless there's pitfalls here with scalability or something? I'm not sure it's a huge practical gain versus just running it as a bare-metal service but at least this way you bring it inside the Docker system and can apply your logging/health monitoring layer or w/e over the top. But I can definitely see an argument that scale-on-demand database instantiation is probably more trouble than it's worth for small-scale applications and you are probably better off just having one DB per docker swarm or whatever. I'll take a look at Kubernetes too.
|
|
#
?
Jun 8, 2017 17:52
|
|
|
It's fine to Dockerize a database as long as you don't expect it to last a long, long time with its data or have a solid grasp on the data volume's lifecycle (see: Postgres K8s operator), but for most people in production with big ol' clusters and such that doesn't apply. I use Docker containers for launching temporary databases in CI builds and to compare / contrast different configuration settings for different use cases.
|
|
#
?
Jun 8, 2017 17:54
|
|
|
necrobobsledder posted:It's fine to Dockerize a database as long as you don't expect it to last a long, long time with its data or have a solid grasp on the data volume's lifecycle (see: Postgres K8s operator), but for most people in production with big ol' clusters and such that doesn't apply. I use Docker containers for launching temporary databases in CI builds and to compare / contrast different configuration settings for different use cases.
|
|
#
?
Jun 8, 2017 18:03
|
|
|
Vulture Culture posted:Also awesome for ephemeral test environments for developers. Static infrastructure is fine for prod and maybe staging, but gently caress it in the face everywhere else. I've been pushing heavily for this at work, making a little headway. I'm trying to untangle the classic knotted-up legacy Java webapp at work, we don't have good testing set up yet and there's so many moving parts that I don't know where to start with unit testing. My thought was to "record" HTTP requests and the request/session state each time we call a controller and assert the redirect and relevant state-changes as outputs. So in other words, if you send in a valid create request with parameters X Y Z then there should be an object created in the database with X Y Z, the object created should be attached to the request store as "SelectedRecord", and the request should have been routed to the "success" view. If it failed then you should have an error object attached with a message "field X was not filled". We have a "routing" controller that each request loops through before getting sent to a specific controller, I added a function there that hooks into a DB and dumps the request parameters, request attributes, and session attributes/etc into a PG database as JSON each time we pass through that controller, so we can generate this data on the fly from the Selenium testing we're (slowly) setting up. The advantage of doing it this way is you have a "unit test" for a controller that doesn't require standing up a whole webserver and firing off requests just to do a test (which is very slow), and by starting up an ephemeral test DB instance you should be able to have a bunch of devs working in parallel (or run your tests in parallel for faster build times).
|
|
#
?
Jun 8, 2017 18:52
|
|
|
Got some more ELK questions, trying to figure out the mapping stuff. When I curl http://localhost:9200/_mapping and pretty print it, it spits out 5MB of json, 143,000 lines in total. It seems like there is a separate mapping for each index, which I suppose makes sense. So I've got hundreds of keys in that json like "filebeat-2017.06.08" which I assume is because I have the following in my logstash.conf (I copied it from https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html):code:
|
|
#
?
Jun 8, 2017 20:11
|
|
|
You can update mappings on multiple indexes by specifying a wildcard. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#_multi_index
|
|
#
?
Jun 8, 2017 20:14
|
|
|
Paul MaudDib posted:This seems to be the shortest way from A to B for small-scale setups in my (very novice) opinion. Have a postgres application container which points to a data directory which is mounted in from the host (eg /srv/postgres/). Unless there's pitfalls here with scalability or something? I'm not sure it's a huge practical gain versus just running it as a bare-metal service but at least this way you bring it inside the Docker system and can apply your logging/health monitoring layer or w/e over the top. Production DBs should definitely not necessarily be containerized, although like you said, it's great for small scale dev/test stuff. For prod, I'd have your DB clusters be outside of docker, but you can do stuff like proxy DB calls through a service container (and collect your logging/health stuff there) and then scale that independently for whatever reason. Can be useful for multi-tenancy.
|
|
#
?
Jun 9, 2017 13:10
|
|
|
What I have found is that container shutdowns sometimes are not clean - it just kills the process and some data loss can occur due to incomplete writes. Maybe related to high system load, maybe just random luck. Have not investigated in depth. It's even worse with Windows containers where a graceful shutdown is flat-out not supported yet (though supposedly coming in RS3). Other than that, I have not encountered any issues. To those saying don't do it - why?
|
|
#
?
Jun 9, 2017 20:06
|
|
|
Vulture Culture posted:You can update mappings on multiple indexes by specifying a wildcard. Sweet, thanks! So I think I've finally got through the confusion of string/analyzed and text/keyword from the 5.x changes: https://www.elastic.co/blog/strings-are-dead-long-live-strings There's only 1 field out of like 40 that I need full text on so I changed pretty much everything from text & keyword to just keyword. However, when I tried to save the new mappings back to ES I ran into this: https://stackoverflow.com/questions/42925217/migrate-field-type-from-text-to-keyword-on-elasticsearchedit I'm a little confused on the creating a new index part of that solution though - would that mean I also have to update my logstash conf to start feeding data into the new index? What about all my existing data in the old index? edit: Looks like I can copy the old index into the new index which has the correct mappings already setup: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html but it seems like I will have to update the logstash conf to point it at the new index fletcher fucked around with this message at 21:45 on Jun 9, 2017 |
|
#
?
Jun 9, 2017 20:08
|
|
|
uncurable mlady posted:that sounds excruciating. why are you sysprepping your agents? Late update - we don't now! We had some weird lovely requirement for the agents to join our domain which required sysprepping for some reason I can't understand. But i got rid of it and now they're up in about half the time, but 10 minutes is still bad cos Windows on AWS is really lovely at booting. Cancelbot fucked around with this message at 21:34 on Jun 9, 2017 |
|
#
?
Jun 9, 2017 21:28
|
|
|
EssOEss posted:Other than that, I have not encountered any issues. To those saying don't do it - why? A better question is why bother? Containers are designed to be ephemeral and short lived. That's why their deployment mechanisms emphasize scale up/scale down behavior and time-to-new-deployment speed. Deploying a db in a container in production just feels unnecessary at best and a data loss risk st worst.
|
|
#
?
Jun 10, 2017 01:49
|
|
|
EssOEss posted:What I have found is that container shutdowns sometimes are not clean - it just kills the process and some data loss can occur due to incomplete writes. Maybe related to high system load, maybe just random luck. Have not investigated in depth. Your containers should gracefully handle SIGTERM/SIGKILL because that's what the daemon is going to send them.
|
|
#
?
Jun 10, 2017 03:40
|
|
|
fletcher posted:Sweet, thanks! Disclaimer: I am stuck on v2.4 of the ELK stack because it runs fine and I haven't had time to upgrade the world, so maybe this isn't true anymore Logstash should be creating a new index every day, named with the date. Something like logstash-2017.06.09. You can view your indices with a command like "curl localhost:9200/_cat/indices". So when I need to make a mapping change, I normally just suck it up and accept that things are going to be different prior to that change. I change the template, and when the index rolls over at midnight, all new data is indexed with the updated rules. I only keep a week's worth of indices since that's what I can fit in the resources available to me, so any old crap rolls off pretty quickly. If you are keeping months (or forever) worth of logs and want them to all be consistently queryable, then yeah, maybe you will need to get into the reindexing API to process everything again according to the new mappings. Basically instead of posting one-off custom mapping rules, you want to post a new template that includes ALL the mappings you want. Logstash has a default template, but it kind of sucks (again, as of 2.4). There's a setting on the elasticsearch output in Logstash to point it at a new template you create, and another to force Logstash to overwrite the existing template. Set those, define the new template, and then the next time the index rolls over it will use all the updated mappings. You can see the default templates here as a starting point to modify. Docjowles fucked around with this message at 04:11 on Jun 10, 2017 |
|
#
?
Jun 10, 2017 04:06
|
|
|
Related question... if I am doing pretty standard ELK stuff, how much am I missing out on by remaining on 2.4? Any killer features or huge performance wins I'm a moron to pass up? Indexing a bit over 1TB/day between http access logs, exim mail logs, and random multiline Java barf. The whole thing Just Runs after a lot of initial tuning so I am reluctant to mess with it. But I don't want to get so out of date that updating is impossible and all documentation and guidance on the web no longer applies, either. My main gripes right now are 1) the dogshit error logging from Logstash itself (you either get nothing at all, or like 5 megabytes of stack traces with no newlines anywhere, GLHF) 2) Logstash will pass a configtest just fine, and then still crash on startup due to invalid config for many classes of errors  3) Missing out on Kibana innovation. The visualizations as of 4.6 are very limited as soon as you want to show more than one thing at a time. Plus random bugs that will never get fixed as the branch is EOL Which are annoying but not so bad that I want to spend like a week upgrading everything if I don't have to Docjowles fucked around with this message at 04:30 on Jun 10, 2017 |
|
#
?
Jun 10, 2017 04:23
|
|
|
Blinkz0rz posted:A better question is why bother? Containers are designed to be ephemeral and short lived. That's why their deployment mechanisms emphasize scale up/scale down behavior and time-to-new-deployment speed. There are more reasons to use containers than them being ephemeral and there is nothing anywhere implying they need to be short lived. Genuinely curious, can we have a valid argument not to beyond "databases need to be persistent and containers are not!"? I just hear a lack of understanding of docker (volumes).
|
|
#
?
Jun 12, 2017 17:29
|
|
|
Mr. Crow posted:There are more reasons to use containers than them being ephemeral and there is nothing anywhere implying they need to be short lived. Do tell? I've only seen docker used in the context or easy deploys and hot swapping because of how fast they spin up and their isolation by design.
|
|
#
?
Jun 12, 2017 21:44
|
|
|
Managing data volumes / persistent volumes to work around the issues around the ephemeral-by-default nature of overlay filesystems, it's just less risky to deploy a mature RDBMS like MySQL or Postgres without the additional overhead of Docker containers involved. There really doesn't seem to be much advantage to containerizing Postgres and MySQL besides a single consistent deployment tool.
|
|
#
?
Jun 13, 2017 01:29
|
|
|
I think containers are an easy sell in the "ephemeral easy to start up" field because developers often need to do that and lack good tools for it otherwise. This does not mean that containers are limited to that, however. Look beyond the hype at what the technology really is. I have so far not found any reason not to containerize everything. Every deployment benefits from the isolation and certainty it offers and the value of targeting a single deployment mechanism company-wide (idealized - of course there are variations) is invaluable in achieving operational efficiency. Containers are not there for fancy features, they are there for uniform deployment. To simplify processes, not to create new technical capabilities. Technical glitches appear to exist in Docker Swarm and other fancy "cloud/cluster management" features, not the container runtime itself. We stay away from the fancy server orchestration features and stick to using it as a deployment mechanism. Not a single problem so far on Linux. The Windows container stack is slightly glitchy, though not overly so (no serious issues in actual production workloads, but artificially we can induce various failures under what should be "normal" conditions).
|
|
#
?
Jun 13, 2017 08:42
|
|
|
EssOEss posted:I have so far not found any reason not to containerize everything. Every deployment benefits from the isolation and certainty it offers and the value of targeting a single deployment mechanism company-wide (idealized - of course there are variations) is invaluable in achieving operational efficiency. More or less this. Talking about databases, I would never containerize the database files, they belong in a data volume and are to be treated with the utmost care and value and replicated and backed up and whatnot. But for the database application? It's true that you don't update it regularly the way you do with your own images, but there's still plenty of value in integrating it with the rest of your stack. There's a big difference between maintaining a single docker-compose file and maintaining a docker-compose file plus a separate installation script for Postgres or whatnot. Now If the overhead from the container is actually having an impact on your database use, then I agree I'd consider moving it to bare metal (though it's not the only option, e.g. it may be easier to switch to single-tenancy). But in that case it also moves out of the devops responsibility chain, and you pay a guy to take care of it and ensure it works smoothly with hot standbys and all that jazz, because you have the traffic to warrant it.
|
|
#
?
Jun 13, 2017 11:16
|
|
|
Can anyone share their thoughts on how containers turn updating from a system problem (run yum update once in the system) to an every app problem? Is this not a big deal in practice or does it turn out that updates don't happen as easily or as often as they should?
|
|
#
?
Jun 13, 2017 11:41
|
|
|
I do not have enough data to have a confident opinion but so far I have noticed that: * People feel more secure updating the host OS when they know that apps live in their own worlds, so the host updates are very unlikely to break anything in the containers. Doesn't make them update faster/better/more as far as I can tell but gives them more confidence in the process, which is at least something. * Updating the software in containers happens mostly by accident, if at all. The second bit does not make me feel quite good but when I look back, it's pretty much what has always been the case. That is, if we deploy 3rd party stuff with our apps, we tend to never update it because everyone is afraid of breaking everything on a shared host. So while there's no real advancement, there is also no regression.
|
|
#
?
Jun 13, 2017 14:13
|
|
|
|
| # ? May 14, 2024 08:48 |
|
|
Containers do not replace configuration management. However, it lets you get to a place where you can at least separate application configuration and artifacts from operations concerns such as logging, monitoring, kernel tuning, etc. The model also makes it easier to enforce 12-F applications instead of allowing lazy developers to dump files all over a filesystem. One new problem that arises is that now containers can become quickly outdated if they're created in a typical lazy manner that includes way more dependencies than necessary (all those lazy people using Ubuntu base images are scary, man). However, you can very quickly update your container hosts (may God have mercy on your soul if you're not using container orchestration) and many patching tasks becomes specific to application containers. This greatly reduces the burden of change management off of operations teams and helps achieve higher density and separation of containers. For example, you can reschedule containers to a set of nodes that are isolated from known updated containers. I still use Puppet and Chef to provision and maintain my worker nodes that host my containers.
|
|
#
?
Jun 13, 2017 15:30
|
|