|
Hadlock posted:So data security isn't a devops problem any more? I just throw it over the wall to the The team who owns the systems containing the data should be responsible for securing it. Maybe I don’t understand what you mean with IAM admin groups but I don’t see how they are responsible for data security on systems they don’t maintain… Data Engineers are likely working with a DevOps mindset, as in they’re both developing and maintaining their systems. The Data part mostly marks what kind of development they do.
|
#
?
Mar 5, 2024 13:44
|
|
|
|
| # ? May 17, 2024 20:38 |
|
|
Anything that's everyone's responsibility quickly becomes in practice nobody's responsibility nor accountability
|
|
#
?
Mar 6, 2024 00:13
|
|
|
I'm still not convinced data engineering is somehow separate. What specifically non operational task do data engineers do that doesn't have deep overlap with either DevOps and traditional analytics groups doing rudimentary ETLnecrobobsledder posted:Anything that's everyone's responsibility quickly becomes in practice nobody's responsibility nor accountability There's a joke involving fighting fires at work, "only you can prevent forest fires" and perpetual raging wildfires in California, but I'm not brave enough to make it
|
|
#
?
Mar 6, 2024 00:18
|
|
|
Hadlock posted:I'm still not convinced data engineering is somehow separate. What specifically non operational task do data engineers do that doesn't have deep overlap with either DevOps and traditional analytics groups doing rudimentary ETL Data engineering is kind of a mashup of ops, analytics, db admin, and data architecture. It's basically the evolution of OLAP groups with devops methodologies applied.
|
|
#
?
Mar 6, 2024 02:32
|
|
|
Hadlock posted:I'm still not convinced data engineering is somehow separate. What specifically non operational task do data engineers do that doesn't have deep overlap with either DevOps and traditional analytics groups doing rudimentary ETL They run the Hadoop and Cassandra so I don’t have to. I run the kubernetes so they don’t have to. Then they run their Hadoop on my Kubernetes and now nobody understands what the gently caress is going on. Anyways our stock hit an all time high today
|
|
#
?
Mar 6, 2024 08:47
|
|
|
necrobobsledder posted:Anything that's everyone's responsibility quickly becomes in practice nobody's responsibility nor accountability
|
|
#
?
Mar 6, 2024 20:22
|
|
|
madmatt112 posted:They run the Hadoop and Cassandra so I don’t have to. Meanwhile I joined my company right as its stock was trading at the all time high and then it immediately went sharply down lol. That first RSU grant uh did not perform very well  Overall I have no complaints about my comp but the stock has been a joke for a long time. For better or worse they pay a lot in cash, the stock isn't meant to be like 80% of your comp FAANG style E: in unrelated news like 3 days later I am still trying to walk this cloud security professional through creating an AWS security group without outright writing the terraform for him Docjowles fucked around with this message at 00:43 on Mar 7, 2024 |
|
#
?
Mar 7, 2024 00:11
|
|
|
Vulture Culture posted:I view this through the lens of people who have to pre-divide chores with their spouses and if that's you, bless you, but there are absolutely other ways to do it Also, I would hesitate to make metaphors between marriages and work without very specific scoping of context. Chores continuously occupying more of my time without any concern, feedback loop, or limiter for what generates these tasks is a large part of why I called it quits for both arrangements before, and security is a chore / afterthought for most companies obsessed with delivering features unless they have the resources to spare by the graces of thine divine holy leadership.
|
|
#
?
Mar 7, 2024 04:38
|
|
|
Docjowles posted:E: in unrelated news like 3 days later I am still trying to walk this cloud security professional through creating an AWS security group without outright writing the terraform for him Sounds like a similar exercise as I’ve been undertaking. I’m getting a dev team to work with git. Concepts like commits, branches, merge requests and the different state between local and remote are completely alien to them. Answering questions like “why do we need version control if we already define version in our pom.xml?” or “can’t we just copy/paste our application directly onto the prod server instead of using a pipeline?” and “we test manually so can’t we just turn off the automated testing?”. Apparently the whole team has the same workflow where they all edit the main branch through the web interface of our git repository and then pull it onto a test environment where they do addtional changes before pushing it back. They’ve messed up the order in which they made changes resulting in merge conflicts. So now I’m dealing with “it always worked and now it’s broken so obviously YOUR team has changed something and YOU need to fix this” 
|
|
#
?
Mar 7, 2024 08:01
|
|
|
LochNessMonster posted:Sounds like a similar exercise as I’ve been undertaking. I’m getting a dev team to work with git. Mildly curious what industry you work in I experienced this at an HR software company that had been stuck in the stone age but that was... 7 years ago? I would have thought we are all on the same page about basic git mechanics at this point but apparently not I guess at my last job, my predecessor had introduced some janky thing that depended on developers force pushing to master
|
|
#
?
Mar 7, 2024 08:18
|
|
|
It’s not par for the course luckily. This is a pretty isolated team doing risk modeling in an ancient statistics program. These folks are extremely good in their job but have worked in the same way for decades and are now forced to start applying “new” tools to their trade.
|
|
#
?
Mar 7, 2024 09:21
|
|
|
LochNessMonster posted:but have worked in the same way for decades and are now forced to start applying “new” tools to their trade. Sometimes I wonder with containers, Kubernetes and terraform if I'm stuck in a rut and need to expand my tool set so I don't become like this. I'm not sure what's out there that might be better, though
|
|
#
?
Mar 7, 2024 09:40
|
|
|
Hadlock posted:Sometimes I wonder with containers, Kubernetes and terraform if I'm stuck in a rut and need to expand my tool set so I don't become like this. I'm not sure what's out there that might be better, though Exposure to k8s, containers and terraform will definitely not hurt you (except emotionally  ). It feels like these major tech shifts happen fairly slowly, but if you find yourself at the end of the bell curve you’re more likely to be less employable. But if you’re one of the last dinosaurs you can make mad bank because nobody knows how ancient tech works anymore.
|
|
#
?
Mar 7, 2024 09:48
|
|
|
A ton of companies are just building Microservice Jenga at this point, and the next big-paying job is just going to be fixing that
|
|
#
?
Mar 7, 2024 20:13
|
|
|
LochNessMonster posted:Sounds like a similar exercise as I’ve been undertaking. I’m getting a dev team to work with git.  What the actual gently caress, did these guys walk out of a portal from 2005 lol. That's about the last time I encountered FTPing code to "the server" as a deployment strategy
|
|
#
?
Mar 7, 2024 21:02
|
|
|
LochNessMonster posted:It’s not par for the course luckily. This is a pretty isolated team doing risk modeling in an ancient statistics program. These folks are extremely good in their job but have worked in the same way for decades and are now forced to start applying “new” tools to their trade. Define some sensible defaults, give them clarity and define what crawl/walk/run look like in observable metrics and get some stakeholder buy in to focus on "engineering excellency". Take it slow and one default at a time until the team is at least crawling; do suggest having "trunk based" within this type of environment over branching strategies as those will just lead to more pain down the road as the team starts to "get smart" with how they understand "git flow" should work etc.
|
|
#
?
Mar 8, 2024 01:03
|
|
|
Docjowles posted:
They work on a very niche product and are more business analists than actual devs, even if they write code/logic in some kind of legacy language. Apparently modern SDLC has never been required in this niche. I was as surprised as you are. drunk mutt posted:Define some sensible defaults, give them clarity and define what crawl/walk/run look like in observable metrics and get some stakeholder buy in to focus on "engineering excellency". Take it slow and one default at a time until the team is at least crawling; do suggest having "trunk based" within this type of environment over branching strategies as those will just lead to more pain down the road as the team starts to "get smart" with how they understand "git flow" should work etc. My team has built their build/release pipeline. They’re migrating yo a different platform altogether so their future is not really my concern. The guardrails are in place and the pipeline gives them a way to build/release from scratch to prod. Whatever manual labor they have introduced is up to them, as long as they don’t come to me for that. I’ve tried helping them out with improving their workflow but was met with “we’ll discuss it with the team” on every occaision so I don’t think they’re eager to change.
|
|
#
?
Mar 8, 2024 18:49
|
|
|
Docjowles posted:
|
|
#
?
Mar 8, 2024 20:08
|
|
|
I don't blame a lot of people for staying far behind modern software trends given how incredibly fashion-driven our silly industry is for supposedly such an "engineering" culture we're supposed to have. But I guess it's about resume-driven development for making sure we don't get stuck at companies that pay on the other end of the bimodal distribution of software, which is honestly the dominant part of the distribution of our industry.Vulture Culture posted:A ton of companies are just building Microservice Jenga at this point, and the next big-paying job is just going to be fixing that
|
|
#
?
Mar 11, 2024 18:20
|
|
|
Yeah, this is a really sensible approach. A lot of service and platform designs fail to consider who is going to be making what kind of changes to the software at what pace, so we end up making precisely the wrong decisions. Core business logic gets broken into dozens and dozens of tiny independently-shipped parts that are substantially harder to change and test for people who are close to, but not on, the owning team. At the same time, we're continuing to build a lot of the central platforms that run the business as monoliths, when there's absolutely no benefit to doing so because the people making contributions are coming from everywhere, and nobody has any context about the system anyway.
|
|
#
?
Mar 11, 2024 18:40
|
|
|
necrobobsledder posted:I don't blame a lot of people for staying far behind modern software trends given how incredibly fashion-driven our silly industry is for supposedly such an "engineering" culture we're supposed to have. But I guess it's about resume-driven development for making sure we don't get stuck at companies that pay on the other end of the bimodal distribution of software, which is honestly the dominant part of the distribution of our industry. The book Kill It With Fire does a good job of talking about the benefits of monoliths without seeming like reactionary reverse-hipster stuff - basically that there's a lot of benefits to them, and if you try and preemptively scale your stuff into microservices you introduce a lot of inefficiencies to the development process that aren't great. It also doesn't pretend that monoliths don't have problems and talks about the right time to migrate away/etc. I really do love that book.
|
|
#
?
Mar 11, 2024 20:04
|
|
|
Conway's Law applying to software architecture really needs to be repeated in that if your microservices don't resemble your organizational structure you're probably going to have a Bad Time. Granted, a ton of organizations are absolutely horribly organized anyway so this may not be the best model but it's also unfortunately true that simply splitting up or collapsing together your software architecture won't cause a re-org eventually to support it either. Sorry, folks, Taylorism is alive and well despite all the Deming books we sacrifice to the altar of agile and quality driven engineering processes or whatever will get people's SEO numbers up these days.
|
|
#
?
Mar 12, 2024 00:23
|
|
|
Something else to remember about Conway's Law is that it's really mentally tempting to simplify that organizational design into a reporting structure org chart, but the question who's doing the work? goes far beyond the nominal conversations about ownership that people like to start from.
|
|
#
?
Mar 12, 2024 15:00
|
|
|
Possibly the dumbest thing you'll read today https://thenewstack.io/with-yamlscript-yaml-becomes-a-proper-programming-language/  I suspect this is what happens when you don't pay your engineers market rate and they start reaching for whatever is closest and trying to improve it to justify a raise
|
|
#
?
Mar 12, 2024 17:27
|
|
|
Justify a trip to the guillotine if you ask me. Some engineers I worked with once upon a time used xml for programming, they felt it was "better" to parse the xml into python and eval it than letting users write python. This project didn't end well.
|
|
#
?
Mar 12, 2024 17:36
|
|
|
Hadlock posted:Possibly the dumbest thing you'll read today The day is just starting but I can't see you being wrong.
|
|
#
?
Mar 12, 2024 18:09
|
|
|
Hadlock posted:Possibly the dumbest thing you'll read today I used to work with the creator, and this doesn't surprise me at all. People struggle enough with YAML as it is, let alone the various ${{}} additions we see all over the place where this is going to add to that
|
|
#
?
Mar 12, 2024 18:43
|
|
|
lol we need another programming language, and it should be built on YAML OK
|
|
#
?
Mar 12, 2024 18:56
|
|
|
Finally, my Ansible can become truly unreadable!
|
|
#
?
Mar 12, 2024 21:38
|
|
|
xzzy posted:Justify a trip to the guillotine if you ask me. lol homegrown cold fusion I kind of appreciate the perverse though process of knowing that JSON is legal javascript and setting out to confabulate the hosed language that YAML is a subset of
|
|
#
?
Mar 12, 2024 22:02
|
|
|
xzzy posted:Justify a trip to the guillotine if you ask me. Let’s add the creator of the abomination that is jsonnet to that trip as well.
|
|
#
?
Mar 12, 2024 22:55
|
|
|
I can’t believe a guy who calls himself “Ingy döt net” came up with a dumb and bad idea
|
|
#
?
Mar 12, 2024 23:16
|
|
|
https://github.com/yaml/yamlscript posted:Even though YAMLScript often has the look of an imperative programming language, it actually is just a (YAML based) syntax that compiles to Clojure code. it's good actually, and so is yaml.
|
|
#
?
Mar 12, 2024 23:39
|
|
|
Docjowles posted:I can’t believe a guy who calls himself “Ingy döt net” came up with a dumb and bad idea https://resume.ingy.net/ quote:Notes at some point it is time to admit you have a problem
|
|
#
?
Mar 12, 2024 23:55
|
|
|
NihilCredo posted:https://resume.ingy.net/ Sums up how some of the software I ended up taking over was written. Every single part of the process had to involve something he had created where even a simple front end website that could have been vanilla JS instead required you to install perl and <some other stuff I cannot remember>. All of this due to the fact it wasn't html and js you were working on, it was some form of created-here templating language that required a 4 stage build to turn into nearly-legal html and JS files that you could then run locally. Your one line change would have to wait for the 5minute build process to finish to see the result locally.
|
|
#
?
Mar 13, 2024 00:30
|
|
|
NihilCredo posted:https://resume.ingy.net/ CPAN? More like CPEP 
|
|
#
?
Mar 13, 2024 15:01
|
|
|
A sales rep from our local isp called me to offer an upgrade to my current package. As we were chatting, I mentioned that I used to work for them, before I retired. The rep politely asked me what I used to do there, to which I explained that I was one of the sysadmins for the entire build and deployment pipeline. She then asked me what I had worked on, to which I named a certain project. She got really excited and told me that she couldn't believe she was talking to me right now. I was a little confused until she explained that they were spinning up five or six new projects based on my work. lol, If only I could get royalties, but that's not how big companies work.
|
|
#
?
Mar 13, 2024 20:45
|
|
|
is this the right thread for docker questions? I'm looking to be able to whip up some demo streamlit apps to get some stuff in front of team mates and other users to try. Got another guy on the data science team who's already stood one up on an on-prem server, and chatting with him mentioned + showed me that I should be able to ssh into there and if I wanted to, mount a volume on it for my own docker containers and sort of "carve out my own space" I'm pretty new when it comes to docker, I've got the concepts down I think fairly, but my only real experience has been on my own machines locally or with docker desktop. I was under the impression that volumes were for persisting storage or databases used by multiple containers on the same docker server, so I'm not really sure I understood what he meant by being able to sort of "carve out my own space" with a volume. I don't get the impression any of this is setup for CI/CD so I'm also scared of messing anything up as I'm mainly used to just launching containers from gitlab runners/pipelines and pre-built yaml configs, so this sounds much more uhh...hands on? Anyone point me in the right direction about what I should be reading up on to understand this workflow? Oysters Autobio fucked around with this message at 02:26 on Mar 14, 2024 |
|
#
?
Mar 14, 2024 02:24
|
|
|
Oysters Autobio posted:is this the right thread for docker questions? If you're going down this path, maybe just install K3S on the server, then you can kubectl exec -it my-docker-container bash without needing to distribute/update ssh keys. Also you can namespace containers so you're not stepping on eachother's shoes. So you'd `create namespace oysters` and then `kubectl exec -it oysters-autobio bash -n oysters` and the data science guys could keep their containers over in the `-n datasci` namespace, or whatever https://github.com/k3s-io/k3s/?tab=readme-ov-file#quick-start---install-script If you want to get real sloppy, everyone can share a single ./kube/config file, or once you figure out what the hell you're doing, you can spawn addtional service accounts for each user and corresponding kube config files. Best practice is to put an IAM system in front of it, but that should get you off the ground for week 1
|
|
#
?
Mar 14, 2024 04:05
|
|
|
|
| # ? May 17, 2024 20:38 |
|
|
Oysters Autobio posted:is this the right thread for docker questions? I had not heard of streamlit before; poked around and saw the App Gallery examples ( https://streamlit.io/gallery ) and it looks cool / useful to me: I know Python, some data science, etc, but not html. Trying to answer your question: How is your team member launching things? Assuming it's "by hand": with 'docker run'? kubectl (kubernetes)? podman (RHEL)? docker compose? ...? Ask him that, first Then, yes, volumes are for persisting storage. How to create a volume depends on the answer to the above question. I'm not sure if you need a volume, at least initially, if you're not needing to create/save data. Maybe you might need one if you want to bring some data. Looks like there are some tutorials, however I'm not sure that they are that newbie friendly: https://docs.streamlit.io/knowledge-base/tutorials/deploy/docker https://docs.streamlit.io/knowledge-base/tutorials/deploy/kubernetes As for the next step....honestly my best suggestion would be to ask for a demo from your coworker. If they have something working, then just replicate whatever they did. Once you have something working, then you can dive into the details of things like "how to create/change a volume"
|
|
#
?
Mar 14, 2024 06:59
|
|
