|
Use ansible or chef manage your vm if it needs to be persistent Or use packer to build images if you want to spin them up on every run Or just use a cloud build service We use the built in cloud agents in azure devops for like 60% of our stuff and it works fine
|
#
?
Dec 30, 2021 01:42
|
|
|
|
| # ? May 15, 2024 04:44 |
|
|
The Fool posted:Use ansible or chef manage your vm if it needs to be persistent It doesn't need to be persistent, but it would cut on build time if it is. A cloud build service is not free, is it? This is for personal projects so I'm not to keen on going on buying an azure subscription just to build my crappy projects. Would ansible or chef work on a brand new OS, with nothing installed, like the pristine windows development VM? Or would it require some manual setup initially? Is ansible and chef pretty much the same thing? Or, is one better/easier than the other? Better for me means "the one with the leanest learning curve, the less I know about it the happier I am, and it still does things".
|
|
#
?
Dec 30, 2021 01:55
|
|
|
Volguus posted:It doesn't need to be persistent, but it would cut on build time if it is. A cloud build service is not free, is it? This is for personal projects so I'm not to keen on going on buying an azure subscription just to build my crappy projects. Why can't you just run an agent for whatever on your personal machine or a VM on that machine? If it's just for personal stuff who cares, right?
|
|
#
?
Dec 30, 2021 01:59
|
|
|
Methanar posted:
this was causing us strange issues for months before we realized it was enabled by default on ubuntu
|
|
#
?
Dec 30, 2021 02:04
|
|
|
New Yorp New Yorp posted:Why can't you just run an agent for whatever on your personal machine or a VM on that machine? If it's just for personal stuff who cares, right? I run/work in linux. A windows VM on my machine would have the same issues as the one running in proxmox: need to be replaced every 70+ days. What I was looking for was an automated way to do that (replace that VM). The obviously easy thing is to just use some key and set it to never expire and just move on with life. But I thought I may as well explore other possibilities before I'd go there. The scripts that I have make me having a working VM capable of building stuff in around 30 minutes to 1 hour, but they require me to click buttons. It's not a big deal once every 70+ days.
|
|
#
?
Dec 30, 2021 02:05
|
|
|
I'd just buy a $15-$25 server key from sa-mart and be done with it.
|
|
#
?
Dec 30, 2021 02:10
|
|
|
Volguus posted:It doesn't need to be persistent, but it would cut on build time if it is. A cloud build service is not free, is it? This is for personal projects so I'm not to keen on going on buying an azure subscription just to build my crappy projects. Azure devops is free for 5 users and 1800 build minutes/month
|
|
#
?
Dec 30, 2021 02:19
|
|
|
Also GitHub actions has a similar deal now iirc
|
|
#
?
Dec 30, 2021 02:20
|
|
|
Volguus posted:Would ansible or chef work on a brand new OS, with nothing installed, like the pristine windows development VM? Ansible needs some winrm setup but is otherwise agentless. Chef needs an agent and a scheduled task, unless you don't use chef-solo, in which case it also needs a server and a database. They are not really the same thing but could both solve this problem for you.
|
|
#
?
Dec 30, 2021 02:21
|
|
|
I would second looking into azure DevOps/ GitHub actions for this as they have images with vs etc installed already (ADO Windows image example). Then you just need to add any other tools in there which the platform should have tasks for. If you do want something self hosted, there is a vs build tools package that can be installed unattended.
|
|
#
?
Dec 30, 2021 02:50
|
|
|
AWS lets you rent a virtual windows server, create an AMI you want and then just spin one up once a week or something and then tear it down Friday afternoon using automation? If you have a bunch of garbage that needs to be cached, just throw that on a ebs volume and/or s3 bucket and remount as necessary. I guess. Building anything for windows sounds like an absolute disaster in 2022 compared to other build systems
|
|
#
?
Dec 30, 2021 04:08
|
|
|
Hadlock posted:Building anything for windows sounds like an absolute disaster in 2022 compared to other build systems It's not that bad. Windows Containers on the other hand are pure trash.
|
|
#
?
Dec 30, 2021 06:00
|
|
|
I just realized I've been at this company now about eight months and I've never had to SSH or otherwise auth into a server, the closest I've come is kubectl exec into a container, I don't think anyone has had to
|
|
#
?
Dec 30, 2021 06:07
|
|
|
Hadlock posted:Building anything for windows sounds like an absolute disaster in 2022 compared to other build systems There's plenty of corners of various industries that's Windows only - finance for one, certainly 
|
|
#
?
Dec 30, 2021 06:09
|
|
|
Oh yeah, I did that years ago, it wasn't fun then either
|
|
#
?
Dec 30, 2021 06:53
|
|
|
Volguus posted:I run/work in linux. A windows VM on my machine would have the same issues as the one running in proxmox: need to be replaced every 70+ days. What I was looking for was an automated way to do that (replace that VM). The obviously easy thing is to just use some key and set it to never expire and just move on with life. But I thought I may as well explore other possibilities before I'd go there. The scripts that I have make me having a working VM capable of building stuff in around 30 minutes to 1 hour, but they require me to click buttons. It's not a big deal once every 70+ days. Buy a key from SA-Mart build your VM snapshot it (if you can) and it should work fine. I have a couple of Windows VMs created this way and it works fine.
|
|
#
?
Dec 31, 2021 21:44
|
|
|
What’s best practice around configuring Jenkins agents? There doesn’t seem to be a sane method I can find to run a “if [app] isn’t installed” check for a stage that I can find so I’m figuring it’s something to be passed off to ansible/puppet/packer. I really wish I could get this god drat Proxmox plug-in working then it wouldn’t be much of an issue.
|
|
#
?
Jan 1, 2022 01:01
|
|
|
I don't know about best practice but I've had a stable install for several years with a "master to agent" setup with a home grown script. It basically just calls ssh but we use kerberos so there's some extra steps in there to get the logins to work. We tried having the agents initiate the connection to jenkins early on and it was unstable.. the jar running on the build nodes would crash after a few days.
|
|
#
?
Jan 1, 2022 01:13
|
|
|
We use Jenkins agents strictly as container workspaces. We have a general DevOps agent with general build tools + terraform/Ansible, and then our teams build off that agent with their desired tooling (via ASDF/brew et El) In general I always advise clients towards making the entire build pipeline ephemeral, and then caching where appropriate.
|
|
#
?
Jan 1, 2022 02:09
|
|
|
Gyshall posted:
This is good advice If you're not running Jenkins in k8s in 2022 it might as well be 2015 where you work
|
|
#
?
Jan 1, 2022 02:27
|
|
|
I loving wish it was 2015 where I work
|
|
#
?
Jan 1, 2022 13:19
|
|
|
Mr Shiny Pants posted:Buy a key from SA-Mart build your VM snapshot it (if you can) and it should work fine. I have a couple of Windows VMs created this way and it works fine. Thanks for all the advice. I am installing now a windows 11 vm with a Win 7 key from an old laptop that I had lying around (which has been running linux for years), and so far it seems to be happy with it. One would think that MS would make it easier for developers to build applications for their stupid platform, yet here we are. If this all ends well, gently caress it, I'll won't janitor it anymore.
|
|
#
?
Jan 6, 2022 05:05
|
|
|
Volguus posted:One would think that MS would make it easier for developers to build applications for their stupid platform, They've invested like 8 years of effort to making their software development ecosystem cross platform and open source.
|
|
#
?
Jan 6, 2022 05:24
|
|
|
New Yorp New Yorp posted:They've invested like 8 years of effort to making their software development ecosystem cross platform and open source. They did. And yet, when one wants to build on windows, for windows, they do their damnest to make it as inconvenient as possible. Or yeah, go to azure, of course. I mean, it's not like I'm not taking advantage of their work, because yes I'm using vcpkg, with cmake and building my applications with very little modifications from their normal linux target, which was inconceivable 5 years ago. I could even build with clang should I dislike msvc for some reason. And this is why I went to their development VM in the first place, to give it a try, to be as honest as possible to their OS. But, I failed ...
|
|
#
?
Jan 6, 2022 06:36
|
|
|
In the last 48 hours I've turned off 3500 CPU cores worth of poo poo that was completely unnecessary. And I've got at least another 3000 I can turn off in the next week. Absolutely ridiculous how wasteful and careless people are.
|
|
#
?
Jan 7, 2022 04:53
|
|
|
Methanar posted:In the last 48 hours I've turned off 3500 CPU cores worth of poo poo that was completely unnecessary. And I've got at least another 3000 I can turn off in the next week. Your ethics are shameful. You should have replaced them with crypto miners and built up a nice "severance bonus".
|
|
#
?
Jan 7, 2022 16:13
|
|
|
Methanar posted:In the last 48 hours I've turned off 3500 CPU cores worth of poo poo that was completely unnecessary. And I've got at least another 3000 I can turn off in the next week. Since you are probably not alone in this, guess the amount of resources just sitting around. It is staggering.
|
|
#
?
Jan 7, 2022 17:03
|
|
|
Yeah I’ve been doing some extremely obvious, bare minimum cost reduction work on our AWS environment and it’s quite frustrating. Identify some S3 buckets named basically “test” costing an obnoxious amount per month. No activity, the data is just sitting there. Speak with application owner, they say yeah they don’t need it any more and will delete. Three months later it’s still loving there lighting money on fire. Multiply this vignette by hundreds of people in the company building stuff in AWS and it’s just sad.
|
|
#
?
Jan 8, 2022 03:52
|
|
|
If that's in your stage/prod accounts, best practice is to enforce ownership tags and send a cost report to those owners every month (not that they will necessarily care, but at least they can't say they weren't aware). If that's in your dev accounts, run a regular resource deleter like AWS Nuke, Cloud Custodian, or one of many others. Let people tag stuff to preserve their resources from the deleter, but put an expiry date on it and force them to re-add it when it expires. This isn't just a cost saving measure, it's also good for security as it can catch VMs left behind by ex-employees or spun up by hackers.
|
|
#
?
Jan 8, 2022 04:26
|
|
|
I have a pipeline in place that builds a Docker container dedicated to a specific tool and pushed it up to Docker hub for consumption and so forth. Ideally I’d like for the build trigger to be whenever a new version of the tool is released but I don’t know a way to do so without just having a cron job just check for a new version ever day or so. Is there any sort of git hook styled solution for a repo you don’t own?
|
|
#
?
Jan 8, 2022 05:42
|
|
|
minato posted:If that's in your stage/prod accounts, best practice is to enforce ownership tags and send a cost report to those owners every month (not that they will necessarily care, but at least they can't say they weren't aware). Is there a decent tool for automating the cost reports? We currently have a lambda that my former manager wrote that generates a total cost of ownership report per team across all of our accounts. It works but it's not something I want to keep maintaining. It also doesn't handle reservations very well but that may just be a limitation of the api.
|
|
#
?
Jan 8, 2022 15:06
|
|
|
Docjowles posted:Yeah I’ve been doing some extremely obvious, bare minimum cost reduction work on our AWS environment and it’s quite frustrating. Identify some S3 buckets named basically “test” costing an obnoxious amount per month. No activity, the data is just sitting there. Speak with application owner, they say yeah they don’t need it any more and will delete. Three months later it’s still loving there lighting money on fire. Multiply this vignette by hundreds of people in the company building stuff in AWS and it’s just sad. Its really, really bad. I've been too busy to really to janitor everyone else's AWS spend the last 6 months, but in the last week I've finally sat down to look over some of it again and its just awful. Yesterday I found multiple things that are scheduled for >150 cores that have literally never been used and have been up for over a year. At list price of $0.05 core-hour that's 150*24*365*.05= $65k/yr. Like come the gently caress on. None of these people's time is so valuable they can't spend an hour doing even the most basic cost auditing for themselves to save $65k a year. And yet does nobody do anything until I start assigning tickets out to SRE asking WTF. Just crazy negligent poo poo. deedee megadoodoo posted:Is there a decent tool for automating the cost reports? We currently have a lambda that my former manager wrote that generates a total cost of ownership report per team across all of our accounts. It works but it's not something I want to keep maintaining. It also doesn't handle reservations very well but that may just be a limitation of the api. We use cloudhealth. It's okay. Methanar fucked around with this message at 19:34 on Jan 8, 2022 |
|
#
?
Jan 8, 2022 19:28
|
|
|
Methanar posted:Its really, really bad. At some point cost management becomes a leadership failure. If teams aren't managing their spend in a way that's sustainable, product management, product owners, and ultimately c-level folks should be asking questions about why costs are so high, especially if it drives up the cost to customers of whatever you're selling. But if no one cares except for you, it's not a great hill to die on. It's the worst kind of pissing into the wind: one where you know you're going to do all the work and still end up being the bad guy for interrupting a team's cycle.
|
|
#
?
Jan 8, 2022 20:01
|
|
|
Warbird posted:I have a pipeline in place that builds a Docker container dedicated to a specific tool and pushed it up to Docker hub for consumption and so forth. Ideally I’d like for the build trigger to be whenever a new version of the tool is released but I don’t know a way to do so without just having a cron job just check for a new version ever day or so. Is there any sort of git hook styled solution for a repo you don’t own? Put the code on GitHub/Gitlab/Bitbucket and use their build systems to run something on every commit. Each product has some slightly different marketing name but they all have a lot of similarities.
|
|
#
?
Jan 8, 2022 20:24
|
|
|
Blinkz0rz posted:At some point cost management becomes a leadership failure. If teams aren't managing their spend in a way that's sustainable, product management, product owners, and ultimately c-level folks should be asking questions about why costs are so high, especially if it drives up the cost to customers of whatever you're selling. I'm doing it because its extremely clearly documented value-add to the company on the order several millions of dollars per year. Somebody somewhere is going to appreciate me fixing this problem not only initially, but to keep pushing for a culture shift away from what lead to this in the first place. I think I deserve a promotion tbh for a lot of reasons, and a well documented annual value to the org of many millions of dollars is a good way to get it.
|
|
#
?
Jan 8, 2022 21:16
|
|
|
Methanar posted:I'm doing it because its extremely clearly documented value-add to the company on the order several millions of dollars per year. Somebody somewhere is going to appreciate me fixing this problem not only initially, but to keep pushing for a culture shift away from what lead to this in the first place. You're right, but if your promotion package is "saved a ton of money but everyone hates that I keep bugging them about cost at the expense of delivery" whoever approves your promotions isn't going to look at the savings without considering the cost. Saying "somebody somewhere will appreciate it" is one promo cycle away from being disappointed 'cause no one will unless leadership at every level is bought in. As a separate thought, at some point if your footprint and net spend is high enough it's probably worth hiring a separate team to handle cloud costs rather than make it ops' responsibility. Yet again, a leadership problem.
|
|
#
?
Jan 8, 2022 21:48
|
|
|
Blinkz0rz posted:You're right, but if your promotion package is "saved a ton of money but everyone hates that I keep bugging them about cost at the expense of delivery" whoever approves your promotions isn't going to look at the savings without considering the cost. I've mostly been just complaining and somewhat flippant. A lot of organization context is absent in a comedy forums post. I disagree that any individual or team's time is so valuable they can't spend an hour following my best practices wiki page and tweak a yaml value from 30 to 10 to save $100k/year, which is not an exaggeration or uncommon situation. I do have buy-in from the SRE director, and my own, as I am getting SRE time allocated towards fixing these things. I want to fix the obvious historical leadership absence on this issue: removing ignorance on the matter is the first step on that, which is why I wrote up 50+ tickets describing the problems at this point with a jira board to hold them along with grafana graphs. In a rare moment of positive naivety, I think I have the opportunity to move the org towards a saner position of cost-consciousness. Maybe this does culminate with a dedicated FTE whose entire job is to keep tabs on cloud costs. I'm taking the bet that individual initiative on high time:value work will be rewarded. Despite all of my negative attitude posting: I care a lot about my career. I want senior II and the way I get it is by documenting high value work and to be as visible as I can be, on this cost issue and all others. I'll report back in 7 months on how it went.
|
|
#
?
Jan 8, 2022 23:21
|
|
|
Yeah, it's completely reasonable to think "I just saved the company a bunch of money! I will be showered with praise and bonuses!" but in my experience it's just not true. Don't assume upper management wants to save money. It's like those dumb situations where managers take everyone out for expensive training near the end of the year because if they don't spend their budget, they'll lose it next year. It makes perfect sense to think "but if I save $1MM here, that's money that could be used to hire more people, upgrade that old system, etc etc" but that's not how budgets work, the money doesn't slosh around between buckets like that. As someone whose whole job is about technical efficiency, seeing these gigantic holes in financial efficiency feels incredibly frustrating. I suggest making a note of these savings opportunities and keeping them in your back pocket. Then if management comes to you in desperation looking for ways to optimize, you can pull them out and look like a hero. Blinkz0rz posted:it's probably worth hiring a separate team to handle cloud costs rather than make it ops' responsibility. - vendor discounts: purchasing RIs and savings plans, vendor negotiations, paying annually vs monthly, partnerships - administrative optimizations like enforcing tagging, setting up budget alerts, making sure users are aware of their spend, following up on cloud automated recommendations (e.g. Trusted Advisor), and securing clouds to ensure the goddamn coin miners don't get in. - ongoing technical optimizations: like writing tooling that will shut down dev workloads on weekends, choosing cheaper clouds, regions, and instance types. all of those are involved enough that it warrants full time positions.
|
|
#
?
Jan 8, 2022 23:34
|
|
|
I don't remember exactly where I saw this anymore but I remember reading about a company where to control AWS costs anyone that made changes that saved the company on monthly AWS costs got a percentage of the savings as a bonus or raise. If it's not big savings people shouldn't spend a lot of time on it obviously, but if it's even a couple hours worth of work to save $10k / month and you get $12k / year more that's better than even a promotion in many companies. So cost controls being practiced and the idea of "treat company money like it's your own" was reinforced. Granted, in a bad company people would probably intentionally make things inefficient at first to collect a bigger check, so something else in policy would need to be done to prevent gaming like that.
|
|
#
?
Jan 9, 2022 18:33
|
|
|
|
| # ? May 15, 2024 04:44 |
|
|
that on the extreme end has the problem of people cutting costs in ways that do not help the business, ex. via risky levels of efficiency or bureaucracy that reduces the amount of good work being done. cost reduction is not something that operates on a single metric. even seemingly low hanging fruit is not a good priority for optimization because of the efficiency it allows elsewhere.
|
|
#
?
Jan 9, 2022 20:40
|
|