|
Methanar posted:why Learning new things is fun and cool, also, we all develop with intellij products, so why not give it a shot?
|
#
?
Aug 19, 2018 02:15
|
|
|
|
| # ? May 15, 2024 20:22 |
|
|
I think we were paying $600/build machine license, once we got over 12 engineers it got really expensive to have enough build machines to keep up with building on every commit.
|
|
#
?
Aug 19, 2018 06:34
|
|
|
Hadlock posted:I think we were paying $600/build machine license, once we got over 12 engineers it got really expensive to have enough build machines to keep up with building on every commit. That’s why we only do a build for pull requests and spawn a AMI when the build is kicked off.
|
|
#
?
Aug 19, 2018 06:36
|
|
|
Yeah, don't. Jenkins sucks but it sucks about a billion times less than TeamCity. (concourse.ci if you're equipped for it, GitLab CI if you're not, Jenkins if you write Java and you believe it is a compiler error to build code while not wearing a blue tie)
|
|
#
?
Aug 20, 2018 05:27
|
|
|
I recommend TeamCity for small development teams who have to manage their own on-premises CI for some reason, but that's a niche that's getting smaller every day. It's not unpleasant to work with, but not nearly as pleasant as a hosted product, and it doesn't give you a quarter of the flexibility I would need to justify actually running a whole CI infrastructure.
|
|
#
?
Aug 20, 2018 15:09
|
|
|
We do just fine 
|
|
#
?
Aug 21, 2018 21:21
|
|
|
I'm the weirdo that likes Bamboo
|
|
#
?
Aug 21, 2018 21:50
|
|
|
JHVH-1 posted:I'm the weirdo that likes Bamboo bamboo is cool if you don't do a lot of stuff to the point where switch pages is a 3+ second wait.
|
|
#
?
Aug 21, 2018 21:52
|
|
|
JHVH-1 posted:I'm the weirdo that likes Bamboo Australian spotted.
|
|
#
?
Aug 21, 2018 23:58
|
|
|
We have 200 devs, 450 active projects, and 10 build agents on TeamCity; doing great except for the massive wave of mercurial checks causing BitBucket to rate limit the poo poo out of us. But we're an anomaly organisation where we are 99% Microsoft stack/Windows running in AWS, using lambdas, expanding into containers. We do the DevOps through PowerShell, C#, and Python and are breaking all sorts of new ground. But it also makes a lot of my skills non-transferable cos I can't do Linux poo poo that well  But hey; Spinnaker deploy an app that's also a NuGet package with Red/Black deploys on Server 2016? I can do that! Or Docker swarm some nano containers running legacy .NET 4.0 apps...
|
|
#
?
Aug 22, 2018 09:01
|
|
|
What's your spend average like with that much MSFT infra?
|
|
#
?
Aug 22, 2018 10:32
|
|
|
Gyshall posted:What's your spend average like with that much MSFT infra? Right now, it's 2 environments (staging, prod) across 15 accounts and 26,000 instance hours: Around $60,000 per month total, or $37,000 for EC2 alone. SQL enterprise is the worst offender at $8,000 per month, before we got reserved instances in which brought them down to $5,500 per month. We have long term goals to bring our entire datacentre into AWS and working hard to get people off the SQL cluster and into RDS/Dynamo/etc. as well as support .NET Core to allow for Linux migrations. But even at the inflated costs of Windows vs. Linux it's still a lot cheaper than what we paid for the physical datacentre amortised over its lifetime. Whats helping us is that our developers are quite on the ball when it comes to DevOps and owning their stack, as the infrastructure and DBA teams are seen as blockers to their work. As long as we can guide them to the cost-effective options we come out on top long term. Cancelbot fucked around with this message at 11:58 on Aug 22, 2018 |
|
#
?
Aug 22, 2018 11:56
|
|
|
Like the idea anyone would suggest Jenkins over anything is just 
|
|
#
?
Aug 22, 2018 19:25
|
|
|
Mr. Crow posted:Like the idea anyone would suggest Jenkins over anything is just We build our Jenkins as ephemeral Docker images and configure via groovy, it works quite well. Basically just fire off jobs from external events.
|
|
#
?
Aug 22, 2018 19:35
|
|
|
Has anyone tried using genuinetools/img to build containers Baking docker in to a Jenkins container seems like gross overkill to build an image that will never run locally.
|
|
#
?
Aug 22, 2018 19:55
|
|
|
Use a plug-in for Jenkins and run your slaves in ECS/fargate/docker somewhere. Don't run jobs on the master.
|
|
#
?
Aug 22, 2018 19:56
|
|
|
Hadlock posted:Has anyone tried using genuinetools/img to build containers Its garbage and literally does not work. I tried every daemonless, non-docker docker image builder and not a single one of them worked at all. If you're trying to build containers inside of containers that are ephemerally spawned by Jenkins, you're hosed. Its basically impossible as far as I'm concerned. Methanar posted:Except containers in containers isn't actually containers in containers because docker-in-docker doesn't exist because even the sub containers are still sharing the same host kernel as everything else docker in docker is really docker beside docker because everything is actually sharing the same docker daemon but it becomes bad because containers started by anything other than kubernetes are outside of the control of the resource scheduler and worse still is trying to pass around the docker unix socket because you're like 5 filesystem abstractions deep (file system abstractions are also side-by-side rather than actually nested but lol have fun parsing docker inspect to find where the gently caress anything actually is on the real host root filesystem) also good luck trying to get containers that are not managed by kubernetes, but still on a kubernetes host, to have any sort of network connection because you won't have the nice resource scheduler hooking you up to whatever flannel/calico/weave CNI you've got
|
|
#
?
Aug 22, 2018 20:27
|
|
|
I just put the gitlab runners on a dedicated build machine and give them access to the docker socket, then I can just run docker and swarm commands from the gitlab-ci.yml file, by using the Docker executor and the standard Docker image. Yes this means that every project has effectively root access to the machine, but there's nothing in it except build images so the potential accidents are minimal.
|
|
#
?
Aug 22, 2018 20:39
|
|
|
NihilCredo posted:
|
|
#
?
Aug 22, 2018 22:03
|
|
|
I starred kaniko back when it was released to try and escape d-in-d: https://github.com/GoogleContainerTools/kaniko Never got around to messing w/ it, but it's the most "official" of the no-docker-socket build projects I've seen (in that Google wrote and uses it).
|
|
#
?
Aug 22, 2018 23:37
|
|
|
Is there anyone else doing non-image artifact-based CM, eg chef-zero or ansible bundles? Where/when do you drop your artifacts? We have a bunch of terraform for setting up instances of our infrastructure and currently it lives separately from the provisioning code, but we want to merge the repos so we can release them as one. Currently our hosts pull provisioning code from git (I know, I know), but continuing to do this is for sure going to gently caress us up for any sort of branching when we merge the two repos (here come the self-dependencies) and also it’s just bad practice. My thought right now is to generate an instance-specific CM bundle with Terraform that’s pushed onto S3, but I’m wondering if anyone has hit any issues with something like that. We also have artifactory available, but we wouldn’t be able to have per-instance artifacts which are important for us for testing new Infra or CM code. I’ve handled this in the past for smaller projects by delivering a hex-encoded tar of the provisioning code in user data, but unfortunately we’re just past the 16k limit 😉
|
|
#
?
Sep 11, 2018 04:08
|
|
|
I was going to ask about artifact repos as well. We're thinking of getting one for general use (we mostly do web or mobile projects and need a place to dump our artifacts locally), and the choices are Artifactory, Nexus, and... Jenkins (lol). Artifactory seems to be the best feature-wise, but it's rather expensive and would be harder to justify the cost to the higher-ups. Any recommendations and/or experiences with them?
|
|
#
?
Sep 12, 2018 07:25
|
|
|
Schneider Heim posted:I was going to ask about artifact repos as well. We're thinking of getting one for general use (we mostly do web or mobile projects and need a place to dump our artifacts locally), and the choices are Artifactory, Nexus, and... Jenkins (lol). Artifactory seems to be the best feature-wise, but it's rather expensive and would be harder to justify the cost to the higher-ups. Any recommendations and/or experiences with them? Both artifactory and nexus are terrible. If you have to use either look at the SaaS versions.
|
|
#
?
Sep 12, 2018 13:22
|
|
|
I've used ProGet which was pretty OK. Nice if you've got .NET stuff in the mix. But yeah, depending on your size I'd probably point you to the SaaS versions of anything you seriously want to use unless you have some real compelling reason to run the server in-house.
|
|
#
?
Sep 12, 2018 13:33
|
|
|
An sftp server with some bash to set up and maintain this directory structure: ${REPO_NAME}/${GIT_SHA}/${MY_COOL_ARTIFACT}
|
|
#
?
Sep 12, 2018 16:54
|
|
|
Helianthus Annuus posted:An sftp server with some bash to set up and maintain this directory structure: ${REPO_NAME}/${GIT_SHA}/${MY_COOL_ARTIFACT} I’m tempted to replace Nexus with this.
|
|
#
?
Sep 12, 2018 19:40
|
|
|
Helianthus Annuus posted:An sftp server with some bash to set up and maintain this directory structure: ${REPO_NAME}/${GIT_SHA}/${MY_COOL_ARTIFACT} I think that is basically how we do some of our HTML5 content, but it just goes directly to S3 and then that has a CDN in front of it and the site embeds the correct version.
|
|
#
?
Sep 12, 2018 19:51
|
|
|
Artifactory is fine for its original purpose of being able to host a lot of different package types. Hosted a yum repo, Docker repo, Maven repo, and a dump of tarballs and the primary complaint for me is the cost and if you’re self hosting it’s just awful. If you’re forced to use Jenkins then may God have mercy and not force you to use the artifactory plugin to publish artifacts. My current place also uses it for the Node packages we’re writing and haven’t heard a peep of complaint. Maybe it doesn’t work for people doing 2000 builds an hour or something but it has been one of my least complicated and most reliable thing. But maybe that’s Stockholm Syndrome talking.
|
|
#
?
Sep 12, 2018 23:24
|
|
|
necrobobsledder posted:Artifactory is fine for its original purpose of being able to host a lot of different package types. Hosted a yum repo, Docker repo, Maven repo, and a dump of tarballs and the primary complaint for me is the cost and if you’re self hosting it’s just awful. If you’re forced to use Jenkins then may God have mercy and not force you to use the artifactory plugin to publish artifacts. My current place also uses it for the Node packages we’re writing and haven’t heard a peep of complaint. Maybe it doesn’t work for people doing 2000 builds an hour or something but it has been one of my least complicated and most reliable thing. But maybe that’s Stockholm Syndrome talking. My self hosted version has been running just fine for 4+ years, but we don't do anything complicated with it
|
|
#
?
Sep 12, 2018 23:28
|
|
|
Janitor Prime posted:My self hosted version has been running just fine for 4+ years, but we don't do anything complicated with it We self host and have a ton of stuff in it, and I think I've seen it down once for about an hour in the last couple of years?
|
|
#
?
Sep 12, 2018 23:42
|
|
|
We have also been using Artifactory for years and it is fine. Other than “it costs money” I don’t really have a single complaint offhand. It’s shared by like a dozen teams pushing and pulling every possible type of artifact and just works. Compared to our self hosted Atlassian poo poo it is a model of stability. Which I guess isn’t saying much. Docjowles fucked around with this message at 05:56 on Sep 13, 2018 |
|
#
?
Sep 13, 2018 05:49
|
|
|
Docjowles posted:Compared to our self hosted Atlassian yeah, that's pure pain.
|
|
#
?
Sep 13, 2018 06:35
|
|
|
I prefer the cost of artifactory over what those NPM assholes wanted to charge. Its like: ok you want me to pay a full-time engineer's salary to you guys for an on-site NPM server that I have to run and maintain, when I could pay a quarter of that for a self-hosted artifactory and get all the functionality of NPM plus maven, docker, yum, rubygems, pypi, etc etc etc? I think its a good product, and you can get a lot of milage out of the free version too. Same with nexus, really. Both solutions have improved a lot over the years.
|
|
#
?
Sep 13, 2018 19:38
|
|
|
Docjowles posted:Compared to our self hosted Atlassian poo poo it is a model of stability. Which I guess isn’t saying much. I've only been in charge of maintaining our three atlassian products since February, and it's only for 150 users but what problems are you having? Other than the JVM running out of memory exactly once it just kind of hums along in the background.
|
|
#
?
Sep 14, 2018 06:30
|
|
|
Mostly the JVM running out of memory lol. We tune the heap and GC settings and eventually give in and give it more RAM, but it always seems to expand to fill it and the OutOfMemory exceptions return. It’s not like every day but enough to be annoying and disruptive.
|
|
#
?
Sep 14, 2018 12:49
|
|
|
Docjowles posted:Mostly the JVM running out of memory lol. We tune the heap and GC settings and eventually give in and give it more RAM, but it always seems to expand to fill it and the OutOfMemory exceptions return. It’s not like every day but enough to be annoying and disruptive. Gotta look at Metaspace now as well on newer Java. It actually uses memory outside of the other settings. From what I remember tuning it a couple years ago it is highly dependent what the developer is including in the code. At that job we set the max and if the developer hit it then it was up to them to try and reduce it or else they have to beg us to change the settings and wait for it to go through release process.
|
|
#
?
Sep 14, 2018 16:30
|
|
|
uncurable mlady posted:I've used ProGet which was pretty OK. Nice if you've got .NET stuff in the mix. We do sensitive R&D work so it has to be in-house. So there.
|
|
#
?
Sep 17, 2018 07:47
|
|
|
Guys, I think Jenkins might be actually good? At least, the Blue Ocean stuff is quite nice.
|
|
#
?
Sep 18, 2018 19:32
|
|
|
ratbert90 posted:Guys, I think Jenkins might be actually good? The Blue Ocean stuff is incredibly hard to read from a devops perspective (in my opinion) and sometimes people will send me links that 404 when I open them. I'd rather vanilla Jenkins.
|
|
#
?
Sep 18, 2018 19:43
|
|
|
|
| # ? May 15, 2024 20:22 |
|
|
I like the look of blue ocean, especially for quick visual reporting of echo commands in jobs, even if the whole thing is weirdly slow, especially compared to the beta. My beef is that file upload from parameters is STILL broken in pipeline. C'mon guys this is basic jenkins functionality. https://issues.jenkins-ci.org/browse/JENKINS-27413?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true
|
|
#
?
Sep 18, 2018 19:53
|
|
