Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
LochNessMonster
Feb 3, 2005

I need about three fitty


Blue Ocean is pretty nice, too bad Jenkins/Groovy are lovely.

Adbot
ADBOT LOVES YOU

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
Look at all these plebs not using Jenkins Job Builder.

qsvui
Aug 23, 2003
some crazy thing

Gyshall posted:

Look at all these plebs using Jenkins.

Mega Comrade
Apr 22, 2004

Listen buddy, we all got problems!
I will die on this beautiful hill of plugins.

LochNessMonster
Feb 3, 2005

I need about three fitty


Mega Comrade posted:

I will die on this beautiful hill of plugins.

If you are forced to work with it you will. Rather sooner than later.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.
Jenkins is so much easier if you deliberately avoid any plugin you don't absolutely have to use

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Vulture Culture posted:

Jenkins is so much easier if you deliberately avoid any plugin you don't absolutely have to use

I keep saying the same thing about VSTS custom build/release tasks and no one listens to me. I'm currently fighting a battle against someone who wants to basically re-implement the concept of "ARM templates" as VSTS release definitions using tiny custom tasks with embedded ARM templates for every type of Azure resource. So a release definition would end up looking like

- Create virtual network
- Create subnet
- Create keyvault
- Create VM
- Join VM to domain
- Add VM extension
- etc

Instead of:
- Deploy a resource group <pointer to ARM template in source control>

This is as awful an idea as I'm saying it is, right?

LochNessMonster
Feb 3, 2005

I need about three fitty


Vulture Culture posted:

Jenkins is so much easier if you deliberately avoid any plugin you don't absolutely have to use

I’ve taken that particular piece of advice very seriously.

Trying to figure out how groovy uses escape characters in specific corner cases is mindblowing though.

Shell session code:
node {
    echo 'No quotes, pipeline command in single quotes'
    sh 'echo $BUILD_NUMBER'
    echo 'Double quotes are silently dropped'
    sh 'echo "$BUILD_NUMBER"'
    echo 'Even escaped with a single backslash they are dropped'
    sh 'echo \"$BUILD_NUMBER\"'
    echo 'Using two backslashes, the quotes are preserved'
    sh 'echo \\"$BUILD_NUMBER\\"'
    echo 'Using three backslashes still results in preserving the single quotes'
    sh 'echo \\\"$BUILD_NUMBER\\\"'
    echo 'To end up with \" use \\\\\\\" (yes, seven backslashes)'
    sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
    echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
    def foo = 'bar'
    sh 'echo "${foo}"'
    echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
    sh "echo \"${foo}\""
    echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
    sh 'echo \\\\\\"' + foo + '\\\\\\"'
    echo 'We still needed all of that escaping and mixing concatenation is hideous!'
    echo 'There must be a better way, enter dollar slashy strings (actual term)'
    def command = $/echo \\\"${foo}\\\"/$
    sh command
    echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
    def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
    sh vash
    echo 'It still requires escaping the escape but that is just bash being bash at that point'
    echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug

LochNessMonster posted:

I’ve taken that particular piece of advice very seriously.

Trying to figure out how groovy uses escape characters in specific corner cases is mindblowing though.

Shell session code:
node {
    echo 'No quotes, pipeline command in single quotes'
    sh 'echo $BUILD_NUMBER'
    echo 'Double quotes are silently dropped'
    sh 'echo "$BUILD_NUMBER"'
    echo 'Even escaped with a single backslash they are dropped'
    sh 'echo \"$BUILD_NUMBER\"'
    echo 'Using two backslashes, the quotes are preserved'
    sh 'echo \\"$BUILD_NUMBER\\"'
    echo 'Using three backslashes still results in preserving the single quotes'
    sh 'echo \\\"$BUILD_NUMBER\\\"'
    echo 'To end up with \" use \\\\\\\" (yes, seven backslashes)'
    sh 'echo \\\\\\"$BUILD_NUMBER\\\\\\"'
    echo 'This is fine and all, but we cannot substitute Jenkins variables in single quote strings'
    def foo = 'bar'
    sh 'echo "${foo}"'
    echo 'This does not interpolate the string but instead tries to look up "foo" on the command line, so use double quotes'
    sh "echo \"${foo}\""
    echo 'Great, more escaping is needed now. How about just concatenate the strings? Well that gets kind of ugly'
    sh 'echo \\\\\\"' + foo + '\\\\\\"'
    echo 'We still needed all of that escaping and mixing concatenation is hideous!'
    echo 'There must be a better way, enter dollar slashy strings (actual term)'
    def command = $/echo \\\"${foo}\\\"/$
    sh command
    echo 'String interpolation works out of the box as well as environment variables, escaped with double dollars'
    def vash = $/echo \\\"$$BUILD_NUMBER\\\" ${foo}/$
    sh vash
    echo 'It still requires escaping the escape but that is just bash being bash at that point'
    echo 'Slashy strings are the closest to raw shell input with Jenkins, although the non dollar variant seems to give an error but the dollar slash works fine'
}
This is a crime. Did you have to figure all that out on your own?

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
The issue is Jenkins’ escaping of variables in some cases when using shell scripts. I figured out with a Jenkins issue that one of my shell scripts (not within a shell Jenkins step) was using an AWS_REGION shell variable. Jenkins decided to expand that to a groovy variable that was being used from a library I had written. That just sucked.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
Also, that list of escaping rules is taken from https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4

poemdexter
Feb 18, 2005

Hooray Indie Games!

College Slice
Groovy is fine and good, but Jenkins uses it's own groovy sandbox and definitely likes to gently caress with your strings in bash scripts. I had that gist bookmarked and opened in a tab for almost a year.

LochNessMonster
Feb 3, 2005

I need about three fitty


Bhodi posted:

This is a crime. Did you have to figure all that out on your own?

I found this on github somewhere when I figured out half of it by myself already. I didn’t know about “dollar slashy string”.

I Inherited a (scripted) pipeline that’s mostly shell scripts strung together by groovy functions.

It actually works surprisingly well and I’m perfectly comfortable with shell scripting and the unique features/behaviour you’ll run into with that. But combining it with groovy can only be described as it’s own layer in hell.

kaynorr
Dec 31, 2003

New Yorp New Yorp posted:

I keep saying the same thing about VSTS custom build/release tasks and no one listens to me. I'm currently fighting a battle against someone who wants to basically re-implement the concept of "ARM templates" as VSTS release definitions using tiny custom tasks with embedded ARM templates for every type of Azure resource. So a release definition would end up looking like

- Create virtual network
- Create subnet
- Create keyvault
- Create VM
- Join VM to domain
- Add VM extension
- etc

Instead of:
- Deploy a resource group <pointer to ARM template in source control>

This is as awful an idea as I'm saying it is, right?

Mostly. The thing with ARM templates as that there are some things that are much more easily done in PowerShell than through ARM templates, especially when it comes to configuring things that have already been provisioned. What you'll probably end up with is custom code before the template which sets up all the necessary parameters, and then custom code after the template which configures everything that was just provisioned. Which is still better than a potentially infinite list of custom tasks for every deploy.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

kaynorr posted:

Mostly. The thing with ARM templates as that there are some things that are much more easily done in PowerShell than through ARM templates, especially when it comes to configuring things that have already been provisioned. What you'll probably end up with is custom code before the template which sets up all the necessary parameters, and then custom code after the template which configures everything that was just provisioned. Which is still better than a potentially infinite list of custom tasks for every deploy.

The tasks are just wrappers around ARM templates and invocation of same. I'm pushing for infrastructure-as-code -- ARM templates and deployment scripts live as first-class citizens alongside the application source code, and all we're doing at deployment time is running that stuff. I don't want to tightly couple them to a particular deployment orchestration platform, create a maintenance nightmare, or create a system that makes rapid iteration difficult.

Eggnogium
Jun 1, 2010

Never give an inch! Hnnnghhhhhh!

Vulture Culture posted:

Jenkins is so much easier if you deliberately avoid any plugin you don't absolutely have to use

My year at a game studio using Jenkins was mostly me taking functionality that they had gotten from OSS plugins and reimplementing it in build scripts so that it could be sanely tested/extended.

Jenkins plugins are a cancer and made me despise Java.

Plorkyeran
Mar 22, 2007

To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed
Blue Ocean's pipeline overview display is kinda nice as long as you don't have more than like 10 jobs at any given step of a pipeline. Actually inspecting what happened in a given pipeline step is an utter disaster, though.

Spring Heeled Jack
Feb 25, 2007

If you can read this you can read

New Yorp New Yorp posted:

The tasks are just wrappers around ARM templates and invocation of same. I'm pushing for infrastructure-as-code -- ARM templates and deployment scripts live as first-class citizens alongside the application source code, and all we're doing at deployment time is running that stuff. I don't want to tightly couple them to a particular deployment orchestration platform, create a maintenance nightmare, or create a system that makes rapid iteration difficult.

On this note, is anyone here using Octopus to deploy Azure resources, either via ARM templates or otherwise?

We’re in the process of building a mobile app that will most likely run in app service with some API gateway, service bus, and CosmoDB components. Just trying to figure out the best way (if there is one) to go the IaC route from the beginning.

12 rats tied together
Sep 7, 2006

I've been using AWX as a replacement for Jenkins and so far have been pretty happy with it.

New Yorp New Yorp posted:

The tasks are just wrappers around ARM templates and invocation of same. I'm pushing for infrastructure-as-code -- ARM templates and deployment scripts live as first-class citizens alongside the application source code, and all we're doing at deployment time is running that stuff. I don't want to tightly couple them to a particular deployment orchestration platform, create a maintenance nightmare, or create a system that makes rapid iteration difficult.

Pretty much exactly this except with CloudFormation and sometimes we'll trigger a post deploy config playbook.

e: in our case instead of powershell ansible will ssh to the node and do normal ansible things so I get to take advantage of the several hundred thousands of lines of code other people have already written instead of writing any powershell or python of my own

FlapYoJacks
Feb 12, 2009
The Jenkinsfile I created is pretty straightforward. It has a few checks for branches, but other than that, what I use it for is calling out a jenkins_pipeline.py script I made with various arguments that does all the real heavy lifting.

Currently, there is 9 stages, but I will be going up in steps pretty soon to include pylint and c++ code coverage as well.

The only real pain point was writing a wrapper around meson tests to spit out a JUnit compatible XML file. However, it seems to work well. :shrug:

FlapYoJacks fucked around with this message at 14:45 on Sep 20, 2018

Pile Of Garbage
May 28, 2007



12 rats tied together posted:

I've been using AWX as a replacement for Jenkins and so far have been pretty happy with it.

This seems kinda jacked. Ansible is a configuration management solution, not CI/CD.

Also for those using Jenkins you should be using this feature, it's brilliant: https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines

Helianthus Annuus
Feb 21, 2006

can i touch your hand
Grimey Drawer

quote:

Only entire pipelines can be defined in shared libraries as of this time. This can only be done in vars/*.groovy, and only in a call method. Only one Declarative Pipeline can be executed in a single build, and if you attempt to execute a second one, your build will fail as a result

i read this in darth vaders voice for some reason

Methanar
Sep 26, 2013

by the sex ghost
Does anyone use ksonnet for handling internal Kubernetes configurations?

I don't want to be maintaining 20 different helm charts that are 90% identical for deploying applications.

Qtotonibudinibudet
Nov 7, 2011



Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

Methanar posted:

Does anyone use ksonnet for handling internal Kubernetes configurations?

I don't want to be maintaining 20 different helm charts that are 90% identical for deploying applications.

But having to deal with lots of slightly different YAML files for deployments is exactly the problem Helm ostensibly solves!

In vaguely-related news, the GCP ingress controller is real bad. Arbitrary healthcheck requirements that more or less ignore your liveness/readiness probes are great fun, especially when there aren't any error logs to indicate why they've failed (attempting to mimic them works great!) and healthcheck success apparently varies based on things other than the Pod configuration (gently caress knows why, but changing the grouping of Ingress objects and rules within them will change whether the healthcheck for an individual rule fails).

Also, it takes something like 20 minutes to provision one, so blind testing takes hours. The end customer wasn't open to using the nginx IC because it's "not the default" (because apparently portability across Kubernetes providers is somehow a downside).

12 rats tied together
Sep 7, 2006

cheese-cube posted:

This seems kinda jacked. Ansible is a configuration management solution, not CI/CD.

Also for those using Jenkins you should be using this feature, it's brilliant: https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines

An old version of the ansible website tried to skip the comparison to chef etc by proudly displaying "Ansible is task orchestration".

One of the tasks you can orchestrate is configuring a server. Another is rolling updates. A quick trip through the documentation these days reveals tons of features explicitly for tasks relevant to CI/CD.

The plus side is you can also use it for config management. Pretty much the only thing I wouldn't do with ansible in 2018 is compile code. Everything else goes straight into AWX.

Pile Of Garbage
May 28, 2007



The architecture of Ansible is entirely different to that of an actual CI/CD tool, specifically that it performs operations against hosts instead of code. I agree that it can functionally perform many of the same tasks that a CI/CD tool would however I wouldn't call it a CI/CD tool.

NihilCredo
Jun 6, 2011

iram omni possibili modo preme:
plus una illa te diffamabit, quam multæ virtutes commendabunt

I remember when I first encountered a software, I think it was Bower, describing itself as a "task runner". I was quite a bit baffled: "a 'task runner'? So... you are built for the specific and sole purpose of 'doing things'? Whoah, thank you, it's all clear now."

Pile Of Garbage
May 28, 2007



GitLab mixed things up by just cutting it in half with "Runners"

Pile Of Garbage fucked around with this message at 18:36 on Sep 22, 2018

StabbinHobo
Oct 18, 2002

by Jeffrey of YOSPOS
CI/CD has a slash for a reason. Ansible was named ansible because mpdehaan couldn't name it "func 2.0: gently caress puppet edition". He left puppet labs thinking the declarative model was wrong and what people really wanted was a combined config and deploy tool that simply ran the checklist of steps automatically, and he no longer worked at redhat who held the func name copyright.

It was and is a "CD" tool from inception. No, it is not a "CI" (build) server. Yes its annoying to draw this distinction as you can usually bleed 80% of one side into the other and get-by in a lot of scenarios.

fuckin yoots

StabbinHobo fucked around with this message at 00:36 on Sep 24, 2018

Bruegels Fuckbooks
Sep 14, 2004

Now, listen - I know the two of you are very different from each other in a lot of ways, but you have to understand that as far as Grandpa's concerned, you're both pieces of shit! Yeah. I can prove it mathematically.

NihilCredo posted:

I remember when I first encountered a software, I think it was Bower, describing itself as a "task runner". I was quite a bit baffled: "a 'task runner'? So... you are built for the specific and sole purpose of 'doing things'? Whoah, thank you, it's all clear now."

why do you think every goddamn class ever written ends up inside a xxxmanager eventually? names are hard.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

StabbinHobo posted:

fuckin yoots



Bruegels Fuckbooks posted:

why do you think every goddamn class ever written ends up inside a xxxmanager eventually? names are hard.
Just rename all Managers to Coordinators and refactor the child components until the name is accurate

This pattern works with teams and departments too

Bhodi
Dec 9, 2007

Oh, it's just a cat.
Pillbug
that way lies a factory factory factory

Che Delilas
Nov 23, 2009
FREE TIBET WEED
https://docs.spring.io/spring/docs/2.5.x/javadoc-api/org/springframework/aop/framework/AbstractSingletonProxyFactoryBean.html

official documentation apparently saying this with a straight face posted:

Convenient proxy factory bean superclass for proxy factory beans that create only singletons.

Convenient.

Helianthus Annuus
Feb 21, 2006

can i touch your hand
Grimey Drawer
i;m thinking about thos factory beans

Methanar
Sep 26, 2013

by the sex ghost
How long does everyone's build system take to produce a full build of everything? 5 minutes? 10? 1 hour? How many microservices do you have?

Do you always do full builds of everything or are you more selective in selecting what services to build?

How do you identify artifacts produced, tagging containers with a git sha of the branch built?

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
It's about 10 minutes while a bunch of tests run across like 15 different services (they are not micro any any way) in parallel with some mocked out AWS services and spinning up Docker containers for MySQL. We produce jars and wars in Artifactory. Releases are tagged and we follow git-flow because we're cowards and don't do CD for our SaaS platform because we are bad at tests and because we're really bad at monitoring / metrics so we are not tall enough to ride the microservices hype train.

geeves
Sep 16, 2004

Does anyone have a link or two to a post or blog that deals with gitlab-ci?

I'm not looking for setup, Gitlab's docs are drat good and overall and I think we have everything pretty solid. In fact, we have all of our services building and deploying with ease. What I'm looking for are some "lessons learned"-type posts.

LochNessMonster
Feb 3, 2005

I need about three fitty


A full build for us takes about 10-15 minutes and includes several automated tests qith quality gates and deploying to docker. Artifacts are stored in Nexus and they’re tagged by release number and git commit, I think. Nexus usually ‘just works’ so I don’t touch it all that much.

An inital build of a new branch will take an additional 15-20 minutes since we do a retore of our production data backup so each branch can be tested 100% production-like. This is kept persistant for the duration of the branch. With new builds only the missing incremental backups are restored. The upside is that this tests our backups tens of times per day. The downside is that our NFS server is taking a “Rodney King”-like beating when we spin up too many new builds at the same time.

It also gives me the peace of mind that whatever goes wrong, I can completely rebuild my production environment in less than half an hour with a single button press and without needing anyone elses help.

Still there are lots of things I’d like to improve.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

LochNessMonster posted:

The downside is that our NFS server is taking a “Rodney King”-like beating when we spin up too many new builds at the same time.
come on man

Adbot
ADBOT LOVES YOU

12 rats tied together
Sep 7, 2006

Legacy .net application takes multiple hours to build and sometimes days to successfully deploy.

Stuff that was implemented during my employment here varies based on the stack but it's usually about 5-15 minutes after a merge to master we're running the new code in prod.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply