|
Does Jamf just pay garbage and work their employees to death? Seems like everyone is collecting their 1 year cliff and walk out the door and never look back Last job we inherited 6-12 Jamf people over a three month period and they were pretty good hires, the last time I saw rats jumping from a sinking ship that fast was Tripwire in 2016
|
#
?
Oct 11, 2021 23:28
|
|
|
|
| # ? Jun 3, 2024 22:12 |
|
|
How do I get pagerduty equivalent functionality if I'm not allowed to use external 3rd party services because of high level fedramp certification reasons. I don't see anything offered by AWS in govcloud that could fill the gap. I'm talking making people's phones ring when things go wrong. I'm imagining a dumb web service that just receives alertmanager posts and proxies them to PD anyway, but that sounds gross and hacky and like it's probably not allowed. I know the answer is probably going to be go talk to my compliance officers and work something out, but I'm wondering if anybody else has already solved this problem. Methanar fucked around with this message at 23:50 on Oct 11, 2021 |
|
#
?
Oct 11, 2021 23:44
|
|
|
Talk to your compliance officer. Get in touch with an account rep at pager duty and see if you can get them onboarded, with a plan to switch over to pagerduty's fedramp-compliant system when it's certified (quick glance says this is in progress). Ideally you get a provisional green light from your CO and then transition to fedramp-compliant when that becomes available, I guess? You could try rolling your own pager duty with SNS but that sounds like a disaster in the making
|
|
#
?
Oct 12, 2021 00:00
|
|
|
Hadlock posted:Talk to your compliance officer. Get in touch with an account rep at pager duty and see if you can get them onboarded, with a plan to switch over to pagerduty's fedramp-compliant system when it's certified (quick glance says this is in progress). Ideally you get a provisional green light from your CO and then transition to fedramp-compliant when that becomes available, I guess? You could try rolling your own pager duty with SNS but that sounds like a disaster in the making https://www.osec.doc.gov/opog/privacy/NOAA%20PIAs/NOAA0900_PIA_SAOP_Approved_FY21.pdf Supposedly PD is good enough for the depeartment of commerce quote:The single external service for the Smartsheet Gov. environment is PagerDuty. Smartsheet I'm definitely trying to avoid rolling my own PD.
|
|
#
?
Oct 12, 2021 00:13
|
|
|
Yeah get an account rep on the horn and I'm sure they have an internal fedramp specialist that can work with you. Pagerduty absolutely wants federal business because they always pay on time and they're very likely a customer for life, so they'll probably throw in 100 hours of sales engineer time to secure the deal if you ask for it
|
|
#
?
Oct 12, 2021 00:23
|
|
|
 I just posted a long term contractor devops job opening over in the jobs thread, aws/terraform/pythonistas take note, hiring yesterday. You'd be working with me and able to correct all my bad opinions in real timehttps://forums.somethingawful.com/showthread.php?threadid=3075135&pagenumber=113&perpage=40&userid=0#post518413722 Edit: is open to Canadian and Mexican goons as well, we just hired a Mexican full stack contactor last week. Primary team is in SF CA but it's FTR Hadlock fucked around with this message at 01:32 on Oct 12, 2021 |
|
#
?
Oct 12, 2021 01:28
|
|
|
Blinkz0rz posted:What's the current approach in terms of k8s and organizing it around applications: one giant cluster that houses everything or a bunch of smaller clusters focused around domains? My workplace runs multiple clusters, we've got a big kitchen sink cluster for small stuff, like random API's and smaller apps, but our larger applications get their own dedicated clusters.
|
|
#
?
Oct 12, 2021 01:38
|
|
|
Blinkz0rz posted:What's the current approach in terms of k8s and organizing it around applications: one giant cluster that houses everything or a bunch of smaller clusters focused around domains? All my stuff goes in either the production or pre-production cluster, as appropriate for Region, unless the service accounts have goofy-rear end RBAC requirements (AWX/dapr/etc) - if you do that you need to become positively hitleresque with admission controllers (eg OPA Gatekeeper and publish your policies on a company-readable git repo) and it'd be reasonable to enforce repo access w/ something like artifactory e: additionally, tainted node-pools for heavy compute teams are your friend and you are going to want to use something like kubecost for chargebacks/showbacks - learn to love the poo poo out of metadata Junkiebev fucked around with this message at 03:43 on Oct 12, 2021 |
|
#
?
Oct 12, 2021 03:40
|
|
|
spinnaker (and all associated ecosystems and tooling and hosed up multi-disciplinary workflows derived therefrom all of which grew organically by many different people on a time budget of zero over the course of several years) BAD
Methanar fucked around with this message at 23:00 on Oct 15, 2021 |
|
#
?
Oct 15, 2021 22:54
|
|
|
I've talked to a lot of people, everyone seems to know about spinnaker, none of them have actually seen it put in to production anywhere for more than a year Jenkins seems to scratch that perfect confluence of "dead-simple task scheduler, extremely extensible, but also maintainable by people of mediocre skill with very little systems knowledge" which is really what most orgs want/need
|
|
#
?
Oct 15, 2021 23:03
|
|
|
Hadlock posted:I've talked to a lot of people, everyone seems to know about spinnaker, none of them have actually seen it put in to production anywhere for more than a year I'm trying to replicate Spinnaker functionality in a new fedramp environment and jesus christ you wouldn't believe the skeletons I'm finding. None of this poo poo has time budgetted for it, and none of its going to work even if I do a direct lift and shift for a long list of complicated (stupid) reasons. And even if it does work, it won't pass compliance. lifting and shifting, which is the simplest way of 'completing' the ticket, and leaving it broken for later is probably what I'm going to end up doing because I have negative 2 days left to finish this (huge fuckoff blocker that is cutely titled as "Build Spinnaker" with no description in the ticket body) abomination. Nobody actually owns spinnaker nor has put any effort into maintaining the many different spinnakers we have in any consistent fashion again for a long list of complicated (stupid) reasons. I was supposed to fix this a long time ago but for 8 months there has been a persistent failure of team to do thing that is an early pre-requisite in that long list. So now here I am building another snowflake spinnaker as a result. Methanar fucked around with this message at 23:26 on Oct 15, 2021 |
|
#
?
Oct 15, 2021 23:23
|
|
|
Methanar posted:I'm trying to replicate Spinnaker functionality in a new fedramp environment lmfao would have loved to be a fly on the wall in the room when your manager was told that this was the best use of your time
|
|
#
?
Oct 15, 2021 23:35
|
|
|
Hadlock posted:lmfao To be clear I am not attempting rewrite spinnaker. I'm taking the main production non-govcloud spinnaker( and associated workflows) and have it also be in the new fedramp environment. Which is not even remotely close to being as simple as it might sound on the surface. Because guess what, for one small example out of many, fedramp environments can't speak to a git server that's hosted outside of the fed ramp environment.
|
|
#
?
Oct 15, 2021 23:42
|
|
|
ansible tower is extremely overengineered for the average user but it is good at this type of thing, if you can temporarily enable network paths into your fedramp environment you wouldn't even need to launch another tower instance, you just could create an "isolated node" which can accept encrypted jobs from your primary tower deployment broadly though, in my experience, the hardest part about fedramp or working under compliance schemes isn't actually the scheme controls themselves, it's dealing with secops people that don't understand the control or the technology but have some checklist items to enforce and won't be convinced otherwise e: Methanar posted:Which is not even remotely close to being as simple as it might sound on the surface. Because guess what, for one small example out of many, fedramp environments can't speak to a git server that's hosted outside of the fed ramp environment. 12 rats tied together fucked around with this message at 23:50 on Oct 15, 2021 |
|
#
?
Oct 15, 2021 23:45
|
|
|
12 rats tied together posted:ansible tower is extremely overengineered for the average user but it is good at this type of thing, if you can temporarily enable network paths into your fedramp environment you wouldn't even need to launch another tower instance, you just could create an "isolated node" which can accept encrypted jobs from your primary tower deployment ask me about secops people telling me to rebuild every container because alpine and ubuntu base images contain openssl or python 2 or something. I've been told in no uncertain terms that we can't even fully automate artifact transfer into fedramp from non-fedramp without a US citizen inside the US literally clicking a button to accept an import or sync job. Maybe that human button is clicking accept on a jenkins job to where Jenkins s3 sync buckets or whatever. But there needs to be a human clicking button. There's been talk of adopting awx for a long time which would go a long way at maybe fixing some of the processes we have, maybe one day all of this will be less hosed up. But it unfortunately its not today.
|
|
#
?
Oct 15, 2021 23:48
|
|
|
yeah, I don't think that's a real fedramp control, but it doesn't matter I guess if you do switch to awx, probably go with the actual Tower subscription because it's not really that expensive, and Red Hat has a vested interest in finding/helping people use Tower to do dumb poo poo like talk to fedramp stuff or help retailers manage device configs in their 40,000+ brick and mortar locations in this specific scenario you would create a Workflow (series of tower jobs that link together). one of the Workflow Nodes would be an Approval Node (someone has to click yes). all Workflow Nodes are RBAC Items so you can restrict who is allowed to click yes on them, probably based on some "US_CITIZEN" user attribute from your SSO provider
|
|
#
?
Oct 15, 2021 23:54
|
|
|
Methanar posted:To be clear I am not attempting rewrite spinnaker. I'm taking the main production non-govcloud spinnaker( and associated workflows) and have it also be in the new fedramp environment. ah my only advice then, is to go back to your manager and do a deep dive planning session(s) with other members of your team. what you're describing sounds like at minimum, a quarter's worth of full time, heads down work for a single FTE, probably more, especially if you're getting break-fix work from the secops people to reengineer base containers
|
|
#
?
Oct 16, 2021 00:02
|
|
|
Methanar posted:spinnaker (and all associated ecosystems and tooling and hosed up multi-disciplinary workflows derived therefrom all of which grew organically by many different people on a time budget of zero over the course of several years) BAD lmao ok, go off dude Methanar posted:I'm trying to replicate Spinnaker functionality in a new fedramp environment and jesus christ you wouldn't believe the skeletons I'm finding. Ok no this sounds like your company is bad and not spinnaker itself. Have you tried managing upwards and setting expectations on what you can deliver? Hadlock posted:I've talked to a lot of people, everyone seems to know about spinnaker, none of them have actually seen it put in to production anywhere for more than a year I've run Spinnaker in production for about 2 years now. It's not perfect - it's a bit complicated and doing anything in the OSS community is a hassle, but if your job is "do blue/green deploys on EC2", it's great at that. Methanar, have you talked to anyone at Armory or OpsMx about this? I can check around if they've done fedramp if you want if not. For a lot of our SOX/SOC2 we've gotten security to accept "a human on the correct team opening a PR, and another human on the correct team reviewing said PR that, upon merging, does automatic things" is controls enough. AWS also has a ton of people who know a lot about getting you through the correct hoops so you can spend more money with them. edit: Armory totally has done FedRAMP for spinnaker for at least one customer (Lookout) according to their website. Lmk if you want an intro to someone on their team, we've got a great relationship. luminalflux fucked around with this message at 01:23 on Oct 16, 2021 |
|
#
?
Oct 16, 2021 01:16
|
|
|
luminalflux posted:It's not perfect - it's a bit complicated I am a cog in the machine. And have meltdowned enough. The situation cannot be given justice through a something awful dot Com comedy forums post. It's a long tragic multi week story of rush and stress at this point. The expectations are clear. Get it done enough to not be a blocker for everyone else to do their things. It is a project management failure for this situation to be as it is. Ive had conservations around that topic all day. I need to pick my battles and just start with the minimum viable product. And deal with all the broken stuff and reinvention later. It was only a 45 minutes ago that I realized the secops guy who gave me my sso stuff probably made a 3rd mistake in the process because my saml responses don't include what they need to. Secops man is long gone for the weekend. Methanar fucked around with this message at 02:34 on Oct 16, 2021 |
|
#
?
Oct 16, 2021 02:31
|
|
|
We use jenkins to run docker in docker then build an image and push it in one step. I want to separate out the steps, build, add testing on the image, then push. I’m a but confused about how to uhhh target the image within my jenkinsfile. I must be missing something?
|
|
#
?
Oct 16, 2021 12:50
|
|
|
barkbell posted:We use jenkins to run docker in docker then build an image and push it in one step. I want to separate out the steps, build, add testing on the image, then push. I’m a but confused about how to uhhh target the image within my jenkinsfile. I must be missing something? If all the steps are executed on the same machine, you can just tag the image (with a -t option in the build command) and then you can run it for the test step without need to push. If they aren't, or might not be in the future, I'd recommend simply pushing and pulling from a temporary registry/repo/tag. In either case, remember to delete the temporary image at the end of the pipeline.
|
|
#
?
Oct 16, 2021 13:54
|
|
|
You might want to look into buildah/podman to streamline your build process, instead of docker in docker
|
|
#
?
Oct 16, 2021 14:03
|
|
|
Spinnaker is poo poo, at least in the K8s world. Has it even gotten support beyond Jenkins as an execution engine? If I was doing blue green ec2 deploys sure; it’s the best, but the shoe horning done to get it into kubernetes is a loving nightmare. Everyone I know has move onto its spiritual successor argocd.
|
|
#
?
Oct 16, 2021 15:07
|
|
|
We use Spinnaker across the company and it's generally fine. For the products that do EC2 ASGs it works great because beyond initial setup it's a drop-in for Asgard. My product deploys k8s stuff with it and it's a little more complicated than it needs to be but overall not terrible. The worst of it is how we handle manifests: because we're all in on Terraform, we have a Spinnaker provider and use it to create pipelines and deployment manifests from templates and stuff in Terraform. Not ideal but it nicely separates configuration from execution. ArgoCD looks nifty though. Might play around with it a little.
|
|
#
?
Oct 16, 2021 15:28
|
|
|
Blinkz0rz posted:We use Spinnaker across the company and it's generally fine. For the products that do EC2 ASGs it works great because beyond initial setup it's a drop-in for Asgard. It is minus the more advanced pipeline stuff spinnaker can do; better in every way imo.
|
|
#
?
Oct 20, 2021 03:52
|
|
|
I'm maintaining a single-tenant app, and I'm looking for something to track versions of a triple-digit deployment count across a couple of cloud Nomad clusters and a gaggle of on-premise sites. Updating the deployment automation to log what it's deploying is off the table. I could technically poll the Nomad clusters for details, but that's not feasible for on-prem, so ideally I'd like to have the application report its info to a central location during startup. I just don't know if there's anything that only lists out what's running where rather than like, charting requests per second by app version or something. Maybe Office 365 has a web API for pushing data into an Excel sheet?
|
|
#
?
Oct 22, 2021 22:24
|
|
|
pentium166 posted:I'm maintaining a single-tenant app, and I'm looking for something to track versions of a triple-digit deployment count across a couple of cloud Nomad clusters and a gaggle of on-premise sites. Maybe look at registering to consul? https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery https://www.consul.io/api-docs/agent/service#register-service Methanar fucked around with this message at 22:55 on Oct 22, 2021 |
|
#
?
Oct 22, 2021 22:49
|
|
|
CoreOS (now dead, bought by redhat) gave users the ability to subscribe to a channel, alpha/beta/stable. Client services would ping the update api/service which would return the signed url to download the update. You could both remotely see who was using what channel, but also what version each client was on and what percent, as well as update the channel's pinned version based on crudely implemented rbac. Someone liked this enough to create an API compatible open source clone although it's a little rough around the edges https://github.com/coreroller/coreroller This managed our testing/staging updates at a small startup for about six months before upper management agreed to let us use kubetnetes, and used the (probably no longer supported) official CoreOS core update service to manage single tenant apps in prod at another company before that. Some High Value customers got their own channels that were hand managed by customer success engineers as those customers effectively lived outside of our regular release cadence Haven't looked at flatcar in forever but they might have a less janky channel solution, or maybe they abandoned that model completely. I'm not suggesting you use this exact software, just saying this channel/subscriber model has worked for me in the past
|
|
#
?
Oct 22, 2021 22:53
|
|
|
Methanar posted:Maybe look at registering to consul? I'll give Consul a look. Sorting out public connectivity to allow the on-prem sites to hit our existing cluster might be a bit too much to ask for (and I don't know if it's even a good idea), so if I go that route I'll probably have to shadow IT up a separate instance and I've not personally stood up a "production" Consul before. This looks cool, but it seems like it wants to handle more of the lifecycle than I'll be realistically be able to swing right now. If I had carte blanche to gently caress with the deployment setup, I'd try to get the on-prem sites into a hybrid Nomad cluster with our cloud stuff, and then this particular thing would be a non-issue 
|
|
#
?
Oct 23, 2021 07:00
|
|
|
pentium166 posted:This looks cool, but You can accomplish 97% as much with redis*  *and some very strict network rules
|
|
#
?
Oct 23, 2021 10:37
|
|
|
pentium166 posted:I'm maintaining a single-tenant app, and I'm looking for something to track versions of a triple-digit deployment count across a couple of cloud Nomad clusters and a gaggle of on-premise sites. this is literally one of my interview questions. Consul is good for this and what i've used to solve this problem. You can also use a database (SQL or redis), there's stuff like Aurora Serverless that has an HTTP API, or hell even google sheets has an API. You can also have them add the version to their log output, have everything log to a centralized kafka and then build a complicated system based on an abandoned "SQL for Kafka" product for querying "what versions have i seen on which hosts in the last 30 seconds", if you're nasty.
|
|
#
?
Oct 23, 2021 18:29
|
|
|
is a 700 line jenkinsfile normal?
|
|
#
?
Oct 25, 2021 21:47
|
|
|
I would expect 70-350 lines, but if your build process is excessively complex 700 doesn't seem insane. I think once you get to 3000 (1500?) lines you need to think about splitting out your build process further.
|
|
#
?
Oct 25, 2021 21:58
|
|
|
I managed to do some surgery the other week to cut down our circleci yaml from 2100 lines to 1600.
|
|
#
?
Oct 25, 2021 22:15
|
|
|
Check out Jenkins shared library to help reduce or reuse Jenkinsfile logic
|
|
#
?
Oct 26, 2021 02:20
|
|
|
barkbell posted:is a 700 line jenkinsfile normal? I'd rather see a simple Jenkinsfile that calls out to shell scripts or whatever, makes it easier to run those in isolation for debugging. Having too much Jenkins specific stuff in the build process makes it a pain to run outside of Jenkins
|
|
#
?
Oct 26, 2021 20:19
|
|
|
Hey thread, I need some help! One of our customers is complaining that we're using weak TSL 1.2 ciphers. They said we shouldn't be using any that include SHA. this is a screenshot they sent from analyzing our domain with ssllabs.com: now I've gone into a rabbit hole to find out if it's really bad or if they're overreacting. I'm just a simple developer (it's not much but it's honest work) but I'm taking on some devops responsibilities, so I'm trying to solve this. from my research, ciphers that use RSA are vulnerable to ROBOT attacks, unless they have ECDHE or DHE in their name (something about forward secrecy). but that's orthogonal to the SHA thing I guess. we use a cloudflare zone for our domain, and we just use the default ciphers that they give you. i went to fetch the current ciphers that we have with their API in order to remove the bad ones, but if you haven't manually changed them, you just get back an empty list. if these ciphers are so bad, why are they included in the default ciphers for cloudflare zones? should I remove these ciphers that use SHA-1? what about the RSA ROBOT vulns? for info, these are the TLS1.2 ciphers we use as per the ssllabs report: code:
|
|
#
?
Oct 27, 2021 16:21
|
|
|
Depends on what you do and where you use them If it's just a web app you can probably just turn them off and probably less than 5% of users won't be able to connect to your website. That'll require analysis of your logs. 15% of our users don't support anything higher than tls 1.1 so it's a known risk but approved by management. If it's talking to ancient ftp servers you might break compatibility with older/badly configured servers
|
|
#
?
Oct 27, 2021 16:34
|
|
|
I'm not a super expert, but we (and many others, including ssl labs and your customer's security scanning tools) follow NIST guidance on this, check out page 14. tl;dr on that section is that to be up-to-date and modern and as safe as possible, disable everything but the ones explicitly specified in the list on page 16-18, but like Hadlock says it might break some legacy things because old things are only going to support older, less secure protocols. As to the why cloudflare allows older, less secure ciphers by default, the answer is "Cloudflare attempts to provide compatibility for as wide a range of user agents (browsers, API clients, etc.) as possible." It sounds like you may need to strike a balance between potentially supporting older less secure customers and having more security conscious customers be angry at you, though I can say that we've never lost customers by being "too secure" for their lovely IE 7 or whatever connections, we (politely) tell them to upgrade and point to our documentation that shows we only support a specific range of browsers. This kind of hardening isn't unreasonable and the vast majority of connections should support the list of NIST approved ciphers. Bhodi fucked around with this message at 16:50 on Oct 27, 2021 |
|
#
?
Oct 27, 2021 16:39
|
|
|
|
| # ? Jun 3, 2024 22:12 |
|
|
Our site security scans cipher types too and yells at us if a weak one is found.. this happens every few years and it requires us to purge the offending entry from any web servers under our umbrella. 10+ years under this regime and we haven't had a single user complain that something broke. So I think you'll probably be okay cleaning things up.. but maybe send out a warning just for a little CYA. Our current string in apache looks like: protocols: 'all -TLSv1.1 -TLSv1 -SSLv3' allowed ciphers: 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256' On the odd chance you find it helpful.
|
|
#
?
Oct 27, 2021 16:45
|
|
