Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
CPColin
Sep 9, 2003

Big ol' smile.

New Yorp New Yorp posted:

Use SQL Server Data Tools and source control your schema then publish a DACPAC during your deployment process to bring the schema in line with your canonical schema.

Ha ha good one. No, there's an Excel spreadsheet called "Release Checklist" that I'm going to have to put all this stuff in.

Edit: The "read" procedure for this one table has six SELECT statements in it, all selecting the same columns, but changing the WHERE clause, depending on which parameters were specified, via a bunch of nested if-else blocks. :psyduck:

Oh gently caress all the tables are like this! Some of them have a @SortOption parameter and add an ORDER BY clause when it's specified! :suicide:

CPColin fucked around with this message at 22:31 on Nov 3, 2017

Adbot
ADBOT LOVES YOU

Mniot
May 22, 2003
Not the one you know

ChickenWing posted:

How do you lot decide what levels you log information at?

ERROR: Always logged. If a customer opens their logs and sees something logged at ERROR level, expect them to call for support.
WARN: Always logged. Things that might be useful to hear about after someone's called support to report some larger problem. In my application, this is the least-common log-level because things either are working or they aren't.
INFO: Optionally logged. General status information like "connecting to server" or "running garbage collector".
DEBUG: Not logged, except in dev environment. The only limit is what your fellow developers will tolerate.

Hughlander
May 11, 2005

ChickenWing posted:

How do you lot decide what levels you log information at? I'm standardizing our server logs and finding it fairly difficult to come up with even a vague standard for what goes in error/warn/info/debug/trace, especially clarifying boundaries for the latter three.

So far I've settled on:

ERROR: soft/hard workflow failures. Something that was supposed to happen did not.
WARN: recoverable problems and indicators that something is wrong with external flows/internal data validity
INFO: basic "server is doing a thing", endpoint logging, SQL query logging, job health checks, aggregate stats
DEBUG: specific "server is doing a thing", specific stats, reporting non-exceptional exceptions
TRACE: method entry/exit, data dumps


obviously there's no One True Logging Standard, different strategies are applicable for different log consumers, etc - just looking for some wider feedback than the two people that provided input in my office

I had a long write up on my phone but it didn't take. The best way I handled it before was for an event based system where I forced the developers to think of the event as "How should the NOC respond to this event at 3AM?"

So for instance it went:
CRITICAL - Multiple applications greatly impacted / down. Site wide issue.
MAJOR - Single Application greatly impacted.
MINOR - Multiple users impacted.
WARNING - Single users impacted or indication of an upcoming issue.
INFO - Situation normal
DEBUG - Moderate spam
TRACE - Oh god make the spam stop.

I think there was one more between Major/Critical but I don't recall what it was. It was basically one app is completely out but it didn't impact people not on that app. So the SLA was basically
CRITICAL - Page out to primary and primary manager. If not resolved in short order escalate to Director then CIO. Updates every 30 minutes.
MAJOR - Page out to Primary. Escalate after an hour. Updates every hour. After 4 hours would go to the CIO.
MINOR - Page during business hours, E-Mail after hours. Escalate if not resolved after 1 business day.
WARNING - E-Mail if not resolved from runbook after 4 hours hours. Escalate after 5 days.
INFO - Always on but filtered from NOC view, no action.
DEBUG - Can be toggled on as part of a runbook or on-call request.
TRACE - Only turned on by on call in response to a MAJOR/CRITICAL. Could put a big load on the monitoring infrastructure depending on where it's applied.

If you are able to always think about it in the concern of, "This happened, therefore you do...?" you'll have a more useful system in my opinion. And having the developers write the operations facing and the developer facing runbook for everything WARNING and above was good practice for documentation.

lifg
Dec 4, 2000
<this tag left blank>
Muldoon
1990s: Look like somebody has a case of the Mondays.

2000s: (poo poo, I cant say that anymore since Office Space was written to make fun of people like me.)

2010s: Posts Looks like somebody has a case of the Mondays. Office Space meme, but, like, ironically.

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS
Moved to a new team and welp stand-ups are now 15 minutes minimum where we go through currently in-flight tickets and points are time-based with everyone having 20 arbitrarily assigned points per sprint.

Luckily I'm in a position to change this poo poo because goddamn that's bad.

AWWNAW
Dec 30, 2008

Blinkz0rz posted:

Moved to a new team and welp stand-ups are now 15 minutes minimum where we go through currently in-flight tickets and points are time-based with everyone having 20 arbitrarily assigned points per sprint.

Luckily I'm in a position to change this poo poo because goddamn that's bad.

Coworker spotted.

geeves
Sep 16, 2004

Sat in on one of our SRE triage meetings today for someone who was sick today and it happened to be one of those days when we'll invite other non-technical employees to help recreate issues.

Someone was having trouble with a link from a client and I asked them to send it to me via hipchat, sure enough it was broken and I told her to try "https" instead of "http".

"Oh! The plural version of http works! Thank you"

Everyone in the room :stare:

CPColin
Sep 9, 2003

Big ol' smile.
I was in two different meetings today that went long enough for the conference room computer to go to sleep. In the second meeting, my supervisor made one of the action items be figuring out what the next three meetings were going to be.

Meldonox
Jan 13, 2006

Hey, are you listening to a word I'm saying?
I called in to eavesdrop on a meeting today where a bunch of dudes went down a laundry list of features for a prototype that was promised to the client a month ago. Almost none of them were functional aside from some minor React stuff my team pitched in to write. I'm pretty glad I only volunteered for the fun part.

Pollyanna
Mar 5, 2005

Milk's on them.


geeves posted:

Sat in on one of our SRE triage meetings today for someone who was sick today and it happened to be one of those days when we'll invite other non-technical employees to help recreate issues.

Someone was having trouble with a link from a client and I asked them to send it to me via hipchat, sure enough it was broken and I told her to try "https" instead of "http".

"Oh! The plural version of http works! Thank you"

Everyone in the room :stare:

Settle down and eat your HTTPs and gravy :v:

Volmarias
Dec 31, 2002

EMAIL... THE INTERNET... SEARCH ENGINES...

Blinkz0rz posted:

Moved to a new team and welp stand-ups are now 15 minutes minimum where we go through currently in-flight tickets and points are time-based with everyone having 20 arbitrarily assigned points per sprint.

:murder:

That said, I have recently had to complain about how long our standups were taking. In the plus side, everyone was receptive and they got better.

Pollyanna posted:

Settle down and eat your HTTPs and gravy :v:

:golfclap:

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
Just wait till that person finds out about TLS

ChickenWing
Jul 22, 2010

:v:

Blinkz0rz posted:

Moved to a new team and welp stand-ups are now 15 minutes minimum where we go through currently in-flight tickets and points are time-based with everyone having 20 arbitrarily assigned points per sprint.

Luckily I'm in a position to change this poo poo because goddamn that's bad.

AWWNAW posted:

Coworker spotted.

Also are points half a day or are your sprints a month long?

Hughlander
May 11, 2005

ChickenWing posted:

Also are points half a day or are your sprints a month long?

They just dont understand why theres so much carry over each sprint!

ChickenWing
Jul 22, 2010

:v:

My favourite thing about points-as-time is that I am a faster dev than most of my team and so I look really loving good when it's 5 days into the sprint and I've got 8 points of work done.


The absolute best was when my team was at peak overestimation because we had a bunch of bad/lazy devs and I did 29 points of work in a sprint. I got told to chill the gently caress out after that. And then the rest of my team got told that creating a DB table+DAO was not 3 points of work, no matter how much they didn't like talking to the DBA.






Points-as-time is real bad you guys

ChickenWing
Jul 22, 2010

:v:

NB: none of the devs from that team (except the lead, who is amazing) are with the project anymore. This is a very good thing.

LLSix
Jan 20, 2010

The real power behind countless overlords

ChickenWing posted:

My favourite thing about points-as-time is that I am a faster dev than most of my team and so I look really loving good when it's 5 days into the sprint and I've got 8 points of work done.


The absolute best was when my team was at peak overestimation because we had a bunch of bad/lazy devs and I did 29 points of work in a sprint. I got told to chill the gently caress out after that. And then the rest of my team got told that creating a DB table+DAO was not 3 points of work, no matter how much they didn't like talking to the DBA.






Points-as-time is real bad you guys

I've always done estimation using time. What's the benefit of story points?

fluppet
Feb 10, 2009

geeves posted:

Sat in on one of our SRE triage meetings today for someone who was sick today and it happened to be one of those days when we'll invite other non-technical employees to help recreate issues.

Someone was having trouble with a link from a client and I asked them to send it to me via hipchat, sure enough it was broken and I told her to try "https" instead of "http".

"Oh! The plural version of http works! Thank you"

Everyone in the room :stare:

This really gets on my tits, is it really that hard to setup a redirect to https

CPColin
Sep 9, 2003

Big ol' smile.

LLSix posted:

I've always done estimation using time. What's the benefit of story points?

Managers can't translate story points to hours. It's an attempt to force managers to Trust the Process. It never works, because they just fixate on story point velocity anyway.

Hughlander
May 11, 2005

LLSix posted:

I've always done estimation using time. What's the benefit of story points?

A main point of Agile is that through some experimentation you find a way of working that increases your velocity. If velocity is a measure of time and not complexity, what does that mean exactly? "Last sprint we worked 80 hours of points, this sprint we're working 90 hours of points!"
To which the Director cluelessly pipes in with:
"Well hey, if you just stay another hour per day we can get these stories (worth 10 more points) into this Sprint!"

Complexity also handles the cases of the difference between a Junior Developer and a Senior Developer doing the same task. It's a 3 in difficulty. But the Junior Engineer may be pulling 6 points for a Sprint while the Senior is pulling 15. If they were both pulling 80 hours there would be some really funny math going on.

ChickenWing
Jul 22, 2010

:v:

Hughlander posted:

Complexity also handles the cases of the difference between a Junior Developer and a Senior Developer doing the same task. It's a 3 in difficulty. But the Junior Engineer may be pulling 6 points for a Sprint while the Senior is pulling 15. If they were both pulling 80 hours there would be some really funny math going on.

This is the big draw that I've always focused on. A point is a point is a point. Different people can get different amounts of points done in a sprint. Assigning a point to a dev day (or whatever measurement) is inherently flawed because Dev A who just graduated and is brand new to dev probably will get a significantly different amount of work done in a day than Dev B, who has been on this project for two years and knows exactly what he's looking for, what it's doing, what knock-on effects there are, etc.

CPColin
Sep 9, 2003

Big ol' smile.
A short narrative of what just happened today:
  • I go to run some test data through the system.
  • The code fails because it can't find some configuration values it needs.
  • I run the unit tests, which also fail.
  • I look in the database where the configuration sits and everything's gone!
  • I look again a few minutes later and everything's back, except for the configuration my code needs, which I added a little while ago.
  • I add the stuff my code needs and the unit tests pass.
  • I try to run the test data through the system again. It succeeds, but it doesn't show up on Dev.
  • I look more closely at the configuration database. Everything's pointed at Prod and there's absolutely no protection against this situation happening.
  • Somebody now has a Social Security Number of 123-12-1234.
  • Coworker says, "Oh yeah, I copied the missing configuration data from Prod to Dev." and then facepalms when the realization hits.
Whee!

Pollyanna
Mar 5, 2005

Milk's on them.


CPColin posted:

A short narrative of what just happened today:
  • I go to run some test data through the system.
  • The code fails because it can't find some configuration values it needs.
  • I run the unit tests, which also fail.
  • I look in the database where the configuration sits and everything's gone!
  • I look again a few minutes later and everything's back, except for the configuration my code needs, which I added a little while ago.
  • I add the stuff my code needs and the unit tests pass.
  • I try to run the test data through the system again. It succeeds, but it doesn't show up on Dev.
  • I look more closely at the configuration database. Everything's pointed at Prod and there's absolutely no protection against this situation happening.
  • Somebody now has a Social Security Number of 123-12-1234.
  • Coworker says, "Oh yeah, I copied the missing configuration data from Prod to Dev." and then facepalms when the realization hits.
Whee!

One of us ONE OF US ONE OF US :unsmigghh:

KoRMaK
Jul 31, 2012



CPColin posted:

A short narrative of what just happened today:
  • I go to run some test data through the system.
  • The code fails because it can't find some configuration values it needs.
  • I run the unit tests, which also fail.
  • I look in the database where the configuration sits and everything's gone!
  • I look again a few minutes later and everything's back, except for the configuration my code needs, which I added a little while ago.
  • I add the stuff my code needs and the unit tests pass.
  • I try to run the test data through the system again. It succeeds, but it doesn't show up on Dev.
  • I look more closely at the configuration database. Everything's pointed at Prod and there's absolutely no protection against this situation happening.
  • Somebody now has a Social Security Number of 123-12-1234.
  • Coworker says, "Oh yeah, I copied the missing configuration data from Prod to Dev." and then facepalms when the realization hits.
Whee!

Lol

So, you have a config db which dictates where stuff points to, and your coworker made it point dev stuff to prod?


heheh, get wrekt

CPColin
Sep 9, 2003

Big ol' smile.

KoRMaK posted:

Lol

So, you have a config db which dictates where stuff points to, and your coworker made it point dev stuff to prod?


heheh, get wrekt

Yes, via the process of noticing some stuff was missing on Dev, deleting everything that was already there on Dev, to make room, copying everything down from Production, and not changing any of the important stuff. Instead of, you know, copying the one or two rows that were actually missing on Dev.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

CPColin posted:


[*]Somebody now has a Social Security Number of 123-12-1234.

Developers have access to production systems and production systems contain PII? What industry are you working in?

CPColin posted:

Yes, via the process of noticing some stuff was missing on Dev, deleting everything that was already there on Dev, to make room, copying everything down from Production, and not changing any of the important stuff. Instead of, you know, copying the one or two rows that were actually missing on Dev.

Wow. Pulling prod data back into a dev environment for testing is fine, but it should be handled via some sort of well-defined process that does things like, say, update configurations to point to the appropriate environment and anonymize PII.

New Yorp New Yorp fucked around with this message at 22:27 on Nov 7, 2017

a foolish pianist
May 6, 2007

(bi)cyclic mutation

New Yorp New Yorp posted:

Developers have access to production systems and production systems contain PII? What industry are you working in?

This is really common.

CPColin
Sep 9, 2003

Big ol' smile.

New Yorp New Yorp posted:

Developers have access to production systems and production systems contain PII? What industry are you working in?

County government!

New Yorp New Yorp posted:

Wow. Pulling prod data back into a dev environment for testing is fine, but it should be handled via some sort of well-defined process that does things like, say, update configurations to point to the appropriate environment and anonymize PII.

Ha ha good one!

Che Delilas
Nov 23, 2009
FREE TIBET WEED
I have the ability at my company to configure my dev machine (and corresponding in-development code) to point directly to our live database, and so does every other dev. We have an article in our wiki that tells us how. That article mostly consists of sentences like, "If you are thinking about doing this, DON'T. If you really need to, think hard about it, and then DON'T. If you absolutely, positively need to, talk to the senior devs about it, clear it with the boss, and then DON'T."

It's been done a couple of times during my tenure, and only one person has ever wiped out our production database :v: . That one was before my time, but as I understand it, it wasn't particularly fun.

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



I don't have write access to prod here, but I also don't have access to the plan cache or any live performance data except filtered through SolarWinds :geno:

It's for the better no question but when a query starts exploding, optimization is a bit of a guessing game.

DELETE CASCADE
Oct 25, 2017

i haven't washed my penis since i jerked it to a phtotograph of george w. bush in 2003
i have root on the prod db host :getin:

Meldonox
Jan 13, 2006

Hey, are you listening to a word I'm saying?
At my last job our strung out customer support reps had prod database and server access. :toot:

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
I fully expect all my personal info to be compromised having worked for the government. Might as well just put my SSN on my Facebook profile and make it my password for every other account given how incompetent most public sector places are with security (organizationally that is).

Ive had access to prod DBs and the source for most of my jobs, but Ive been an ops guy thats supposed to know wtf Im doing by having root everywhere. Its fine if devs have prod DB access but its not good for it to be that easy to connect to it from goddamn Jenkins.

Portland Sucks
Dec 21, 2004
༼ つ ◕_◕ ༽つ
How should programmatic authentication for internal production systems typically look?

We're a non-customer facing, non-tech oriented manufacturing company with a lot of home grown automation, data collection, and software systems in place.

We have zero sense of security. You can hop on any production server in our plant and dig around and find logins and passwords to all of our databases and servers stored in plaintext in our python scripts, .NET applications, etc... that need to authenticate programmaticly.

Obviously this is a terrible idea. So what is the standard for a situation like this? I'm not familiar at all with secure coding methods or enterprise level credential management so a barney style explanation would be very helpful.

TIA

redleader
Aug 18, 2005

Engage according to operational parameters

CPColin posted:

A short narrative of what just happened today:
  • I go to run some test data through the system.
  • The code fails because it can't find some configuration values it needs.
  • I run the unit tests, which also fail.
  • I look in the database where the configuration sits and everything's gone!
  • I look again a few minutes later and everything's back, except for the configuration my code needs, which I added a little while ago.
  • I add the stuff my code needs and the unit tests pass.
  • I try to run the test data through the system again. It succeeds, but it doesn't show up on Dev.
  • I look more closely at the configuration database. Everything's pointed at Prod and there's absolutely no protection against this situation happening.
  • Somebody now has a Social Security Number of 123-12-1234.
  • Coworker says, "Oh yeah, I copied the missing configuration data from Prod to Dev." and then facepalms when the realization hits.
Whee!

At least you know your backups work!

New Yorp New Yorp posted:

Developers have access to production systems and production systems contain PII? What industry are you working in?

...most of them? I think you'll find that there are more important things than handling PII properly, like features and fixes and sales!

streetlamp
May 7, 2007

Danny likes his party hat
He does not like his banana hat
yeah im a state employee 'web applications engineer' which means im the sysadmin of multiple servers running a bunch of public facing applications and internal apps/tools with dev/staging/prod envs which use a few sql databases which i support. and of course i maintain really really terrible legacy applications, build new ones and break poo poo all the time. a few thousand users every day.

nothing I do requires any PII data, other departments do though and bet your rear end I end up running across it all the time mostly because no one actually keeps track of who was given access to what and are they even still working here and oh some group access role you were given also granted access to way too many other things.

Steve French
Sep 8, 2003

I have full production database access, but not directly via my machine (have to go through a DMZ). I am also in an oncall rotation of ~8 people responsible for responding to and resolving (with delegation if necessary) any and all production issues.

B-Nasty
May 25, 2005

Portland Sucks posted:

How should programmatic authentication for internal production systems typically look?

In Windows land, it's all Active Directory based. Apps shouldn't be using SQL auth to SQL Server, they should be using integrated auth. So, the app pool, Windows service, or scheduled task is run under a particular AD account that has login rights to SQL.

Beyond that, it's mostly your standard need-to-know/least-privilege type access control along with some kind of deployment tool (e.g. Octopus Deploy) that also allows fine-grain permissioning (e.g. devs can deploy to QA, but not PROD.) Another tool which is quite useful is something like Azure KeyVault, where secrets/encryption keys can be retrieved by the code at runtime (via HTTPS calls) instead of floating around in config files somewhere. One could have a DEV KeyVault instance that has dev/preprod secrets and one for PROD. The prod instance would be locked down via certificates only installed on PROD servers, which devs don't have access to.

As a developer, the stance really needs to be that I don't have, nor do I want to have, any access to/knowledge of production systems, prod AD account passwords, or prod-level secrets/keys. It slows things down and it requires IT support that is on-the-ball, but it's the right way to do things.

KoRMaK
Jul 31, 2012



I dont let my devs get prod passwords, and even I hate that i need to get in there sometimes. We have a well defined process to get data from prod to dev/test. It's saved my rear end and our rear end a bunch of times. I've also gotten dangerously close to running a rake command in prod (when I ment to do it in test) when I have test and prod ssh's open.

I habitually close out of prod as soon as I'm done running anything, and rarely leave it open for longer than I am typing.

I've altered our bashrc so that it screams .test at the command prompt now and that helps remind my dumbass a lot.

Adbot
ADBOT LOVES YOU

geeves
Sep 16, 2004

KoRMaK posted:

I dont let my devs get prod passwords, and even I hate that i need to get in there sometimes. We have a well defined process to get data from prod to dev/test. It's saved my rear end and our rear end a bunch of times. I've also gotten dangerously close to running a rake command in prod (when I ment to do it in test) when I have test and prod ssh's open.

I habitually close out of prod as soon as I'm done running anything, and rarely leave it open for longer than I am typing.

I've altered our bashrc so that it screams .test at the command prompt now and that helps remind my dumbass a lot.

I have prod DB keys and I grudgingly hate it (though watching the replicas and watching replication, I don't mind if there's a major bug) but write access? gently caress that. I want everything to go through code review and liquibase to run any thing other than a select on prod.

Hell, there's a non-dev, non-tech person with write access to one or two tables and it drives me up the loving wall that that is allowed. If he wants to query a replica for Gainsight access, fine. But altering a live prod DB. I'm trying to get access revoked but the decision was made above my pay grade and I'm the principal software engineer :/

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply