Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Corla Plankun
May 8, 2007

improve the lives of everyone

elcannon posted:

The guy who wrote it was one of the cofounders and sole developer until a couple months ago when he supposedly just stormed out after some dispute. They picked up a couple contract devs to try to put out a bunch of existing fires so I'm supposed to be out before the end of the year but we'll see. His work has really cemented my belief that code is a liability and not an asset though. Guy wrote his own installer, logger, CRM, helpdesk system, billing system, job application system and it all exists across like 3 assemblies so when the job that parses the resumes to the open HR position fails for whatever reason then there's a good chance that we won't receive emails about errors or someone won't get billed etc.

I can't complain really, getting paid to push buttons and all, and I regularly get to make some really delete heavy commits, like removing almost 100 instances of this:
code:
	//this is a space
	private char SPACE												= Char.Parse(" ");

this kind of job is honestly my dream

dont have to go to any loving meetings or plan new greenfield bs, just investigate hosed up things and reimplement them with sane design patterns

Adbot
ADBOT LOVES YOU

eschaton
Mar 7, 2007

Don't you just hate when you wind up in a store with people who are in a socioeconomic class that is pretty obviously about two levels lower than your own?

elcannon posted:

Guy wrote his own installer, logger, CRM, helpdesk system, billing system, job application system and it all exists across like 3 assemblies so when the job that parses the resumes to the open HR position fails for whatever reason then there's a good chance that we won't receive emails about errors or someone won't get billed etc.

your company has The Application

The Application does all

all bow to The Application

Sapozhnik
Jan 2, 2005

Nap Ghost
there is indeed something rather soothing about fixing lovely code written by some loving idiot

sometimes the loving idiot is me from six months ago

mystes
May 31, 2006

Chalks posted:

Might be some prevent default stuff to do to solve it - either that or we're in the territory of trying to use CSS to implement password masking on a text box which I'd rather stay the hell away from.
At least this would be a much more legitimate reason than the people making fake password fields to try to get around the unencrypted http warnings.

The MUMPSorceress
Jan 6, 2012


^SHTPSTS

Gary’s Answer

Corla Plankun posted:

this kind of job is honestly my dream

dont have to go to any loving meetings or plan new greenfield bs, just investigate hosed up things and reimplement them with sane design patterns

this was literally my job description at epic and while in a vacuum it might sound fun, the reality is that the software wouldn't be in that state if the business itself wasn't also terrible. your actual work might be fun but your life will be existential misery.

bob dobbs is dead
Oct 8, 2017

I love peeps
Nap Ghost

cis autodrag posted:

this was literally my job description at epic and while in a vacuum it might sound fun, the reality is that the software wouldn't be in that state if the business itself wasn't also terrible. your actual work might be fun but your life will be existential misery.

wasn't the reason your life was existential misery that the company was poo poo about gendering you correctly, or am i thinking of someone else

The MUMPSorceress
Jan 6, 2012


^SHTPSTS

Gary’s Answer

bob dobbs is dead posted:

wasn't the reason your life was existential misery that the company was poo poo about gendering you correctly, or am i thinking of someone else

trans issues were a small part of a massive cosmos of problems with that company. my point is that crusty poo poo software comes from crusty poo poo environments.

kitten emergency
Jan 13, 2008

get meow this wack-ass crystal prison

cis autodrag posted:

this was literally my job description at epic and while in a vacuum it might sound fun, the reality is that the software wouldn't be in that state if the business itself wasn't also terrible. your actual work might be fun but your life will be existential misery.

yeah it sucks. things that are in a bad state don't get there because everyone is incompetent, it's because the organization doesn't value those things. if they did, those things wouldn't be in a bad state.

Stereotype
Apr 24, 2010

College Slice

cis autodrag posted:

this was literally my job description at epic and while in a vacuum it might sound fun, the reality is that the software wouldn't be in that state if the business itself wasn't also terrible. your actual work might be fun but your life will be existential misery.

The main problem with epic is that judy is literally an insane person and everyone who works there is a child

Stereotype
Apr 24, 2010

College Slice
"oh me? my small windowless cube is in the wizard building. I'm jealous of the people in the cowboy building though, they just seem so cool" *whizzes away in Tesla for 40 minute commute to willy st.*

a witch
Jan 12, 2017

everything I’ve heard about Epic makes it sound like a cult.

Stereotype
Apr 24, 2010

College Slice

a witch posted:

everything I’ve heard about Epic makes it sound like a cult.

Its a funnel for government and old people money into the hands of crazy, youthful, and artistic people so I'm basically okay with it. I'm sure it sucks to work there but as an outsider who goes in and surprises friends when I'm in town and gets a cheap lunch, it is a phenomenally cool campus. It is amazing what you can do when you spend a billion dollars on themed buildings and art.

The MUMPSorceress
Jan 6, 2012


^SHTPSTS

Gary’s Answer

Stereotype posted:

Its a funnel for government and old people money into the hands of crazy, youthful, and artistic people so I'm basically okay with it. I'm sure it sucks to work there but as an outsider who goes in and surprises friends when I'm in town and gets a cheap lunch, it is a phenomenally cool campus. It is amazing what you can do when you spend a billion dollars on themed buildings and art.

Ya, my office is the only thing I miss from there.

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat
ive been fired from better places for less

Luigi Thirty
Apr 30, 2006

Emergency confection port.

okay cool, the perspective math was correct. the rotation matrix math was hosed up.



well... it still is since it crashes when we flip past 90 degrees, but at least the universe box isn't being squished anymore

cinci zoo sniper
Mar 15, 2013




what's the least painful way to sync a folder from a git repo (on gitlab, likely ce) to another git repo hosted in the same place, in a way that it stays in sync? basically my job has had a brief moment of responsibility so they cant let me see the entire monorepo, but there are parts that i absolutely should see, and it is more or less onto me to figure out if there's a better solution that then cloning the folder and me manually pestering the it to update it every now and then

JawnV6
Jul 4, 2004

So hot ...

Luigi Thirty posted:

okay cool, the perspective math was correct. the rotation matrix math was hosed up.



well... it still is since it crashes when we flip past 90 degrees, but at least the universe box isn't being squished anymore

does that have two vertices ending up on the same pixel?

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


cinci zoo sniper posted:

what's the least painful way to sync a folder from a git repo (on gitlab, likely ce) to another git repo hosted in the same place, in a way that it stays in sync? basically my job has had a brief moment of responsibility so they cant let me see the entire monorepo, but there are parts that i absolutely should see, and it is more or less onto me to figure out if there's a better solution that then cloning the folder and me manually pestering the it to update it every now and then

this vaguely sounds like it might be a case for a git submodule but idk how those really work if i'm honest, just that they exist and have something to do with splitting up a git repo by folders

Ciaphas fucked around with this message at 09:31 on Nov 30, 2017

Workaday Wizard
Oct 23, 2009

by Pragmatica
anyone here familiar with The Twelve-Factor App? is it achievable or too wishy washy?

MononcQc
May 29, 2007

Shinku ABOOKEN posted:

anyone here familiar with The Twelve-Factor App? is it achievable or too wishy washy?

It is achievable, but only for some classes of applications I think (web front-ends). Most of the time someone could get 8-10 out of 12 comfortably.

kitten emergency
Jan 13, 2008

get meow this wack-ass crystal prison

Shinku ABOOKEN posted:

anyone here familiar with The Twelve-Factor App? is it achievable or too wishy washy?


MononcQc posted:

It is achievable, but only for some classes of applications I think (web front-ends). Most of the time someone could get 8-10 out of 12 comfortably.

idk, i'd be a bit more generous than this. it takes a lot of rigor to make any sort of complex application a 12-factor app, though. honestly, it's probably not worth it in most cases.

that said, k8s and whatnot make it a more achievable since you already have an answer to 'how do i deploy all of these microservices in a sane fashion and have them communicate with each other'

cinci zoo sniper
Mar 15, 2013




c tp s: gently caress, i am developing an r shiny app for the prod after all ;_;

cinci zoo sniper
Mar 15, 2013




can someone help me out with minimum feature checklist for "analytics dashboard web microservice"?

im thinking:

1) incremental data load
2) data validation
3) basic auth
4) graphs and poo poo
5) logging of everything

my questions i probably have some idea about :

1) what are the key things to log
2) how bad is an idea to store that in rds, or, say, tsv, and incrementally append lines to it?

my question i have no loving idea about :

1) how do i prepare this for deployment with little it involvement? i know that our it guys have discovered docker a month ago, and are now experimenting with it . right now, i think i am looking at maybe 5 docker containers where 1 is proxy and 4 are apps, one per each major data source. is that just some docker containers for deployment? a docker gaggle swarm? kubernetes or some other buzzword?? assume that every party involved in this process of product hand off and deployment is extremely incompetent (you dont even have to assume with me, theres ample evidence on most pages itt)

cinci zoo sniper
Mar 15, 2013




also, whats a git-friendly note taking tool? is there something like poor man's onenote with markdown?

DONT THREAD ON ME
Oct 1, 2002

by Nyc_Tattoo
Floss Finder

cinci zoo sniper posted:

also, whats a git-friendly note taking tool? is there something like poor man's onenote with markdown?

org mode

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender

Shinku ABOOKEN posted:

anyone here familiar with The Twelve-Factor App? is it achievable or too wishy washy?
It's totally achievable and pretty much all the principles are necessary to do containerization well. In my experience most of the checkboxes were pretty easy to tick, but some of the items contain multitudes that you only find out from experience.

> Store config in the environment

This refers to anything you might want to change at launch time. It's ok to bake in (say) Apache/Nginx/PHP server config to the image at build-time. But that can complicate things too: Say you need to be able to configure Apache server var $X which it reads out of its config file. If that config file is baked in, then your container entrypoint needs to be a wrapper script that parses $X from the env and overwrites it in the Apache config file before launching Apache. But don't forget to do that with an "exec /usr/bin/apache2" to replace the wrapper script instead of just "/usr/bin/apache2", because otherwise your wrapper script will remain as the top level PID which can cause problems when debugging and sending signals.

It's subtle annoying stuff like that which makes containerization a pain, but modern apps are becoming more container-friendly so they'll offer ways to override config via the environment.

I personally think passing in all config via the environment doesn't really scale and it's often better to have the app retrieve its config from some external network service at launch, but that's a whole other kettle of fish.

> Execute the app as one or more stateless processes

Pretty straightforward for many apps, but obviously a bit tricky for databases since their whole purpose is to store state. I had a heck of a time containerizing PostgreSQL. Also decisions need to be made like: is my DB container a generic all-purpose DB instance that's empty at startup, or one that contains my app's schema? If the former, how does my app's schema get inside the DB, and how is it updated, e.g. does my app have to connect tot the database and then create the schema itself if it doesn't exist? If the schema is baked into the container, how do schema changes work?

Not to mention that some DBs need sidecar containers to run cronjobs like table vacuuming and log shipping.


cinci zoo sniper posted:

can someone help me out with minimum feature checklist for "analytics dashboard web microservice"?
What's the high level goal here? What are some of the use-cases?

cinci zoo sniper posted:

i know that our it guys have discovered docker a month ago, and are now experimenting with it .
Docker's very easy to get into as a dev, but that belies a whole host of issues you'll see once you try to build a CI/CD pipeline and get results into production. It can take a looong time to learn all those idiosyncrasies.

quote:

assume that every party involved in this process of product hand off and deployment is extremely incompetent
I hate to say it, but while Kubernetes and Docker et al are pretty amazing, I firmly believe that the people running it need to have a rock solid understanding of how it all works. It's hella complex and you need those fundamentals when poo poo goes wrong. If everyone involved is a clownshoe then it's just gonna be way painful.

cinci zoo sniper
Mar 15, 2013




minato posted:

What's the high level goal here? What are some of the use-cases?

Docker's very easy to get into as a dev, but that belies a whole host of issues you'll see once you try to build a CI/CD pipeline and get results into production. It can take a looong time to learn all those idiosyncrasies.

I hate to say it, but while Kubernetes and Docker et al are pretty amazing, I firmly believe that the people running it need to have a rock solid understanding of how it all works. It's hella complex and you need those fundamentals when poo poo goes wrong. If everyone involved is a clownshoe then it's just gonna be way painful.
concerning but more or less as expected, maybe except the last paragraph.

so, the use case etc

4 isolated shiny apps. always online, incrementally load data overnight from our aws infrastructre, serve several dozen of visualisations each for up to some three dozen concurrent users.

since the post you quote, ive found a seemingly decent way to make a docker image packaged with the apps.

i still know nothing about deployment, and from recent chat with the it and pms it seems that our techs are in early investigation phase either.

so, i assume my options are
1) one or more industry standards (i hope there are some for the dumb and the desperate too)
2) shinyproxy that shits out a container per app per use which seems like a terrible loving idea unless i redo entire backend and delegate the data gathering to something entirely else, which doesn't seem to be feasible

oh and random, stupid question - docker containers have persistent storage right?

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat

minato posted:

I personally think passing in all config via the environment doesn't really scale and it's often better to have the app retrieve its config from some external network service at launch, but that's a whole other kettle of fish.

Sure, but the config values you pass in via the environment would be the location of the service, like the host/port of your zookeeper instance or whatever

DELETE CASCADE
Oct 25, 2017

i haven't washed my penis since i jerked it to a phtotograph of george w. bush in 2003

cinci zoo sniper posted:

c tp s: gently caress, i am developing an r shiny app for the prod after all ;_;

no no stop that no

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender

cinci zoo sniper posted:

so, the use case etc
I'm still kinda fuzzy on what you're trying to achieve, so if I understand you correctly, you have 4 apps that slurp in data from AWS and shove it into some local database, and another service that slurps data out of that database to present data visualizations?

If that's the case, then the Standard Way Of Doing It is to keep all 6 services (4 data scrapers + 1 database + 1 visualization front end) separated into different container images. You generally want a single process per container image, because the smaller unit size makes it easier to build, debug, and horizontally scale. Then you'll use something like "docker compose" to spin up the 6 containers and wire them together.

AIUI docker compose is more a dev tool than a production tool, so production would be launched differently.

By default a container has no persistent storage. And in environments where you're never sure which host your container will be spun up on, file-based storage can be a pain. So that's why most 12-factor apps try to do all their storage over the network, and that way the pain is at least centralized to the one database container that has to talk to a disc.

For containers that need storage, the 2 routes are either to use Docker volumes or to bind mount in some storage directory from the host. In my experience, bind mounting is the way to go. Docker volumes were convenient to use but the directories created on the host are dynamically named which can make some external tooling more painful to write. With bind mounts you know exactly where the directories are on the host, and you explicitly specify them to the container at launch. Docker volumes were also kind of buggy and weird when I played around with them, but that was a couple of years ago so maybe things have improved.

cinci zoo sniper
Mar 15, 2013




DELETE CASCADE posted:

no no stop that no

i would never start that, but i have 2 week deadline with 70-80% completed app handed to me

Sapozhnik
Jan 2, 2005

Nap Ghost
ur taking on a payday loan of technical debt

cinci zoo sniper
Mar 15, 2013




minato posted:

I'm still kinda fuzzy on what you're trying to achieve, so if I understand you correctly, you have 4 apps that slurp in data from AWS and shove it into some local database, and another service that slurps data out of that database to present data visualizations?
sorry, let me try once more. we have 4 existing sql databases running on aws, to all of which same kind of stats can be applied. hence, i have 4 apps that each:
  • incrementally fetch data from their respective db on a regular basis
  • incrementally write the fetched data into a local file
  • process this file in R
  • display users the processing results as visualiastions
this is done as a temporary stop-measure to make management shut up until we decide which bi suite we would like to drop stacks on.

minato posted:

If that's the case, then the Standard Way Of Doing It is to keep all 6 services (4 data scrapers + 1 database + 1 visualization front end) separated into different container images. You generally want a single process per container image, because the smaller unit size makes it easier to build, debug, and horizontally scale. Then you'll use something like "docker compose" to spin up the 6 containers and wire them together.

AIUI docker compose is more a dev tool than a production tool, so production would be launched differently.

By default a container has no persistent storage. And in environments where you're never sure which host your container will be spun up on, file-based storage can be a pain. So that's why most 12-factor apps try to do all their storage over the network, and that way the pain is at least centralized to the one database container that has to talk to a disc.

For containers that need storage, the 2 routes are either to use Docker volumes or to bind mount in some storage directory from the host. In my experience, bind mounting is the way to go. Docker volumes were convenient to use but the directories created on the host are dynamically named which can make some external tooling more painful to write. With bind mounts you know exactly where the directories are on the host, and you explicitly specify them to the container at launch. Docker volumes were also kind of buggy and weird when I played around with them, but that was a couple of years ago so maybe things have improved.
docker compose, separate database for data, and horizontal scaling concerns sound a bit too serious/time consuming for what i can afford. i dont understand the shiny architecture enough to separate the frontend from the backend either.

more specifically about database for data, that would mean that i must build a db grounds up and design scripts and poo poo to push data into it, and i have never done anything more complicated in sql than a select query. hence, i'll consider bind mounting instead, unless i finish everything in under 4 days so i have some 7-8 days remaining for playing a dba. how would a database store its data in a docker container anways, bind mounting again (but just once)?

cinci zoo sniper
Mar 15, 2013




Sapozhnik posted:

ur taking on a payday loan of technical debt

fittingly, ive worked in payday loans :v:

Shaggar
Apr 26, 2006
just write an SSIS package to export the data and kick off the r job.

cinci zoo sniper
Mar 15, 2013




Shaggar posted:

just write an SSIS package to export the data and kick off the r job.

we didn't end up going with microsoft sql server, and that project is maybe half a year out from completion anyways. the principal "dev" on the mssql side of argument turned out to be a bust who had read less about it on msdn than i did after i heard about mssql proposal :v:

Shaggar
Apr 26, 2006
rip

syntaxrigger
Jul 7, 2011

Actually you owe me 6! But who's countin?

cinci zoo sniper posted:

we didn't end up going with microsoft sql server, and that project is maybe half a year out from completion anyways. the principal "dev" on the mssql side of argument turned out to be a bust who had read less about it on msdn than i did after i heard about mssql proposal :v:



shaggar'd

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender

cinci zoo sniper posted:

this is done as a temporary stop-measure to make management shut up until we decide which bi suite we would like to drop stacks on.
In that case I'd step away from docker, it's an interesting and useful tech but it's not gonna solve your immediate problem. Reconsider it when you're much further down the road and have something up and running. It sounds like you have a fairly simply deployment, you could run it with bash scripts at first and then evolve towards RPM/DEB packages and host config software like Chef/Puppet/Ansible.

quote:

more specifically about database for data, that would mean that i must build a db grounds up and design scripts and poo poo to push data into it, and i have never done anything more complicated in sql than a select query.
Choosing the DB store depends on the kind of data you're storing (well-formed structure? random-rear end looking data?), the expected volume of data, and how you expect to search data (arbitrary queries? or just key-value retrieval?). Could be anything from just sticking it into a set of files, to some NoSQL solution, or a full-fledged RDBMS.

I'm personally biased to using an RDBMS like PostgreSQL. Rock solid, excellent docs, will definitely do everything you need. It's got a bit of a shallow learning curve and it might be cracking a walnut with a sledgehammer in this case, but it's an excellent investment of your time for future projects.

quote:

more specifically about database for data, that would mean that i must build a db grounds up and design scripts and poo poo to push data into it, and i have never done anything more complicated in sql than a select query. hence, i'll consider bind mounting instead, unless i finish everything in under 4 days so i have some 7-8 days remaining for playing a dba. how would a database store its data in a docker container anways, bind mounting again (but just once)?
FWIW a database will typically bind-mount in the DB's data directory so that persists when the container instance disappears.

Edit: Caveat here is that the user namespace inside the container is logically separate from the host's users, but from the filesystem's POV the user/group IDs are shared. So unless you're careful, you get issues like when the container runs as root (which is the default, and not recommended) and creating files on the bindmount which are then owned by root/root, so any processes on the host (like backup scripts) won't be able to access it.

minato fucked around with this message at 21:31 on Nov 30, 2017

Adbot
ADBOT LOVES YOU

cinci zoo sniper
Mar 15, 2013




minato posted:

In that case I'd step away from docker, it's an interesting and useful tech but it's not gonna solve your immediate problem. Reconsider it when you're much further down the road and have something up and running. It sounds like you have a fairly simply deployment, you could run it with bash scripts at first and then evolve towards RPM/DEB packages and host config software like Chef/Puppet/Ansible.

Choosing the DB store depends on the kind of data you're storing (well-formed structure? random-rear end looking data?), the expected volume of data, and how you expect to search data (arbitrary queries? or just key-value retrieval?). Could be anything from just sticking it into a set of files, to some NoSQL solution, or a full-fledged RDBMS.

I'm personally biased to using an RDBMS like PostgreSQL. Rock solid, excellent docs, will definitely do everything you need. It's got a bit of a shallow learning curve and it might be cracking a walnut with a sledgehammer in this case, but it's an excellent investment of your time for future projects.

FWIW a database will typically bind-mount in the DB's data directory so that persists when the container instance disappears.

Edit: Caveat here is that the user namespace inside the container is logically separate from the host's users, but from the filesystem's POV the user/group IDs are shared. So unless you're careful, you get issues like when the container runs as root (which is the default, and not recommended) and creating files on the bindmount which are then owned by root/root, so any processes on the host (like backup scripts) won't be able to access it.

i could just get us a vps, that's deffo an option. can say i know much bash, but ive janitored some centos, debian, and ubuntu servers before (im windows guy normally)

data itself is strictly defined and i have no problems describing entire schema in 10-20 minutes tops, variable types and whatnot including, including all the indices we may need. ive just never done it and not sure if now is the appropriate time (+ how many rocks i would hit were i do it on postgresql 10 (we have future deployment on that which i partially oversee, so why not get some knowledge mileage out of this too?))

initial volume of data, across all apps, is around 50 70x10 tables, where one row is one month of data and last row is being rewritten every night.

and yeah so i figured about bind mounting for db container, that was me thinking out loud that id be storing plain files or db the same method (wrt docker infrastructure)

  • Locked thread