terrible programmers: my god, it's full of chars

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > YOSPOS > terrible programmers: my god, it's full of chars

«‹›1446 »

the talent deficit: Dec 20, 2003; self-deprecation is a very british trait, and problems can arise when the british attempt to do so with a foreign culture

flyway is pretty good if you don't need rollbacks ever

liquibase kind of sucks but has rollbacks

# ? May 24, 2016 22:47

Adbot: ADBOT LOVES YOU

# ? May 12, 2024 02:14

pokeyman: Nov 26, 2006; That elephant ate my entire platoon.

sweet thanks

# ? May 24, 2016 23:01

HoboMan: Nov 4, 2010

that reminds me: i hung out with our computer janitors today. turns out there's no way to rollback a push to live, lol.

# ? May 24, 2016 23:14

cinci zoo sniper: Mar 15, 2013

MononcQc posted:

Rather than going in and explicitly listing all the relevant cases and finding the edge conditions yourself, you write a specification that says "I expect this type of data" and then a bit of code that says "here's what I expect to be true no matter what data you have". Then the test tools generates all the tests for you and finds edge conditions you wouldn't have thought of.

So coming back to the example in the email, let's say I have a piece of code that finds me the greatest value in a list of value. I could write tests such as:
code:
max_test() {
    assert(312, max([2,9,18,312]);
    assert(0.1, max([-2,-9.0,-18,0,0.1]);
    ...
}
which show that I expect the max value to work with floats and integers and seems to properly cover all cases that I have in mind. The problem is that it doesn't check for a few number stuff that is funky: what about negative and positive 0? Does this handle larger integers fine? What if the list iteration is buggy on lists of items greater than 128 items? Starting to list out all these conditions is costly.

So the thing I can do is find the properties of max($list). The trick is to figure out how to say what it does without the implementation itself. One way would be, for example, to use an existing 'min' function and say that the biggest number is the one for which all the other ones are smaller. Or I could take a sort() function that returns me all the items in order, and then picking the last item means it needs to be the greatest:
code:
max_test() {
    for each $list in <list of numbers generator>:
         max($list) == last(sort($list));
}
What this would be told to do is to randomly generate lists with numbers, and then run the assertion ( max($list) == last(sort($list)) ) against each of them. If any one fails, it complains. That will find a bunch of cases mine didn't carry: what about empty lists? What if the list has a single element but the function expected two? Maybe we can't work with empty lists, so we modify the generator to say <nonempty list of numbers generator> in whatever language notation one has for that.

So what's interesting is that instead of having say, 30 lines of test code for all kinds of interleavings for the max() function, I have 2 effective lines of code that cover the same data. I can ask the test framework to generate longer or more complex cases, or to generate a hundred million variations of them if I want. As iterations go forward, the framework adds complexity to the types in the dataset.

Let's say I have the example of the property above failing on the input:

[120, -2321, -41234.2123, 18.32130, -1, -3412312, 120.0, -3123]

Shrinking (the framework goes back and tries to simplify the input) could produce say:

[120, 120.0]

or even better:

[0, 0.0]

This means that max([0, 0.0]) != last(sort([0, 0.0])). It's not obvious why that is. Maybe the code is wrong, or maybe the test is. In this case, a likely candidate could be that sort() is stable (if two values are compared equal, they remain in the same order in the final list) whereas max() could always favor the leftmost item in the list to be the greatest. This requires me to refine my property: do I want max() to be stable, or do I not care? If I don't care, then I need to modify my property such that either value can be good (make sure 0 == 0.0 is true); if not, then I need to modify my implementation to always return the rightmost operand in the list as greatest.

And that stuff would likely not have been exposed by manual testing, but property-based testing uncovers all kinds of crap like that in code.

The advanced use cases work on the same principle, but instead of having <list of numbers generator>, you generate sequences of commands a more complex state machine would contain (log in, write poo poo post, get suspended, refresh, log out, buy new account, shitpost, etc.) and plug that into the system.

aha, i see whats going on. thanks for clarification!

half of my current thing should be easily testable by means like these, ill scoop around to see how people do this in python in general

# ? May 25, 2016 01:35

MononcQc: May 29, 2007

kalstrams posted:

aha, i see whats going on. thanks for clarification!

half of my current thing should be easily testable by means like these, ill scoop around to see how people do this in python in general

Hypothesis is the name I've seen around in Python, but I never tried it. http://hypothesis.works/articles/intro/

# ? May 25, 2016 01:50

cinci zoo sniper: Mar 15, 2013

MononcQc posted:

Hypothesis is the name I've seen around in Python, but I never tried it. http://hypothesis.works/articles/intro/

bookmarked, will go after i finish some urgent changes. cheers!

# ? May 25, 2016 02:00

quiggy: Aug 7, 2010; [in Russian] Oof.

Star War Sex Parrot posted:

just got handed an old C project with K&R style function declarations

now i'm sad

you just got owned, my friend

# ? May 25, 2016 02:17

Luigi Thirty: Apr 30, 2006; Emergency confection port.

now I can read |-delimited vertices from a text file and draw a wireframe 2D shape to the screen yay. I've been looking up a lot of algebra 1 geometry this week

I think I try making this thing into a scorched earth game if I can figure out animating shots on a parabolic arc

or making pixel art with the VGA palette in an easy to read format

# ? May 25, 2016 02:31

Bloody: Mar 3, 2013

i am working through http://neuralnetworksanddeeplearning.com/ and have written baby's first feedforward neural network. rather than do it in python like the book which would probably amount to running sample code and saying "yup that worked" i am reimplementing it in c# with math.net.

i have successfully created a neural network. unfortunately, it is very stupid, and generally gets stupider as it trains

this maths shall be such fun to debug

# ? May 25, 2016 05:06

Luigi Thirty: Apr 30, 2006; Emergency confection port.

you have created YOSPOS

# ? May 25, 2016 05:13

JawnV6: Jul 4, 2004; So hot ...

Bloody posted:

i am working through http://neuralnetworksanddeeplearning.com/ and have written baby's first feedforward neural network. rather than do it in python like the book which would probably amount to running sample code and saying "yup that worked" i am reimplementing it in c# with math.net.

i have successfully created a neural network. unfortunately, it is very stupid, and generally gets stupider as it trains

this maths shall be such fun to debug

the core math is just floating point multiply and add (maybe a tanh activation function in there somewhere), what are you reimplementing? i'd really hate to rebuild all the bookkeeping around dropout and other training functions

and machine learning is the one place where running sample code doesn't necessarily lead to "yup that worked"

# ? May 25, 2016 05:20

raminasi: Jan 25, 2005; a last drink with no ice

how long does a property test suite take to run

and can I do it in c#

# ? May 25, 2016 05:23

DONT THREAD ON ME: Oct 1, 2002; by Nyc_Tattoo; Floss Finder

if it's broken then your data ismt big enough yet. make your data bigger

# ? May 25, 2016 05:25

Bloody: Mar 3, 2013

i am an idiot and was treating my labels incorrectly! they should represent the encoding of the final output layer of the network, not some arbitrary representation of a thing!

i am a terrible programmer.

# ? May 25, 2016 05:30

the talent deficit: Dec 20, 2003; self-deprecation is a very british trait, and problems can arise when the british attempt to do so with a foreign culture

raminasi posted:

how long does a property test suite take to run

and can I do it in c#

as long as you want and probably not but idk

# ? May 25, 2016 05:31

JawnV6: Jul 4, 2004; So hot ...

autoencoders are unsettling

# ? May 25, 2016 05:40

Progressive JPEG: Feb 19, 2003

JawnV6 posted:

seriously, Jabor's right and they have specified a "header" format if you just wrap messages up like he did. each one gets a tag and length delimiter. you still have to chew through each one, but idk how much more definition you'd need

defining a top level message that's just a repeated submessage worksforme

yeah, if you could keep a cap on the chunk size then i could see that being okay, but say youve got data arriving in chunks 512kb or larger (this will not be uncommon), you'd need to wait for protobuf to chuuug through that whole blob in one go since you don't know beforehand where one submessage ends and another starts

to figure that out you'd need to DIY your own protobuf parser to scan out individual submessages by hand

quote:

really, how difficult is it to provide both the reader & writer schema, versioning them, etc.? your link made it sound like an awful pain

if you use ocf then the header contains the schema, this also makes the data completely self-describing which is p :coal:

, and it means you can have your parsers do all kinds of poo poo dynamically based on the advertised fields

but since fields are defined directly off the schema, names are effectively the field keys, so renaming fields in particular isnt trivial like in protobuf. but even in protobuf that likely means you're needing to simultaneously update any code that touches that field so its not like thats a walk in the park either for anything used by more people than yourself

for my tcp stream thing i just defined it as 'as if piping an ocf file, the sender sends the schema header when the session first opens and all following messages are expected to follow that schema', and the stream is expected to be held open indefinitely so this works great

but if you want to pass around small messages individually then it starts getting messy and the docs get kinda handwavy with 'run a schema repo in a separate channel or something idk'

quote:

i haven't worked on something with the unimaginable luxury of 16mb lying around without having to checksum every 512kb or whatever, but it kinda sounds like you're using it for long term storage as well as encoding? why aren't you using something else as a data store once you're off the streaming device?

the avro data would ultimately be ephemeral (outside of the convenience of piping files around for testing), thered be end consumers which may take the data and store it in whatever native format they like

# ? May 25, 2016 06:04

VikingofRock: Aug 24, 2008

LeftistMuslimObama posted:

i'm not advocating writing a 40-line chain of if-then-else-else-else or whatever, but this guy is pathological to the point that he will not write the word "if"

Did this guy use to work in a brainfuck shop or something?

# ? May 25, 2016 06:41

Wheany: Mar 17, 2006; Spinya^{ha^{ha^haha}ha}ha_{ha_{ha_haha}ha}ha!; Doctor Rope

the talent deficit posted:

liquibase kind of sucks but has rollbacks

there might be something fucky about our environment, but liquibase on the command line doesn't recognize migrations run when the app starts up.

if i run liquibase -updatesql, it will just vomit every changeset into the console

probably because it thinks that the changesets haven't been run because their hashes are different. but why are their hashes different? does packing the changeset file into a jar change something? gently caress if i know.

all i know is that testing my changesets is a pain the rear end, because i can't do it on the command line because it wants to run all the changesets since the dawn of time. and if i just run the app and realize i hosed up, i have to manually clean up the mess

thankfully i rarely have to deal with that poo poo

# ? May 25, 2016 06:48

Wheany: Mar 17, 2006; Spinya^{ha^{ha^haha}ha}ha_{ha_{ha_haha}ha}ha!; Doctor Rope

i made a thing in the database, and it seemed to work, but then another member of the team questioned it and i was like :shrug:

feel free to remove it, i like database coding as much as you like ui coding lol.

# ? May 25, 2016 06:53

JawnV6: Jul 4, 2004; So hot ...

Progressive JPEG posted:

yeah, if you could keep a cap on the chunk size then i could see that being okay, but say youve got data arriving in chunks 512kb or larger (this will not be uncommon), you'd need to wait for protobuf to chuuug through that whole blob in one go since you don't know beforehand where one submessage ends and another starts

to figure that out you'd need to DIY your own protobuf parser to scan out individual submessages by hand

grabbing two bytes validating the first and skipping ahead by the second doesn't strike me as the horrifying difficulty you keep painting it as. that aside, you said there was no framing/header format, which is strictly incorrect. that post also links to two separate libraries, it's not clear why you need the speed of hand-rolled when OTS is available. also, a cap on the chunk size is equivalent to a cap on the latency

i totally believe you've selected the correct protocol for your application, you're just veering into a lot of incorrect statements about the ones you didn't select and treating your particular use cases as absolutes

Progressive JPEG posted:

if you use ocf then the header contains the schema, this also makes the data completely self-describing which is p , and it means you can have your parsers do all kinds of poo poo dynamically based on the advertised fields

catting the schema file onto the front of a large enough message, meaning the amortized cost of a plain text file is trivial, doesn't strike me as all that elegant?

Progressive JPEG posted:

but if you want to pass around small messages individually then it starts getting messy and the docs get kinda handwavy with 'run a schema repo in a separate channel or something idk'

i still haven't heard an expected time for dealing with 16mb, if there's any expectation of the system responding in real time then buffering megs worth of data generally isn't an option. that's part of the pressure I had to use lots of small messages instead of coalescing things

# ? May 25, 2016 07:31

Progressive JPEG: Feb 19, 2003

JawnV6 posted:

i totally believe you've selected the correct protocol for your application, you're just veering into a lot of incorrect statements about the ones you didn't select and treating your particular use cases as absolutes

im just parroting the official docs here and what id needed to do to before when dealing with streams of protos circa 2012 :shrug:

a cap on buffer length is also a cap on how much memory the parsed objects are taking up at once

# ? May 25, 2016 07:47

redleader: Aug 18, 2005; Engage according to operational parameters

is the msdn magazine any good? work just bought a bunch of msdn subscriptions and i'm wondering if it's worth cashing in the ~free~ 1 year subscription for a ~hard copy~ of the msdn magazine

# ? May 25, 2016 11:12

MononcQc: May 29, 2007

raminasi posted:

how long does a property test suite take to run

and can I do it in c#

As talentdeficit said, as long as you want. Tools like Quickcheck and PropEr let you specify how many iterations you want. By default they tend to run 100 tests, which is small, but what I run for trivial changes I just want a safety check on:

code:

$ rebar3 proper
===> Verifying dependencies...
===> Compiling shunk
===> Testing prop_shunk_tab:prop_first()
....................................................................................................
OK: Passed 100 test(s).

[... 11 more properties here ...]

===> Testing prop_shunk_tab:prop_thresholds()
....................................................................................................
OK: Passed 100 test(s).
13/13 properties passed

Each period represents a test and they're usually showed in real time to know how tests go. When I modify something trickier, I can configure things to run more tests, like 1,000 or 10,000, or pick a single property I have doubts about and run it 100,000 times if I want.

Given it's mostly just running ever-growing tests in a loop, the speed of the test case depends mainly on:
a) how long it is to generate the data set
b) how long each test takes to run on the given data set, times n iterations.

Data generation is generally fast if you stick to basic types. If you have custom recursive generators with all sorts of ways to go deeply complex (I have a suite where I generate two data sets of >10,000 key/val pairs with a diff between them wanting 5-10% differences), it's sometimes easy to write things in such a way generating the data set eats everything up; you have to think of ways to use the framework's laziness macros and balance probabilities to speed things up/make sure they don't run infinitely, but it's usually worth it and it's possible to bring it down. There's also tricks like knowing some fancy properties of your systems; the vast majority of distributed system errors can be found with just 3 nodes for example, so if I were to write a property test, I'm unlikely to need to run them over 50 nodes. That can let things go faster.

For each test to run, that's what is usually hard to optimize, especially if you're testing asynchronous systems and must insert pauses for things to sync up. Tests can sometimes be run in parallel to cope with that, but if the procedure is particularly slow, I find it pays to look at the data I'm generating (there's functions for that in the frameworks) and make sure they cover a lot of ground so that running the test 100 times may dig into as many cases as a simpler one run 5,000 times, for example.

In the end, I usually end up sticking with that thing where I do faster runs for the common case, and once in a while I'm gonna run the whole suite over large samples just to make sure nothing funny happened over the course of changes.

For C#, I guess your best bet is usually to look at what F# people build, since the communities are closer to the functional ones who tend to love property-based testing. FsCheck seems to be the obvious candidate: https://github.com/fscheck/FsCheck -- which appears to be callable from C#.

# ? May 25, 2016 13:21

Bloody: Mar 3, 2013

my garbage neural network still doesnt work it gets to like a 55% or so classification rate but doesn't really improve from there. not sure if my hyperparameters are just lovely or if my maths are subtly wrong

# ? May 25, 2016 15:20

Finster Dexter: Oct 20, 2014; Beyond is Finster's mad vision of Earth transformed.

HoboMan posted:

that reminds me: i hung out with our computer janitors today. turns out there's no way to rollback a push to live, lol.

They can't even redeploy an older build?

# ? May 25, 2016 15:47

HoboMan: Nov 4, 2010

Finster Dexter posted:

They can't even redeploy an older build?

a "push to live" i believe just means they copy-replace the files from the dev server. i'm sure they could just back up the old files first, but apparently no one thought of doing that?

Bloody posted:

my garbage neural network still doesnt work it gets to like a 55% or so classification rate but doesn't really improve from there. not sure if my hyperparameters are just lovely or if my maths are subtly wrong

oh boy a chance to do math, gimme some deets. i don't know anything about neural networks (despite making an alrorithim for two of them) and i'm very curious how they work

# ? May 25, 2016 16:21

DONT THREAD ON ME: Oct 1, 2002; by Nyc_Tattoo; Floss Finder

Bloody posted:

my garbage neural network still doesnt work it gets to like a 55% or so classification rate but doesn't really improve from there. not sure if my hyperparameters are just lovely or if my maths are subtly wrong

are you using a neurotypical network by any chance? they are easier to work with but tend to pretty bad at classifying things

# ? May 25, 2016 16:32

Bloody: Mar 3, 2013

HoboMan posted:

a "push to live" i believe just means they copy-replace the files from the dev server. i'm sure they could just back up the old files first, but apparently no one thought of doing that?

oh boy a chance to do math, gimme some deets. i don't know anything about neural networks (despite making an alrorithim for two of them) and i'm very curious how they work

i am working through http://neuralnetworksanddeeplearning.com/

# ? May 25, 2016 16:37

Finster Dexter: Oct 20, 2014; Beyond is Finster's mad vision of Earth transformed.

HoboMan posted:

a "push to live" i believe just means they copy-replace the files from the dev server. i'm sure they could just back up the old files first, but apparently no one thought of doing that?

oh boy a chance to do math, gimme some deets. i don't know anything about neural networks (despite making an alrorithim for two of them) and i'm very curious how they work

Oh. You're not doing continuous integration or have a deployment server. You should do those. In theory TFS can do a lot of that, I don't remember what source control you are on.

# ? May 25, 2016 17:02

HoboMan: Nov 4, 2010

Yeah got TFS, but i don't know poo poo about it. I just do checkouts and commit changes since it's integrated into VS(2010) and was all set up already.

# ? May 25, 2016 17:09

DONT THREAD ON ME: Oct 1, 2002; by Nyc_Tattoo; Floss Finder

HoboMan posted:

that reminds me: i hung out with our computer janitors today. turns out there's no way to rollback a push to live, lol.

is there a reason or do they just not care about being good at their job?

at my old job if i wanted to look at the application logs i had to run a Capistrano task that would run a grep on each host. I asked them why not use elk or splunk and they were like 'but why' and then they'd be dicks (you know how ops feels like a boys club)

now i do more ops stuff and ive realized that those guys were just bad at their jobs

# ? May 25, 2016 18:16

Finster Dexter: Oct 20, 2014; Beyond is Finster's mad vision of Earth transformed.

HoboMan posted:

Yeah got TFS, but i don't know poo poo about it. I just do checkouts and commit changes since it's integrated into VS(2010) and was all set up already.

Oh poo poo I forgot you were still on VS2010. What version of TFS are you on? I bet it's a lovely version that doesn't have good build integration. Well, what I would do in your shoes is get TeamCity or Bamboo (I don't think either of those are free, sadly, so maybe get a manager to sign off on it) and just start setting up projects in there. Those are the build servers I've used, but not sure if people in here have used better stuff for .NET.

# ? May 25, 2016 18:41

Finster Dexter: Oct 20, 2014; Beyond is Finster's mad vision of Earth transformed.

MALE SHOEGAZE posted:

is there a reason or do they just not care about being good at their job?

at my old job if i wanted to look at the application logs i had to run a Capistrano task that would run a grep on each host. I asked them why not use elk or splunk and they were like 'but why' and then they'd be dicks (you know how ops feels like a boys club)

now i do more ops stuff and ive realized that those guys were just bad at their jobs

Every IT group I've ever worked with has been stupid when it comes to log aggregation.

But you hit the nail on the head for why I like a DevOps approach.

# ? May 25, 2016 18:44

Valeyard: Mar 30, 2012; Grimey Drawer

HoboMan posted:

that reminds me: i hung out with our computer janitors today. turns out there's no way to rollback a push to live, lol.

You mean, you the developers, do not need to provide rollback mechanisms and rollback instructions for every release?

# ? May 25, 2016 18:44

Valeyard: Mar 30, 2012; Grimey Drawer

On a call with a business user, we were showing him our JIRA. First thing he asks; can I export this (the backlog) to excel

lol

(you can)

# ? May 25, 2016 18:57

MrMoo: Sep 14, 2000

Valeyard posted:

You mean, you the developers, do not need to provide rollback mechanisms and rollback instructions for every release?

pfSense is like this, always awesome roll of the dice whether ipSec will be working or not. It's not.

They do have an embedded platform that supports two copies in different partitions so you can roll back, but if you have a standard HDD install :lol:

# ? May 25, 2016 19:18

Shaggar: Apr 26, 2006

Valeyard posted:

On a call with a business user, we were showing him our JIRA. First thing he asks; can I export this (the backlog) to excel

lol

(you can)

tell him no. if he needs it he can login to Jira

# ? May 25, 2016 19:20

DONT THREAD ON ME: Oct 1, 2002; by Nyc_Tattoo; Floss Finder

Shaggar posted:

tell him no. if he needs it he can login to Jira

no jira access? no problem, just open a ticket with helpdesk, they'll get you setup in a week or two.

*two weeks later, the day before the ticket breaks SLA*
MALE SHOEGAZE. We're closing your ticket because helpdesk routed it to the wrong team. Please open a ticket with $TEAM instead.

# ? May 25, 2016 21:09

Adbot: ADBOT LOVES YOU

# ? May 12, 2024 02:14

AWWNAW: Dec 30, 2008

Valeyard posted:

On a call with a business user, we were showing him our JIRA. First thing he asks; can I export this (the backlog) to excel

lol

(you can)

my prev. job let business "stakeholders" login to JIRA and nickel n' dime all our estimates and bitch if something went overestimate

# ? May 25, 2016 21:20

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > YOSPOS > terrible programmers: my god, it's full of chars

«‹›1446 »