Ask General Programming Questions Not Worth Their Own Thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »

Sulla Faex: May 14, 2010; No man ever did me so much good, or enemy so much harm, but I repaid him with ENDLESS SHITPOSTING

Maybe it's because I haven't slept much but I'm struggling with the solution to a simple concept. For context I'm using python but this solution is bound to be super simple and probably cross-applicable to any language, my brain is just broken.

I have two arrays/lists/collections of data:

The first array is a series of values whose order is important. The second contains numbers that correspond to positions in the first array.

code:

values = [andrew, bob, charlie, daniel, edward]
positions = [2, 3]

So the first item in positions, '2', would correspond to values[2] which is 'charlie'.

If a value's position exists in the array positions, it means that value can be nullified when outputting the contents of the first array values. E.g. these are all the valid possible outputs of values according to positions:

code:

andrew, bob, edward
andrew, bob, charlie, edward
andrew, bob, daniel, edward
andrew, bob, charlie, daniel, edward

What I'm looking for is a way to iterate through every single possible permutation of that factorial format. I have the feeling like it should just be a couple nested for loops that iterate through the array and create a third array containing which positions to toggle, using just "2 => true", "3 => false" etc. But my brain is completely fried and I'm struggling to make the final connection with how to implement this. I want a final step where I can go:

code:

for i, row in enumerate(factorialised):
        for currentposition,name in enumerate(values):
                if currentposition in positions:
			if factorialised[i][currentposition] = True:
				print name
		else:
			print name

But what I'm missing is the in-between step where I take the array 'positions' (which isn't always 2 values) and sort it into every possible permutation of true/false combinations, saving the results into 'factorialised'.

Does that make sense?

Sulla Faex fucked around with this message at 09:44 on Jun 26, 2014

# ? Jun 26, 2014 09:35

Adbot: ADBOT LOVES YOU

# ? May 22, 2024 08:54

LOOK I AM A TURTLE: May 22, 2003; "I'm actually a tortoise."; Grimey Drawer

Sulla-Marius 88 posted:

If a value's position exists in the array positions, it means that value can be nullified when outputting the contents of the first array values. E.g. these are all the valid possible outputs of values according to positions:
code:
andrew, bob, edward
andrew, bob, charlie, edward
andrew, bob, daniel, edward
andrew, bob, charlie, daniel, edward

If I'm understanding you correctly, you want something like this:

Python code:

import itertools as it

# I lifted this one from [url]https://docs.python.org/2/library/itertools.html[/url]
def powerset(iterable):
    s = list(iterable)
    return it.chain.from_iterable(it.combinations(s, r) for r in range(len(s)+1))

def exclude_indices(L, indices):
    return (name for idx, name in enumerate(L) if idx not in indices)

def subtracted(L, indices):
    return (exclude_indices(L, to_exclude) for to_exclude in powerset(indices))

values = ["andrew", "bob", "charlie", "daniel", "edward"]
positions = [2, 3]
for t in subtracted(values, positions):
    print(tuple(t))

This is not necessarily the most efficient solution, but it's easier to understand than a more imperative solution and maps very nicely to mathematical definitions. In case you're unaware, the powerset of a set is the set of all subsets of that set, so for e.g. [2,3] it's [[], [2], [3], [2,3]]. If the order things come out of subtracted is important you may need to tweak the powerset function a bit. As it stands the results will be yielded in the reverse order of your example.

Another advantage of this solution is that it evaluates everything lazily, so if you only need the first N values from subtracted then you don't have to load the rest into memory, or even calculate them.

# ? Jun 26, 2014 10:08

Sulla Faex: May 14, 2010; No man ever did me so much good, or enemy so much harm, but I repaid him with ENDLESS SHITPOSTING

LOOK I AM A TURTLE posted:

If I'm understanding you correctly, you want something like this:

This is not necessarily the most efficient solution, but it's easier to understand than a more imperative solution and maps very nicely to mathematical definitions. In case you're unaware, the powerset of a set is the set of all subsets of that set, so for e.g. [2,3] it's [[], [2], [3], [2,3]]. If the order things come out of subtracted is important you may need to tweak the powerset function a bit. As it stands the results will be yielded in the reverse order of your example.

Another advantage of this solution is that it evaluates everything lazily, so if you only need the first N values from subtracted then you don't have to load the rest into memory, or even calculate them.

That is exactly, 100% what I was looking for, thank you. The order of subtracted's results isn't important. I had found itertools permutable but was thrown off course because that was just a different way of shuffling the order rather than toggling the different options. Perfect! Thank you

# ? Jun 26, 2014 11:09

Pie Colony: Dec 8, 2006; I AM SUCH A FUCKUP THAT I CAN'T EVEN POST IN AN E/N THREAD I STARTED

Your problem is pretty similar to the one of computing power sets in general.

code:

p_set = set(positions)

def f(v):
    if not v:
        return [[]]

    top = v.pop() # the last element
    i = len(v)    # the index of the last element
    rest = f(v)   # the list of valid sequences WITHOUT the last element
    res = []

    for r in rest:
        if i in p_set:
            res.append(r[:])
        r.append(top)
        res.append(r)

    return res

Personally I find this one is easier to understand than the one posted above. You are generating a list of sequences of names (a list of lists). Pretend you have a valid list of sequences for all the values EXCEPT the last one, so what do you do? If it's not in the set of positions, easy, you can just append it to all the other sequences and obtain a new valid list of sequences. If it is in the set of positions, it's trickier, because for each old valid sequence, you must have two new sequences -- one with the last value and one without.

If the set of positions includes all the positions, you've written a powerset function.

# ? Jun 26, 2014 14:38

TheEffect: Aug 12, 2013

If I have more than one version of Java installed can I specify the version I want to kick a program off with via batch file, or do I have to change my environment variable? If I have to change the environment path will the programs that are currently using the other version stop working? I'm thinking no but wanted to make sure.

# ? Jun 26, 2014 16:41

pr0k: Jan 16, 2001; "Well if it's gonna be
that kind of party..."

I'm trying to learn XSL in a hurry and not in a good way.

Say I have input xml:

code:

<thing>
 <createdate>2014-04-09T10:37:25-05:00</createdate>
</thing>

I want to make a fixed-length file with this data, but the date and time go in separate fields.

like:

code:

DATE:20140409
TIME:103725

Here's my xsl (1.0)

code:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>
<xsl:output method="text" indent="no"/>
  <xsl:template match="/">
    DATE:<xsl:value-of select="$CreatedDate"/>
    TIME:<xsl:value-of select="$CreatedTime"/>
  </xsl:template>
  <!--==================    VARIABLES  ==================-->
  <xsl:variable name="CreatedDate">
    <xsl:apply-templates select="/thing/createdate"/>
  </xsl:variable>
  <xsl:variable name="CreatedTime">
    <xsl:apply-templates select="/thing/createdate"/>
  </xsl:variable>

  <!--==================   FORMATTING  ==================-->
  <xsl:template match="createdate">
    <xsl:value-of select="substring(concat(translate(., '-', ''), '        '), 1, 8)"/>
  </xsl:template>
  <xsl:template match="createdate">
    <xsl:value-of select="substring(concat(translate(substring-after(.,'T'), ':', ''), '        '), 1, 6)"/>
  </xsl:template>
</xsl:stylesheet>

Result:

code:

    DATE:103725
    TIME:103725

This isn't going to work, is it? I googled it but I just don't understand why this is so freakin hard? I know the formatting is matching the node both ways and it's just picking the last one, but I want to apply the formatting to the variable, not the node.

# ? Jun 26, 2014 16:54

qntm: Jun 17, 2009

pr0k posted:

I'm trying to learn XSL in a hurry and not in a good way.

Say I have input xml:
code:
<thing>
 <createdate>2014-04-09T10:37:25-05:00</createdate>
</thing>
I want to make a fixed-length file with this data, but the date and time go in separate fields.

like:
code:
DATE:20140409
TIME:103725

You want to ignore critical time zone information?

# ? Jun 26, 2014 16:59

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

Sulla-Marius 88 posted:

Maybe it's because I haven't slept much but I'm struggling with the solution to a simple concept. For context I'm using python but this solution is bound to be super simple and probably cross-applicable to any language, my brain is just broken.
What I'm looking for is a way to iterate through every single possible permutation of that factorial format. I have the feeling like it should just be a couple nested for loops that iterate through the array and create a third array containing which positions to toggle, using just "2 => true", "3 => false" etc. But my brain is completely fried and I'm struggling to make the final connection with how to implement this. I want a final step where I can go:
code:
for i, row in enumerate(factorialised):
        for currentposition,name in enumerate(values):
                if currentposition in positions:
			if factorialised[i][currentposition] = True:
				print name
		else:
			print name
But what I'm missing is the in-between step where I take the array 'positions' (which isn't always 2 values) and sort it into every possible permutation of true/false combinations, saving the results into 'factorialised'.

Does that make sense?

As a general loop algorithm (so you're not generating a stack of lists to iterate over), couldn't you take advantage of the fact that combinations of either/or variables can be written

pre:

O O O
O O X
O X O
O X X
X O O
X O X
X X O
X X X

So you have 8 (2ⁿ) combinations, and the first variable alternates every 8/2 iterations, the second every 8/4, the nth every 8/2ⁿ. If you do some integer division with these numbers, you get

pre:

And then if you %2 them into odds and evens you get the pattern from above

pre:

So your 'should I include this' (pseudo)code in the end would be something like

code:

combinations = 2 ^ positions.length

for i=0; i < combinations:
  for j=0; j < values:
    intDivisor = combinations / 2^(j+1)
    if j in positions AND (i / intDivisor) % 2 == 1:
      add null
    else add values[j]

I'm not saying this is the best solution or anything! I was just interested in the general algorithmic problem, and if it's terrible let me know. Don't bother telling me my pseudocode style is bad, I just made it up and there's a cat bothering me

# ? Jun 26, 2014 17:48

Sedro: Dec 31, 2008

TheEffect posted:

If I have more than one version of Java installed can I specify the version I want to kick a program off with via batch file, or do I have to change my environment variable? If I have to change the environment path will the programs that are currently using the other version stop working? I'm thinking no but wanted to make sure.

Environment variables are scoped to the process, so you can set your PATH or JAVA_HOME at the beginning of your script without impacting the rest of the system. Or you could just call java with an absolute path e.g. "C:\Program Files\Java\jdk1.8.0_05\bin\java.exe" -jar app.jar

# ? Jun 26, 2014 18:40

SurgicalOntologist: Jun 17, 2004

I would like to package a C library as a Python module (it already includes low-level Python bindings but I have made a high-level interface). I have already done all the coding, testing, etc, now I just need to figure out the packaging. I'm posting here because my question has mostly to do with make and I'm not sure what the best practices are, but if this should go in the Python thread let me know.

The instructions for installing this library involve editing makefiles to uncomment lines specifying architecture and options then running make in four different places, and finally in the last directory (the included low-level Python bindings) also running make install. This is not the correct way to do things, right? And it seems it creates problems to automate it as part of my package's installation.

Another wrinkle is that the library uses a server-client architecture (generally all running on the same machine) so there are a couple extra steps. Currently I have a shell script that does the following:

Heavy use of sed on the makefiles.
Run make and make install (this installs the client part) in four places.
Copy the server part to /opt and create a symbolic link to the executable in /usr/local/bin.
Install my high-level interface.

What I would like to do is automate the above so that installing my library also installs the C library if it is not already installed. But I'm not sure how to go about this. Should I...

make upstream changes to the C library so that manual editing of makefiles is not required?
clean up my shell script, or port it to Python, and run it during installation?
create my own makefile that does the above steps?

I'm clearly pretty clueless about how to package this for other people. My own shell script is basically just a documentation of the steps I took that eventually worked, and in places makes significant changes to makefiles. Even if you don't have answers to my specific situation, maybe something to read on best practices would be helpful. This is basically the first time poking my head out of the scientific Python world and into C and make.

Edit to include specifics:
The original C library: http://www.cs.unc.edu/Research/vrpn/index.html
My shell script: https://gist.github.com/hsharrison/24cbe284bd50973052ee

SurgicalOntologist fucked around with this message at 19:07 on Jun 26, 2014

# ? Jun 26, 2014 18:58

TheEffect: Aug 12, 2013

Sedro posted:

Environment variables are scoped to the process, so you can set your PATH or JAVA_HOME at the beginning of your script without impacting the rest of the system. Or you could just call java with an absolute path e.g. "C:\Program Files\Java\jdk1.8.0_05\bin\java.exe" -jar app.jar

Thanks! It seems to be showing both versions when I do a version check now, witht he version I want being labeled as a "HotSpot"? Is that correct?
-
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)
ConComMon v.1.11.2
-

TheEffect fucked around with this message at 19:39 on Jun 26, 2014

# ? Jun 26, 2014 19:24

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

TheEffect posted:

Thanks! It seems to be showing both versions when I do a version check now, witht he version I want being labeled as a "HotSpot"? Is that correct? Screenshot-
http://tinypic.com/view.php?pic=261ivki&s=8#.U6xlHrH5dbR

You can copy/paste from the terminal by clicking the icon in the top left corner and going to Edit->Mark. Then use your mouse to highlight what you want to copy, and then right click. Now you can paste it somewhere.

Also, use http://imgur.com/ for posting images on the forums

# ? Jun 26, 2014 19:29

Pollyanna: Mar 5, 2005; Milk's on them.

Is there a better book for learning about algorithms than Skiena? I don't think I'm quite at the level that Skiena expects me to be. At least, most of the exercises completely escape me.

# ? Jun 26, 2014 19:32

TheEffect: Aug 12, 2013

fletcher posted:

You can copy/paste from the terminal by clicking the icon in the top left corner and going to Edit->Mark. Then use your mouse to highlight what you want to copy, and then right click. Now you can paste it somewhere.

Also, use http://imgur.com/ for posting images on the forums

Oops, thank you. Fixed.

Here's what I was talking about-

code:

java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)

That's what gets output when I run the following batch file-

code:

@echo off
TITLE Java v.6u13
"C:\Program Files\JavaTest\jre6\bin\java.exe" -version
PAUSE

I'm not sure why it mentioned v6u20 first, and labels 6u13 as a "HotSpot" and says "mixed mode, sharing". Does that mean it's running with version 6 update 13 or 6u20?

TheEffect fucked around with this message at 20:01 on Jun 26, 2014

# ? Jun 26, 2014 19:41

Sedro: Dec 31, 2008

The version number on the first line is your version, ignore the rest

# ? Jun 26, 2014 20:09

TheEffect: Aug 12, 2013

But is that the version that is being run in the current console instance? Because in that example I specifically chose the directory where I have 6u13.

# ? Jun 26, 2014 20:22

carry on then: Jul 10, 2010; by VideoGames
(and can't post for 10 years!)

No, it's not right. It should say 1.6.0_13 if you're running Update 13. HotSpot is the name of the specific Java Virtual Machine that will be interpreting the byte code, not the version of the Java runtime environment (which includes much more than just the VM; yeah, I know it's a bit confusing.)

What sort of folders do you have in C:\Program Files*\Java? Here's what mine looks like with multiple JREs and JDKs installed to the default location, maybe what you want is in here?

(I'm not on windows now but that's on my windows drive)

# ? Jun 26, 2014 20:38

TheEffect: Aug 12, 2013

carry on then posted:

What sort of folders do you have in C:\Program Files*\Java? Here's what mine looks like with multiple JREs and JDKs installed to the default location, maybe what you want is in here?

In this instance the older version of Java was installed to its own directory, "C:\Program Files\JavaTest" and it only has a directory named "jre6" in it. My script points to "C:\Program Files\JavaTest\jre6\bin\java.exe".

Not sure what's wrong.

Edit- If it matters, my "actual" Java directory has two directories inside of it. "jdk1.6.0_20" and "jre6". They seem to have been installed back in 2010 according to the date modified. 6u13 was just installed yesterday to its own directory.

Editx2-

Figured it out by checking the "javacpl.exe" in the new directory. Looks like my counterpart installed the same version, 6u20, and not 6u13. He told me it was because it wouldn't let him install an older version so he just copied the files over from his machine, and he must've copied his 6_20 install. Confused me because the console kept saying 6u13 for that "HotSpot" thing and I kept wondering where it could have gotten that from other than the fact that (I assumed) 6u13 was installed.

I think I have it now. Thanks everyone!

TheEffect fucked around with this message at 21:56 on Jun 26, 2014

# ? Jun 26, 2014 21:21

DholmbladRU: May 4, 2006

resolved

DholmbladRU fucked around with this message at 22:22 on Jun 26, 2014

# ? Jun 26, 2014 22:05

carry on then: Jul 10, 2010; by VideoGames
(and can't post for 10 years!)

TheEffect posted:

In this instance the older version of Java was installed to its own directory, "C:\Program Files\JavaTest" and it only has a directory named "jre6" in it. My script points to "C:\Program Files\JavaTest\jre6\bin\java.exe".

Not sure what's wrong.

Edit- If it matters, my "actual" Java directory has two directories inside of it. "jdk1.6.0_20" and "jre6". They seem to have been installed back in 2010 according to the date modified. 6u13 was just installed yesterday to its own directory.

Editx2-

Figured it out by checking the "javacpl.exe" in the new directory. Looks like my counterpart installed the same version, 6u20, and not 6u13. He told me it was because it wouldn't let him install an older version so he just copied the files over from his machine, and he must've copied his 6_20 install. Confused me because the console kept saying 6u13 for that "HotSpot" thing and I kept wondering where it could have gotten that from other than the fact that (I assumed) 6u13 was installed.

I think I have it now. Thanks everyone!

I think you should read it closer, because it's saying 16.3-b01 which isn't the same at all. But good that you figured it out.

# ? Jun 26, 2014 22:48

Kenishi: Nov 18, 2010

A link to web page with technical interview questions was posted in the Interview help thread. This question kind of confused me, maybe someone can explain it.

quote:

Implement an arraylist. Need constant time for adding and removing elements.

Implementing an arrayList/vector is generic but I couldn�t figure out a good way to fill up the gaps in the array when elements were removed. If you figure out a good solution then please share it with us.

Can you actually create an ArrayList that can do add/remove in O(k)? I thought about it tried something but I think my solution was only O(n), that was using a regular array and resizing occasionally for adds. But then I thought an ArrayList masking a linkedlist might make the removes pretty close to constant since you don't have to fill in gaps. But the search for the element to remove could potentially be O(n) if its not sorted. Just curious.

# ? Jun 26, 2014 23:32

ShoulderDaemon: Oct 9, 2003; support goon fund; Taco Defender

Kenishi posted:

Can you actually create an ArrayList that can do add/remove in O(k)?

If you don't need to preserve the order of elements in the list: To remove the ith element, first swap it with the last element, then shrink the array by 1. On an array with O(1) access times, these are both O(1) operations.

# ? Jun 26, 2014 23:45

LOOK I AM A TURTLE: May 22, 2003; "I'm actually a tortoise."; Grimey Drawer

Kenishi posted:

A link to web page with technical interview questions was posted in the Interview help thread. This question kind of confused me, maybe someone can explain it.

Can you actually create an ArrayList that can do add/remove in O(k)? I thought about it tried something but I think my solution was only O(n), that was using a regular array and resizing occasionally for adds. But then I thought an ArrayList masking a linkedlist might make the removes pretty close to constant since you don't have to fill in gaps. But the search for the element to remove could potentially be O(n) if its not sorted. Just curious.

You should look into amortized analysis, specifically for dynamic arrays (i.e. arraylists): http://en.wikipedia.org/wiki/Dynamic_array#Geometric_expansion_and_amortized_cost. As a somewhat simplified explanation: On the surface it may seem like Add is O(n), but if you consider the entire program as opposed to individual operations you'll see that it will only need to resize the array once for every time it adds N elements, so the constant time cost is maintained.

This only holds when you add to the end of the list, and the same goes for Remove(). If we're talking about Insert(idx, item), Remove(item) or Remove(idx), I don't see how any of those can run in constant time -- amortized or otherwise.

# ? Jun 27, 2014 10:14

redleader: Aug 18, 2005; Engage according to operational parameters

pr0k posted:

I'm trying to learn XSL in a hurry and not in a good way.

Say I have input xml:

code:

<thing>
 <createdate>2014-04-09T10:37:25-05:00</createdate>
</thing>

I want to make a fixed-length file with this data, but the date and time go in separate fields.

like:

code:

DATE:20140409
TIME:103725

Here's my xsl (1.0)

code:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>
<xsl:output method="text" indent="no"/>
  <xsl:template match="/">
    DATE:<xsl:value-of select="$CreatedDate"/>
    TIME:<xsl:value-of select="$CreatedTime"/>
  </xsl:template>
  <!--==================    VARIABLES  ==================-->
  <xsl:variable name="CreatedDate">
    <xsl:apply-templates select="/thing/createdate"/>
  </xsl:variable>
  <xsl:variable name="CreatedTime">
    <xsl:apply-templates select="/thing/createdate"/>
  </xsl:variable>

  <!--==================   FORMATTING  ==================-->
  <xsl:template match="createdate">
    <xsl:value-of select="substring(concat(translate(., '-', ''), '        '), 1, 8)"/>
  </xsl:template>
  <xsl:template match="createdate">
    <xsl:value-of select="substring(concat(translate(substring-after(.,'T'), ':', ''), '        '), 1, 6)"/>
  </xsl:template>
</xsl:stylesheet>

Result:

code:

    DATE:103725
    TIME:103725

The issue here is that the XSLT processor has no way of distinguishing the two templates matching createdate. Both your createdate templates have the exact same priority when choosing which template to apply - see this StackOverflow question.

This goes into far more detail than you probably need, but the takeaway point is this (my emphasis):

W3C posted:

It is an error if this leaves more than one matching template rule. An XSLT processor may signal the error; if it does not signal the error, it must recover by choosing, from amongst the matching template rules that are left, the one that occurs last in the stylesheet.

I assume this is why both variables end up with the value from the time template. You could fix this by changing the priority of the two templates by adding a mode to the template and the apply-template:

XML code:

  <xsl:variable name="CreatedDate">
    <xsl:apply-templates select="/thing/createdate" mode="date"/>
  </xsl:variable>
  <xsl:variable name="CreatedTime" >
    <xsl:apply-templates select="/thing/createdate" mode="time"/>
  </xsl:variable>

  <xsl:template match="createdate" mode="date">
    <xsl:value-of select="substring(concat(translate(., '-', ''), '        '), 1, 8)"/>
  </xsl:template>
  <xsl:template match="createdate" mode="time">
    <xsl:value-of select="substring(concat(translate(substring-after(.,'T'), ':', ''), '        '), 1, 6)"/>
  </xsl:template>

(This may or may not be possible depending on what else you have going on in your XML/XSLT.)

# ? Jun 27, 2014 10:33

pr0k: Jan 16, 2001; "Well if it's gonna be
that kind of party..."

Aha, thank you! I saw the 'mode' suggestion somewhere, probably on SO, but I didn't grok it.

Is what I am doing stupid? I have two different backend systems that create the same XML. I need at least two different flat-file outputs, one delimited, one fixed. I figured using XSL would be the way to go. I could just make perl scripts, which seems a lot neater and easier to read and follow...?

# ? Jun 27, 2014 14:59

gandlethorpe: Aug 16, 2008

Are there any Youtube channels that teach non language-specific programming topics (e.g. different paradigms) that aren't boring as hell?

# ? Jun 27, 2014 20:38

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

Anyone got any go-to references (preferably on the internet) about unit testing philosophy and approaches? I feel like there's a balance to be struck between retesting the same things (because other tests make use of those methods to test something else), and testing once-only and having every single test flame out because a key method broke. Seems like it comes down to how much you divide your code into separate test units too, and reworking some methods can make updating tests less clear and straightforward than I feel it should.

Basically unit testing is great, how can I get better at it?

# ? Jun 28, 2014 01:07

redleader: Aug 18, 2005; Engage according to operational parameters

pr0k posted:

Aha, thank you! I saw the 'mode' suggestion somewhere, probably on SO, but I didn't grok it.

Is what I am doing stupid? I have two different backend systems that create the same XML. I need at least two different flat-file outputs, one delimited, one fixed. I figured using XSL would be the way to go. I could just make perl scripts, which seems a lot neater and easier to read and follow...?

Nothing too stupid about processing XML with a tool designed to process XML :v:

I can think of a few places at work where we use XSLT to turn XML to CSV and/or JSON (not that I'd recommend creating JSON this way).

I guess it depends on how complicated the rest of the output needs to be. If you need to e.g. aggregate data then it may be easier to do it in Perl. If you're got the XSLT mostly written and working and you're happy with it then you might as well keep it.

If you're much more comfortable in Perl (and if other people in your organisation are proficient too) then you could definitely use Perl. Just make sure you use a proper XML library for parsing the documents instead of writing a bunch of regexes!

Personally, I'd be slightly more inclined to use another language rather than running an XSL transform for writing to a flat file. If the output was XML, I'd almost certainly do XSLT.

# ? Jun 28, 2014 03:42

Maluco Marinero: Jan 18, 2001; Damn that's a
fine elephant.

gandlethorpe posted:

Are there any Youtube channels that teach non language-specific programming topics (e.g. different paradigms) that aren't boring as hell?

I've recently discovered destroyallsoftware.com, which has screencast series for $40 a season. Not free for sure, but I'm very happy with what I've seen in the first season, and you can have a look at his talks for free to get a feel for his style. 'Boundaries' is a great one.

He spends a lot of time in Ruby but his work is largely applicable to more than just Ruby.

# ? Jun 28, 2014 10:09

Loezi: Dec 18, 2012; Never buy the cheap stuff

baka kaba posted:

Anyone got any go-to references (preferably on the internet) about unit testing philosophy and approaches? I feel like there's a balance to be struck between retesting the same things (because other tests make use of those methods to test something else), and testing once-only and having every single test flame out because a key method broke. Seems like it comes down to how much you divide your code into separate test units too, and reworking some methods can make updating tests less clear and straightforward than I feel it should.

Basically unit testing is great, how can I get better at it?

Unit testing should be just a layer in your greater testing scheme. Obviously there are people with many idea and different theories on this, but this is one way to look at it:

Layer zero: Static analysis.
Running your code through a static code analyzer, for example checkstyle. This step will assure that you are following whatever style standards you have configured. That is, correct indentation, no deeply nested if-structures etc. This step - with some sensible rules - could have stopped the (in)famous "goto fail" bug.

Layer one: Unit testing.
You are testing single units of your code. Most of the time, this means testing a single public method. Now the thing here is, you'll only want to test the one method under scrutiny. Let's take an example. You have a piece of code that gets the most recent Fart from the FartDAO and then asks the HTTPThingy to send the Fart to whatever server by HTTPThingy.send(). As this is still unit testing, you only want to test two things here: That you are correctly retrieving the latest Fart, and that the method is correctly asking HTTPThingy to send the Fart forwards. After all, we are only testing this methods, not HTTPThingy or FartDAO. HTTPThingy will have it's own tests.

The best way, in my experience, to do this is via mocking and Dependency Injection. Example:

code:

public class Butts {
   HTTPStack http;
   FartDAO farts;

   public Butts(HTTPStack http, FartDAO farts) {
      this.http = http;
      this.farts = farts;
   }
 
   public boolean do() {
      Fart f = farts.getMostRecent();
      if (f == null) {
          return false;
      }
      return http.send(f);
   }
}

code:

public class ButtsTest {
   private HTTPThingy http;
   private FartDAO farts;
   private Butts butts;

   @Before
   public void setUp() {
      http = mock(HTTPThingy.class);
      farts = mock(FartDAO.class);
      butts = new Butts(http);
   }

   @Test
   public void do_getsNewestFart() {
      butts.do();
      verify(farts, times(1)).getMostRecent();
   }

   @Test
   public void do_returnsFalseIfFailsToGetMostRecentFart() {
      when(farts.getMostRecent()).thenReturn(null);
      assertFalse(butts.do());
   }

   @Test
   public void do_sendsFartViaHttp(){
      Fart f = new Fart(); //Mock if complex
      when(farts.getMostRecent()).thenReturn(f);

      butts.do();
      verify(http, times(1)).send(f);
   }

   @Test
   public void do_returnsFalseIfHttpSendFailes(){
      when(farts.getMostRecent()).thenReturn(new Fart());
      when(http.send(any(Fart.class)).thenReturn(false);

      assertFalse(butts.do());
   }

   @Test
   public void do_returnsTrueIfHttpSendSucceeds(){
      when(farts.getMostRecent()).thenReturn(new Fart());
      when(http.send(any(Fart.class)).thenReturn(true);

      assertTrue(butts.do());
   }
}

Now we've made sure of two things here: We get a X from the correct place and we ask the HTTPThingy to do the right thing. Checking whether the HTTPThingy actually does the right thing when send() is called with X, is part of the HTTPThingy's tests just like testing whether the FartDAO really return the most recent is part of FartDAO:s tests.

A perfect unit test would test the method under scrutiny completely. That is, each line, each code path, each possible exception state. If you run into code you can't test, it's often a sign of either unnecessary or badly structured code. You'll probably also want to look at mutation testing. I've found PIT to be the thing I like the most. What it does is fiddle with your code and see if your tests caught the change. This is useful for determining if your tests are rigorous enough to allow for safe refactoring down the line.

The purpose of unit testing is to make sure Things work correctly in isolation and independently. The independently part meaning that any of your tests shouldn't depend on a previous test.

Layer two: Integration testing.
Now that we've made sure that the things work correctly in isolation, it's time to make sure they work when interfacing with each other. One way to look at this is by searching for entry points: Parts where other code would initiate something. If you have a ButtFactory that gets some Farts from FartDAO and creates a new Butt, priming it with all the Farts you would need, you'd likely want to test this part at this stage.

Write tests that call the ButtFactory like your other code would, and then assert that they get correctly primed Butts back. Mocking is less useful here. Most of the time you'll just end up rewriting your unit tests. Still, mocking is most likely unavoidable when the thingy you are testing ends up - somewhere down the line - doing something with a third party system. After all, you probably don't want to send real "Kill all humans" messages to your robot control server.

The purpose here is to verify that Larger collections of things work together as planned.

Layer three: System testing
This part is often a bit complicated and I've seen quite a many definition. The one I like the most - especially with WEB stuff - is: "Test things as a user". Whatever you are writing, it most likely has some kind of an interface for third party interactions. This could be the Web front end, the (C/G)UI the user uses to do stuff or possibly just a public interface for third party code to ask your component to do stuff.

Write tests that do stuff the user would do and assert that correct things happen. If it's a web thingy, test stuff like "When user fills this form with these things and presses this button, he should see this to indicate that the correct thing happened". Since you've already made sure that your individual components do the right things and that the components interact in the correct way, this step doesn't need to delve deep into the insides of your code. It's basically black box testing, from the viewpoint of the user.

If you have the system specs as user stories, this step will probably be about writing tests that make sure that the tasks described in those user stories can be accomplished.

This step is also where you test for usability ("Does this UI suck balls"), performance ("Does it crawl to stop after 5 concurrent users") and security ("Does everything break when little Bobby Tables checks in?").

Layer four: Acceptance testing
This is something you can't automate. The idea here is to determine if the software fills it purpose: whether it fixes whatever problem the user had and the software was designed to fix. IF you are writing a thingy to allow phone rep people to do thing X easier than before, this step would probably involve getting a few phone rep guys to come over and try the thing out.

What we've accomplished
So, what have we accomplished?

At layer 0, we made sure that code was stylistically up to spec. You have correct indentations, new lines etc. We've made sure that your code is easy for you and other developers to work with.

At layer 1, we made sure that the things you wrote worked like they should when in isolation. There are no glaring logical mistakes like you returning the oldest Fart when someone calls FartDAO.getNewest().

At layer 2, we made sure things work together. That when you ask the Http stack to update your Farts from the interweb, you make the correct Http calls to where-ever and the retrieved farts are correctly stored. The thingy works as a black box.

At layer 3, we made sure that the thingy works like it should when the user is doing user things.

At layer 4, we made sure that we actually built a thing that does what it was supposed to.

A random pile of writings.
Here's some links, hopefully some of them will be helpful.

Writing Great Unit Tests: Best and Worst Practices
Top 12 Selected Unit Testing Best Practices
Using Customer Tests to Drive Development
Mocks Aren't Stubs
Acceptance TDD Explained
Exploratory Testing Explained
What is Exploratory Testing?

Loezi fucked around with this message at 14:33 on Jun 28, 2014

# ? Jun 28, 2014 10:12

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

Wow, thanks for that! That's amazing, I really appreciate it. Right now I'm basically mocking my own classes (which generally works ok) and testing classes in isolation that way. I guess my issue is that it's hard to test methods in isolation, because some of them involve using other methods to build some state, so I can either test once and assume they're fine in the other test (and get multiple failures if they're not), or I can keep checking validity as I go, so I at least know why the other test failed and what needs looking at first.

The mocking system you're using there (what is it?) looks way more usable and robust though, so that might be the solution. I'm writing for Android so I'll need to look at what's available. Thanks for the links, I'll have a read through them in a bit

# ? Jun 28, 2014 18:05

pr0k: Jan 16, 2001; "Well if it's gonna be
that kind of party..."

Looks like Moq.

redleader posted:

Nothing too stupid about processing XML with a tool designed to process XML I can think of a few places at work where we use XSLT to turn XML to CSV and/or JSON (not that I'd recommend creating JSON this way).

I guess it depends on how complicated the rest of the output needs to be. If you need to e.g. aggregate data then it may be easier to do it in Perl. If you're got the XSLT mostly written and working and you're happy with it then you might as well keep it.

If you're much more comfortable in Perl (and if other people in your organisation are proficient too) then you could definitely use Perl. Just make sure you use a proper XML library for parsing the documents instead of writing a bunch of regexes!

Personally, I'd be slightly more inclined to use another language rather than running an XSL transform for writing to a flat file. If the output was XML, I'd almost certainly do XSLT.

I'm kind of with you there. It's been ten years since I used perl, but I think it would be easier. However, I have another output that I need to make which is PDF. I have XSL that creates XSL-FO for that. Needs a lot of tweaking but it should work, so I was trying to confine the solution to one technology. Since you showed me how to do modes it's getting less painful, so I'm going to keep working on this proof of concept. If I get stuck I'll change gears. Thanks again.

# ? Jun 28, 2014 22:29

Loezi: Dec 18, 2012; Never buy the cheap stuff

The mocking framework in the example is Mockito for Java. Another good one (so I've heard) is powermock.

# ? Jun 28, 2014 22:30

pliable: Sep 26, 2003; this is what u get for "180 x 180 avatars"

this is what u fucking get u bithc; Fun Shoe

My programming skills have quickly been growing rusty, and so I need some small projects to program to help get my feet wet again. Emphasis on "small", as I would like to do many of them.

Any suggestions? Thank you much!

# ? Jun 29, 2014 03:27

Jewel: May 2, 2009

pliable posted:

My programming skills have quickly been growing rusty, and so I need some small projects to program to help get my feet wet again. Emphasis on "small", as I would like to do many of them.

Any suggestions? Thank you much!

I've been trying out https://www.exercism.io lately and it's really neat. You get a small exe which is a command line application. You can run "exercism fetch python" (replace python with one of the many languages they have if you want) and it gets you an exercise.

The exercise consists of a unit test and a readme. You have to make a class/functions/a file that will pass the unit test. Once you've passed the unit test you can run "exercism submit <path to your file>" and then fetch the next task.

There seems to be a ton of tasks (no idea how many yet) and they're all pretty small-scale and I've enjoyed the python ones so far as a small thing to jump into if I'm waiting for something.

# ? Jun 29, 2014 03:40

pliable: Sep 26, 2003; this is what u get for "180 x 180 avatars"

this is what u fucking get u bithc; Fun Shoe

Jewel posted:

I've been trying out https://www.exercism.io lately and it's really neat. You get a small exe which is a command line application. You can run "exercism fetch python" (replace python with one of the many languages they have if you want) and it gets you an exercise.

The exercise consists of a unit test and a readme. You have to make a class/functions/a file that will pass the unit test. Once you've passed the unit test you can run "exercism submit <path to your file>" and then fetch the next task.

There seems to be a ton of tasks (no idea how many yet) and they're all pretty small-scale and I've enjoyed the python ones so far as a small thing to jump into if I'm waiting for something.

That does look really neat, I'll try this out. Thank you

EDIT: I just tried the first exercise in Python and you're absolutely right, this is very, very cool. Unfortunately, it doesn't seem like much of an active community. Is there a Something Awful team I can join though? If not, I'll create one!

pliable fucked around with this message at 23:09 on Jun 29, 2014

# ? Jun 29, 2014 06:36

KoRMaK: Jul 31, 2012

I was going to post a GBS style thread about this but I'll start small. I had an idea and wanted to see if anyone else thought it was worth while to invest their time in it. Goons should rewrite the forums in Ruby on Rails and opensource it or better yet only license it to Lowax to use expressly only for SA.

Then we write some migrations to port all the old data.

We start with gathering a list of all the current features and replicating those, including styles and layout so that the UI doesn't change at all. People hate UI changes.

Anyone else think this is a good idea?

# ? Jun 30, 2014 03:06

raminasi: Jan 25, 2005; a last drink with no ice

Are you trolling?

# ? Jun 30, 2014 03:29

Blotto Skorzany: Nov 7, 2008; He's a PSoC, loose and runnin'
came the whisper from each lip
And he's here to do some business with
the bad ADC on his chip
bad ADC on his chiiiiip

Waffleimages is about the closest thing there has been to a successful goon project, and that's been dead & buried for a while now. I don't think a second-system clone of a hacked up version of an ancient vBulletin codebase done in Rails has much of a chance of succeeding nor is worth doing - use your time for some better project imo.

# ? Jun 30, 2014 03:31

Adbot: ADBOT LOVES YOU

# ? May 22, 2024 08:54

KoRMaK: Jul 31, 2012

Well I mean, we all use these forums a lot, so it has a purpose. And our code wouldn't be hacky because we are just replicating functionality while re-writing the backend cleaner. In fact, we'd be better off never seeing the existing code and just doing this based on replicating the UI and UI functionality or improving it where it makes sense.

# ? Jun 30, 2014 03:36

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »