Ask General Programming Questions Not Worth Their Own Thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

leper khan posted:

What does LIMS stand for?

Google says Laboratory Information Management System, but I've never used one myself so I don't have a feel for what that actually means. If you need some kind of shared note-taking system, I know there are apps for phones/tablets you can use. Presumably everything ends up on The Cloud� instead of in your personal databases, but that's probably not a good reason to reinvent the wheel.

# ? Aug 4, 2016 19:34

Adbot: ADBOT LOVES YOU

# ? May 16, 2024 00:57

Mathematicus: Mar 10, 2004; Gozintas a specialty

TooMuchAbstraction posted:

Google says Laboratory Information Management System, but I've never used one myself so I don't have a feel for what that actually means. If you need some kind of shared note-taking system, I know there are apps for phones/tablets you can use. Presumably everything ends up on The Cloud� instead of in your personal databases, but that's probably not a good reason to reinvent the wheel.

Well, we don't know what we need yet - why get specs when you have a vision? But, apparently we're absolutely sure that none of the currently existing solutions could possibly fill the needs we may or may not have.

# ? Aug 4, 2016 19:48

Stinky_Pete: Aug 16, 2015; Stinkier than your average bear; Lipstick Apathy

Mathematicus posted:

Well, we don't know what we need yet - why get specs when you have a vision? But, apparently we're absolutely sure that none of the currently existing solutions could possibly fill the needs we may or may not have.

If you don't know the needs, how do you know the other software doesn't fulfill them?

# ? Aug 4, 2016 19:54

Mathematicus: Mar 10, 2004; Gozintas a specialty

Stinky_Pete posted:

If you don't know the needs, how do you know the other software doesn't fulfill them?

Because the new boss wrote a LIMS at his last job, and by god he's going to write one here.

# ? Aug 4, 2016 19:57

leper khan: Dec 28, 2010; Honest to god thinks Half Life 2 is a bad game. But at least he likes Monster Hunter.

This sounds like the type of thing that will be easier, cheaper, and more functional to license. Why not just use the best off the shelf one?

# ? Aug 4, 2016 22:20

Mathematicus: Mar 10, 2004; Gozintas a specialty

leper khan posted:

This sounds like the type of thing that will be easier, cheaper, and more functional to license. Why not just use the best off the shelf one?

That's what I'm trying to get the boss to consider, and is why I asked if anyone had any experience with open-source LIMSes.

# ? Aug 4, 2016 22:22

pro starcraft loser: Jan 23, 2006; Stand back, this could get messy.

Not sure if this is the right section or I should check out the Support sub thread, but I've been trying to figure out this problem for a while now and nothing I find on google has done the trick.

Recently found out program was running with Debug=True in web config. Turned it off and things running better for the most part, except now I'm getting timeouts with RDLC reports.

Upped the execution timeout in the webconfig for both the site and the site default in IIS to over 5 mins, but still getting timeout in about 2.

Put Server.ScriptTimeout = 300 in the program as well, still no luck.

Any ideas on what I might check?

# ? Aug 5, 2016 00:49

Robo Reagan: Feb 12, 2012; by Fluffdaddy

What should I be using to store snippets and keep general notes on what I'm doing? Never really used any notetaking software. Are personal wikis any good? Being able to have directories would be nice because I'm a sperg

# ? Aug 5, 2016 08:00

nielsm: Jun 1, 2009

Robo Reagan posted:

What should I be using to store snippets and keep general notes on what I'm doing? Never really used any notetaking software. Are personal wikis any good? Being able to have directories would be nice because I'm a sperg

As in keep track of unfinished work items? Maybe look into using an issue tracker, open tickets for new work items, add notes as relevant, and close tickets when done working.

# ? Aug 5, 2016 09:43

pokeyman: Nov 26, 2006; That elephant ate my entire platoon.

Robo Reagan posted:

What should I be using to store snippets and keep general notes on what I'm doing? Never really used any notetaking software. Are personal wikis any good? Being able to have directories would be nice because I'm a sperg

My experience is that I completely forget to keep using whatever I've been using after a couple weeks. So my answer is literally anything as long as you'll think to open it up again months later.

# ? Aug 5, 2016 12:21

No Safe Word: Feb 26, 2005

Robo Reagan posted:

What should I be using to store snippets and keep general notes on what I'm doing? Never really used any notetaking software. Are personal wikis any good? Being able to have directories would be nice because I'm a sperg

Trello is a pretty good quick way of keeping track of in-progress stuff and you can group it pretty well

# ? Aug 5, 2016 17:05

peepsalot: Apr 24, 2007; ��PEEP THIS...
��BITCH!

I take a lot of notes as google documents which go in google drive folders. That or just plain text files on my local computer, in a regular text editor.

There's a thing called google keep which is more of a post-it/pinboard that I always forget to use, but you might find that useful?

# ? Aug 5, 2016 17:19

dougdrums: Feb 25, 2005; CLIENT REQUESTED ELECTRONIC FUNDING RECEIPT (FUNDS NOW)

pokeyman posted:

My experience is that I completely forget to keep using whatever I've been using after a couple weeks. So my answer is literally anything as long as you'll think to open it up again months later.

I used to do this too, and what fixed it was using a repo service like bitbucket or github where a wiki and issue tracker is already provided. It seems silly to use an issue tracker by yourself, but it works well. I don't think there are any folders involved though.

Besides that, I keep a notebook on my desk...

# ? Aug 5, 2016 19:20

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

I'm struggling to wrap my head around the XML DOM, specifically for parsing using xerces-c and searching using xqilla. Specifically, attributes: are attributes not considered child nodes of their corresponding elements?

In the C++ specific case I have a DOMNode* pointing at my root element (not the document, the root element--I had enough fun figuring that one out already), and it has several xmlns: attributes. Iterating through the DOMNodeList* returned by getChildNodes() on the DOMNode* isn't finding any attributes, however.

(edit) I see how to do it now, it has a separate getAttributes function, but still, I would have thought iterating through all of an element node's children would get the attributes, unless I completely misunderstand the XML DOM.

Ciaphas fucked around with this message at 20:19 on Aug 5, 2016

# ? Aug 5, 2016 20:14

rjmccall: Sep 7, 2007; no worries friend; Fun Shoe

Attributes are not considered to be children of their node. It's just how it works.

# ? Aug 5, 2016 20:49

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

Chalk another one up to misunderstanding, then; thanks for clarifying. Seems harder and harder every day for me to learn new things :smith:

# ? Aug 5, 2016 21:02

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

I'm rapidly approaching wit's end. I've got a document that looks something like this:

XML code:

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2" xmlns:atom="foo" xmlns:gx="bar">
  <name>FooDocument</name>
  <atom:author>John Doe</atom:author>
  <Placemark>
    <gx:Track>
      <when>TimeTag</when>
      <when>TimeTag</when>
      <when>TimeTag</when>
      <!-- ... -->
    </gx:Track>
  </Placemark>
  <Placemark>
    <!-- ... -->
  </Placemark>
</kml>

(edit: typed from memory, probably not accurate KML, whatever)

When I use the xqilla command line app to do an XPath search on /kml/Placemark/*:Track/when (with appropriate command line flag to use xerces-c to parse) I get back a list of all of the <when> tags as I expect and want. When I do this in a C++ program using xerces-c to parse and xqilla to search, I get nothing. I've tried various forms like //Placemark/..., //*[local-name()='Track']/when, and others, with the same result every time. I've even tried loving around with DOMXPathNSResolvers, even though I wanted to explicitly avoid that using the * syntax in my query, to no result. Anyone have any idea what gives?

Ciaphas fucked around with this message at 22:23 on Aug 5, 2016

# ? Aug 5, 2016 21:47

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Robo Reagan posted:

What should I be using to store snippets and keep general notes on what I'm doing? Never really used any notetaking software. Are personal wikis any good? Being able to have directories would be nice because I'm a sperg

I find that there's nothing better than pen and paper, especially if I want to draw diagrams and such inline.

# ? Aug 5, 2016 21:57

nielsm: Jun 1, 2009

Last I used DOM with namespaced XML, I had to manually set up a name for the "default" namespace set up on the root element, in the XPath resolver. That was with the C#/.NET XML DOM library though, I'm not sure how standardized this is.

But what I'd have to do in your example, it manually configure a "kml"="http://www.opengis.net/kml/2.2" namespace on the XPath resolver, and then in my XPath expressions use fully qualified element names, such as /kml:kml/kml:Placemark/gx:Track/*.

# ? Aug 5, 2016 22:02

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

nielsm posted:

Last I used DOM with namespaced XML, I had to manually set up a name for the "default" namespace set up on the root element, in the XPath resolver. That was with the C#/.NET XML DOM library though, I'm not sure how standardized this is.

But what I'd have to do in your example, it manually configure a "kml"="http://www.opengis.net/kml/2.2" namespace on the XPath resolver, and then in my XPath expressions use fully qualified element names, such as /kml:kml/kml:Placemark/gx:Track/*.

I tried getting the XPathNSResolver to automatically work it out by passing in, variously, the document node and the root element node, but it didn't take. I tried manually adding atom and gx prefixes to the NS resolver, but it still came up empty. I didn't try adding a no-prefix NS to the resolver though, I'll go upstairs and try that and report back.

# ? Aug 5, 2016 22:12

carry on then: Jul 10, 2010; by VideoGames
(and can't post for 10 years!)

I like OneNote for notes but I'm in a position to take advantage of it, you might not be.

# ? Aug 5, 2016 22:15

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

carry on then posted:

I like OneNote for notes but I'm in a position to take advantage of it, you might not be.

OneNote owns bones, especially if you have a touchscreen because then you can just doodle if you have to. I just fall back to Google Keep most of the time, though, because it's free and :effort:

# ? Aug 5, 2016 22:19

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

Ciaphas posted:

I tried getting the XPathNSResolver to automatically work it out by passing in, variously, the document node and the root element node, but it didn't take. I tried manually adding atom and gx prefixes to the NS resolver, but it still came up empty. I didn't try adding a no-prefix NS to the resolver though, I'll go upstairs and try that and report back.

Nope, no dice. In fact, adding an NS resolver at all breaks it so namespace-less XPath expressions like /kml/name (I edited that in to my example above) don't return anything anymore.

I'm desperate enough at this point to go through the DOM tree and in-place-remove namespaces from all elements entirely and replacing their local names with a prefixed version, so <gx:Track> becomes the un-namespaced <gx-Track> for example. Or even taking some stupid converter from XML to JSON and parsing that instead. Argh!

# ? Aug 5, 2016 22:25

Cuntpunch: Oct 3, 2003; A monkey in a long line of kings

Suspicious Dish posted:

I find that there's nothing better than pen and paper, especially if I want to draw diagrams and such inline.

I'm going to come across as an evangelist, but since moving into the Surface owners club, I've fallen in love. Endless paper? Synchronization through OneNote? It really is brilliant how productive in terms of note-keeping this device family has made me.

# ? Aug 6, 2016 01:15

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

Say I'm getting and expected to parse :airquote:

XML documents that look something like this.

XML code:

<kml xmlns:atom="foo" xmlns:atom="bar">
  <atom:author>John Doe</atom:author>
</kml>
...
<kml>
  <atom:author>Jane Doe</atom:author>
</kml>

Obviously these aren't valid XML at all, but god damnit if one of my customers is trying to get me to parse data like it anyway. The obvious manual fix works fine in both cases, but I have enough data that I need an automatic solution.

My first thought (and an extreme one, but I'm getting frustrated as hell right now) is to remove all instances of xmlns: attributes full stop, and change any namespaced elements to use dashes instead of colons (so <atom:author> becomes <atom-author>). Basically make the namespace thing and invalid XML go away in one fell nuclear step.

Unfortunately I know gently caress all about regexes in UNIX land. Can I do some command line magic to make these two steps happen?

Ciaphas fucked around with this message at 01:01 on Aug 10, 2016

# ? Aug 10, 2016 00:59

Asymmetrikon: Oct 30, 2009; I believe you're a big dork!

That seems like valid namespaced XML? What are you trying to parse it with that's failing?

# ? Aug 10, 2016 01:08

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

Asymmetrikon posted:

That seems like valid namespaced XML? What are you trying to parse it with that's failing?

Today I've been experimenting with libxml2 but up to now this code base has been using xerces-c and XQilla for XPath support. In the former case, with doubled-up attributes, the parse outright fails with no hope of recovery on both. In the latter, the parse succeeds for both (though in the xerces-c case I have to configure it to continue parsing after "fatal" errors; libxml2 just emits to stderr and continues), but XPath refuses to play no matter how I use it.

# ? Aug 10, 2016 01:14

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

Ciaphas posted:

Unfortunately I know gently caress all about regexes in UNIX land. Can I do some command line magic to make these two steps happen?

If you know what the bad attribute names are, you can do for example

code:

cat file.txt | perl -pe 's|xmlns:atom=".*?"||g'

to remove the xmlns:atom attributes, and

code:

cat file.txt | perl -pe 's|atom:author|atom_author|g'

to fix the atom:author tags. You can chain the two together, too:

code:

cat file.txt | perl -pe 's|xmlns:atom=".*?"||g' | perl -pe 's|atom:author|atom_author|g'

I used Perl just because it has easy access to a non-greedy regex; otherwise, removing the individual attributes without munging other attributes could go badly. Also, my above approach won't work if there's an escaped " in the attribute, for example

code:

<kml xmlns:author="foo \" and bar">

That may not be an issue for you, though.

EDIT: oh, and my own question. I have a naively-generated JSON file that includes a comma at the end of every element in an array, which is not valid JSON and thus fails my parser. So I need to omit the last comma in a file. I suppose I can do this by joining all of the lines together, then doing s/(.*),/\1/, but is there a more elegant solution?

TooMuchAbstraction fucked around with this message at 01:43 on Aug 10, 2016

# ? Aug 10, 2016 01:34

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

TooMuchAbstraction posted:

Thanks, that'll be an awesome start. Unfortunately those were just examples of bad attributes (it's usually xmlns:atom, but not always!!!) and namespaced tags, but that gives me something I can work with. Hopefully.

I'll be back if it doesn't :v:

(edit) Luckily all the xmlns attribute values are (so far) URIs so I don't have to worry about escaped quotes ruining the job.

Ciaphas fucked around with this message at 01:55 on Aug 10, 2016

# ? Aug 10, 2016 01:53

pokeyman: Nov 26, 2006; That elephant ate my entire platoon.

TooMuchAbstraction posted:

EDIT: oh, and my own question. I have a naively-generated JSON file that includes a comma at the end of every element in an array, which is not valid JSON and thus fails my parser. So I need to omit the last comma in a file. I suppose I can do this by joining all of the lines together, then doing s/(.*),/\1/, but is there a more elegant solution?

eval

# ? Aug 10, 2016 01:55

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

Ciaphas posted:

Thanks, that'll be an awesome start. Unfortunately those were just examples of bad attributes (it's usually xmlns:atom, but not always!!!) and namespaced tags, but that gives me something I can work with. Hopefully.

I'll be back if it doesn't

(edit) Luckily all the xmlns attribute values are (so far) URIs so I don't have to worry about escaped quotes ruining the job.

That is good. I believe that

code:

perl -pe 's|\w+?:\w+?=".*?"||g'

will remove any attribute whose name contains a :, but I haven't rigorously considered it. Could be worth a shot though.

pokeyman posted:

eval

(Or did you mean that seriously? And if so, how?)

# ? Aug 10, 2016 02:06

pokeyman: Nov 26, 2006; That elephant ate my entire platoon.

I was mostly serious. Surround your faux-json with console.log(JSON.stringify(...)) and run it through jsc or node or whatever JavaScript of your choosing.

# ? Aug 10, 2016 03:10

Ciaphas: Nov 20, 2005; > BEWARE, COWARD

TooMuchAbstraction posted:

That is good. I believe that
code:
perl -pe 's|\w+?:\w+?=".*?"||g'
will remove any attribute whose name contains a :, but I haven't rigorously considered it. Could be worth a shot though.

I'm not at work but the whole thing ended up being something like

code:

cat blah.kml | perl -pe "s|xmlns:\w+\s*=\s*['"'"'"].+['"'"'"]||g" | perl -pe "s|<\s*(/?)(\w+):(\w+)|<\1\2_\3|g"

Working out those quotes was fun lemme tell ya the regex itself should have been just xmlns:\w+\s*=\s*["'].+["'] but shells are assholes

# ? Aug 10, 2016 03:37

JawnV6: Jul 4, 2004; So hot ...

The trick is to print the comma before the nth, not after the 1st

# ? Aug 10, 2016 04:09

zerofunk: Apr 24, 2004

I'm thinking about writing an iPhone app to dump all my pictures to S3 along with keeping track of meta data... probably in DynamoDB . It's mostly just an excuse to get some practice with things that I won't get any exposure to in my day job. Is there any legitimate reason to not have the iPhone app upload directly to S3 and instead go through some intermediate web service? I'm having trouble thinking of any good reasons to do that, but like I said I don't work with any of this stuff. The only possible concern I could think of would be security related since it means the app needs credentials for accessing S3. That's probably not a big deal though if you handle everything correctly. And if you don't, you're possibly going to mess up things between the app and your web service anyway.

# ? Aug 10, 2016 09:02

Extortionist: Aug 31, 2001; Leave the gun. Take the cannoli.

Ciaphas posted:

I'm not at work but the whole thing ended up being something like
code:
cat blah.kml | perl -pe "s|xmlns:\w+\s*=\s*['"'"'"].+['"'"'"]||g" | perl -pe "s|<\s*(/?)(\w+):(\w+)|<\1\2_\3|g"
Working out those quotes was fun lemme tell ya the regex itself should have been just xmlns:\w+\s*=\s*["'].+["'] but shells are assholes

Guess it's solved now, but if you need to use it again the cat file | perl | perl thing is kinda unnecessary, you can modify files in place with a single command:
perl -p -i -e 's/whatever/whatever1/g; s/whatever2/whatever3/g' *.kml

Add an extension after the -i (eg, -i.bak) and it'll also back up the original files for you using that extension.

# ? Aug 10, 2016 09:26

nielsm: Jun 1, 2009

Ciaphas posted:

Say I'm getting and expected to parse XML documents that look something like this.
XML code:
<kml xmlns:atom="foo" xmlns:atom="bar">
  <atom:author>John Doe</atom:author>
</kml>
...
<kml>
  <atom:author>Jane Doe</atom:author>
</kml>
Obviously these aren't valid XML at all, but god damnit if one of my customers is trying to get me to parse data like it anyway. The obvious manual fix works fine in both cases, but I have enough data that I need an automatic solution.

My first thought (and an extreme one, but I'm getting frustrated as hell right now) is to remove all instances of xmlns: attributes full stop, and change any namespaced elements to use dashes instead of colons (so <atom:author> becomes <atom-author>). Basically make the namespace thing and invalid XML go away in one fell nuclear step.

Unfortunately I know gently caress all about regexes in UNIX land. Can I do some command line magic to make these two steps happen?

Do those documents really have multiple root elements? If so, that's definitely a problem, and you'd have to wrap it in a new (fake) root element around the entire document minus XML declarations to make that part valid.

# ? Aug 10, 2016 11:33

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

pokeyman posted:

I was mostly serious. Surround your faux-json with console.log(JSON.stringify(...)) and run it through jsc or node or whatever JavaScript of your choosing.

My other scripts for this stuff are all in Perl; this isn't a web context. Specifically, this is me consuming a JSON listing of the files in my account on lpix.org, a site for helping with screenshot Let's Plays.

In any event, I don't consider a solution that relies on eval() to be more elegant than a solution that relies on string manipulation and regexes. :v:

JawnV6 posted:

The trick is to print the comma before the nth, not after the 1st

I unfortunately do not have control over the code that generates this JSON. And frankly it's dumb as hell that it's a parse error to have extra commas in JSON.

# ? Aug 10, 2016 12:33

Steve French: Sep 8, 2003

zerofunk posted:

I'm thinking about writing an iPhone app to dump all my pictures to S3 along with keeping track of meta data... probably in DynamoDB . It's mostly just an excuse to get some practice with things that I won't get any exposure to in my day job. Is there any legitimate reason to not have the iPhone app upload directly to S3 and instead go through some intermediate web service? I'm having trouble thinking of any good reasons to do that, but like I said I don't work with any of this stuff. The only possible concern I could think of would be security related since it means the app needs credentials for accessing S3. That's probably not a big deal though if you handle everything correctly. And if you don't, you're possibly going to mess up things between the app and your web service anyway.

I've designed and built a system that does this before (uploading photos to S3 directly from mobile clients). You can have your server (which does have credentials for accessing S3) generate a signed request for the client to use to upload a file.

http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

# ? Aug 10, 2016 14:45

Adbot: ADBOT LOVES YOU

# ? May 16, 2024 00:57

zerofunk: Apr 24, 2004

Steve French posted:

I've designed and built a system that does this before (uploading photos to S3 directly from mobile clients). You can have your server (which does have credentials for accessing S3) generate a signed request for the client to use to upload a file.

http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

Ahh, thanks for reminding me about that possibility. I had considered that, got distracted by some other things while digging through SDK documentation, and then kind of forgot about it.

# ? Aug 10, 2016 18:00

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »