Ask General Programming Questions Not Worth Their Own Thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »

ufarn: May 30, 2009

chutwig posted:

code:

$ jq '.[] | "<" + .email + "> " + .lastname + ", " + .firstname' some.json
"<foo> baz, bar"
"<goo> gaz, gar"
"<hoo> haz, har"

You can also pipe the JSON in.

Oh you can just straight up concatenate it into the input, I couldn't figure that out from the docs, cheers, bud.

# ? Dec 2, 2017 17:14

Adbot: ADBOT LOVES YOU

# ? May 30, 2024 16:02

SaTaMaS: Apr 18, 2003

This is more of a Linear Algebra question than General Programming, but maybe this is still the best place for it. I'm working on an Augmented Reality project, and I have five markers whose positions will vary somewhat every use, and I need to measure the distances between them to scale the 3D model I'm using. The markers are arranged in a pentagon like so:

pre:

  A
E   B
 D C

I use the AR library to measure the orthonormal affine transformations between A>B, B>C and A>E,E>D in order to construct a new orthonormal basis in the middle. The problem is that the AR library isn't capable of measuring the affine transformations with enough precision, so the errors accumulate, and the values I end up with for D>C don't line up and are about 1cm off. I can measure D>C, but I have no idea what sort of strategy I might use from there to adjust the previous matrices to make them more precise while keeping them orthonormal. Are there some techniques out there that could accomplish this?

SaTaMaS fucked around with this message at 18:04 on Dec 3, 2017

# ? Dec 3, 2017 18:02

Mr. Crow: May 22, 2008; Snap City mayor for life

The Fool posted:

Squarespace has ecommerce templates.

That�s what I would recommend.

https://www.squarespace.com/tour/ecommerce-website/

I'll pass it along, thanks.

# ? Dec 6, 2017 03:20

EPICAC: Mar 23, 2001

I didn�t see bash or awl threads so hopefully this is a good place to ask. I�m writing a bash script to analyze some data. I end up with a set of tab delimited files (100 in the short term, 800 once data is fully collected in the long term) of 500k-1m lines each, output by a previous program. These files are guaranteed to the same row order, and identical values in columns 1-10. The only difference is the values in column 11.

I want to merge these files into a single output file where I have the identical values from the file in columns 1-10, and then a column containing the values in column 11 for each file. So for 100 files, there would be 110 columns.

I was planning on outputting some headers for 1-10, and then processing the output of a wildcard expansion to get the relevant strings for column labels. Then using awk to assemble the rows for output.

So that gets me to my questions:
1) does wildcard expansion always give the same file order if the files with a given extension are unchanged.

2) Is there a way to slice in awk, similar to python?
My command is:

code:

 awk 'BEGIN {OFS="\t"}; {v[FNR]=(v[FNR] OFS $11)}; {$NF=""}; {p[FNR]=(p[FNR] $0)} END {for (i=1; i<=FNR; i++) { print p[i]  v[i]}}' *coverage > test.file

This almost works, but outputs columns 1-10 once for each file in the wildcard expansion. I�m not a fan of the extra space that $NF=�� gives. I can think of several ways to do this. One is just building p[] once for the first file, the second is slicing p[i] to print the first 10 columns.

I�m new to awk. I could do this very easily in python, but would prefer to just have everything I need contained in a single file.

# ? Dec 7, 2017 17:58

ButtWolf: Dec 30, 2004; by Jeffrey of YOSPOS

If you were gonna run a Traveling Salesman problem using a pi cluster with efficiency in mind, is there any specific language you'd go with, mainly for built in libraries or just speed in general?

# ? Dec 7, 2017 19:46

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

If you're only after efficiency at execution, and you have a firm grasp of hardware-level optimisations (cache misses, pipeline stalls, mutual exclusion, etc.) I doubt there're better options than C or a similar low-level language. Higher level languages all come with some kind of overhead for the abstractions they offer. That said, you'll probably be working a lot more for your solution in C than in something like Python by a significant factor.

# ? Dec 7, 2017 20:50

fritz: Jul 26, 2003

ButtWolf posted:

If you were gonna run a Traveling Salesman problem using a pi cluster with efficiency in mind, is there any specific language you'd go with, mainly for built in libraries or just speed in general?

First question I'd ask is why you're writing your own TSP solver instead of using one of the existing ones, second question is what kind of computing environment you're in that you're running on a bunch of rpis instead of something more capable, third question comes back around to the first and is how big are your TSP problems anyway because if they're small enough who cares.

# ? Dec 8, 2017 03:06

Newf: Feb 14, 2006; I appreciate hacky sack on a much deeper level than you.

fritz posted:

First question I'd ask is why you're writing your own TSP solver instead of using one of the existing ones, second question is what kind of computing environment you're in that you're running on a bunch of rpis instead of something more capable, third question comes back around to the first and is how big are your TSP problems anyway because if they're small enough who cares.

I hope the problems are all about packet routing on the network of pis.

# ? Dec 8, 2017 03:17

ButtWolf: Dec 30, 2004; by Jeffrey of YOSPOS

fritz posted:

First question I'd ask is why you're writing your own TSP solver instead of using one of the existing ones, second question is what kind of computing environment you're in that you're running on a bunch of rpis instead of something more capable, third question comes back around to the first and is how big are your TSP problems anyway because if they're small enough who cares.

1. I'm a student trying to learn stuff. 2. I haven't worked with pi before, the cluster is laying around the lab, and professor told me to mess with it. 3. Just for optimizations sake. There is some computing challenge, but I probably won't do it. I've been considering finishing my degree and not using it because I'm loving garbage at it.

# ? Dec 8, 2017 07:20

NihilCredo: Jun 6, 2011; iram omni possibili modo preme:
plus una illa te diffamabit, quam multæ virtutes commendabunt

Not sure if this is the right thread, let me know if there's a more suitable one. I've been working on desktop or client/server software for a few years and I'm slowly moving into the 2010s.

Let's say you have a network of servers running MyService in several physical locations, each of them exposing an API both on the Internet and on their local LAN (wifi). And you have a bunch of employee users, each with their own Android smartphone running MyApp, who move between locations usually (but not necessarily) connecting to each location's wifi.

When a user starts up MyApp, the app should ask the user which instance of MyService he wants to connect to (ideally suggesting the geographically nearest one), then it should figure out if the chosen instance is on the same LAN as the smartphone, and if so use the local address to connect to the server and upload/download data. (If the wifi gets turned off/on, the app should smoothly switch to the other connection.)

My first idea was to host in the cloud a dead-simple lookup 'service' (it could even be a single json file), where each instance of MyService would post their currently geographic location, public IP, WiFi SSID, and local IP. MyApp would check this list to figure out where to connect to.

But I get the suspicion that even this much may be overdoing it, and that there is a simpler way. Is there?

# ? Dec 8, 2017 18:18

lifg: Dec 4, 2000; <this tag left blank>; Muldoon

It sounds like you�re building a custom reverse proxy.

(Edit: not DNS)

lifg fucked around with this message at 21:14 on Dec 8, 2017

# ? Dec 8, 2017 21:07

NihilCredo: Jun 6, 2011; iram omni possibili modo preme:
plus una illa te diffamabit, quam multæ virtutes commendabunt

lifg posted:

It sounds like you�re building a custom reverse proxy.

(Edit: not DNS)

I thought about just doing that, but I don't want the data traffic to be bounced through the Internet if a local connection to the instance is available.

# ? Dec 8, 2017 21:30

nielsm: Jun 1, 2009

NihilCredo posted:

I thought about just doing that, but I don't want the data traffic to be bounced through the Internet if a local connection to the instance is available.

Is it okay if it's limited to discovering local instances within the broadcast domain? If so you could do an occasional discovery-broadcast while running, with a very short timeout (e.g. 20 ms) on getting a response, and use the local instance if it's discovered, otherwise discover one via internet DNS. By repeating the discovery once in a while you can switch over to use a local instance instead of an internet instance if the device enters a network with one.
If you do build a system like this, make sure to include good diagnostics so the user and any support people can tell what's really going on, and be careful not to flood the network with discovery traffic.

# ? Dec 8, 2017 21:53

NihilCredo: Jun 6, 2011; iram omni possibili modo preme:
plus una illa te diffamabit, quam multæ virtutes commendabunt

nielsm posted:

Is it okay if it's limited to discovering local instances within the broadcast domain? If so you could do an occasional discovery-broadcast while running, with a very short timeout (e.g. 20 ms) on getting a response, and use the local instance if it's discovered, otherwise discover one via internet DNS. By repeating the discovery once in a while you can switch over to use a local instance instead of an internet instance if the device enters a network with one.
If you do build a system like this, make sure to include good diagnostics so the user and any support people can tell what's really going on, and be careful not to flood the network with discovery traffic.

By 'discovery-broadcast', I assume you don't mean janitoring a raw IP multicast. Do you mean DNS-based service discovery? That looks a lot like what I've been looking for, and this looks like an implementation I could use on the server-side as well.

# ? Dec 8, 2017 22:27

The Fool: Oct 16, 2003

https://www.consul.io

could be an option if you're not inclined to roll your own service.

# ? Dec 8, 2017 22:44

csammis: Aug 26, 2003; Mental Institution

ButtWolf posted:

If you were gonna run a Traveling Salesman problem using a pi cluster with efficiency in mind, is there any specific language you'd go with, mainly for built in libraries or just speed in general?

Since you're a student and/or just doing this for kicks, get OpenMP installed on that cluster and then use whatever language has bindings for OpenMP. Here's someone who used C++.

# ? Dec 9, 2017 23:13

Hughmoris: Apr 21, 2007; Let's go to the abyss!

Does anyone have any recommended books/blogs/YT videos on learning to write SQL queries? My work has recently given me read-only access to a very large database, and I'm going to need to teach myself if I want to get anything out of it. I believe I'll be using SQL Server Management Studio.

# ? Dec 16, 2017 05:33

ultrafilter: Aug 23, 2007; It's okay if you have any questions.

The w3schools SQL tutorial seems pretty good, and it'll take you through pretty much all of the standard features.

# ? Dec 16, 2017 17:35

General_Failure: Apr 17, 2005

Does anyone else use the forum post equivalent of rubber ducky debugging? Just curious.

Sometimes when I'm absolutely stuck on something and can't find the information I'm after I write a post explaining the nature of the issue. The vast majority of the time I don't hit the submit button because in trying to explain the problem to others I've made the solution obvious.
I just did that with my arch nemesis, the Makefile.

# ? Dec 17, 2017 10:28

pokeyman: Nov 26, 2006; That elephant ate my entire platoon.

Forum posts, bug reports, slack messages, emails, yep. Getting the problem clear enough in your head to communicate it is very helpful in solving it!

# ? Dec 17, 2017 13:42

leper khan: Dec 28, 2010; Honest to god thinks Half Life 2 is a bad game. But at least he likes Monster Hunter.

pokeyman posted:

Forum posts, bug reports, slack messages, emails, yep. Getting the problem clear enough in your head to communicate it is very helpful in solving it!

I�ve definitely run into those. I wonder if there�s a next level version of it involving writing a book.

# ? Dec 17, 2017 17:15

Plorkyeran: Mar 22, 2007; To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed

Scott Meyer has half-jokingly said that writing books about C++ was a key step in actually understanding C++ for him.

# ? Dec 17, 2017 18:01

Hughmoris: Apr 21, 2007; Let's go to the abyss!

ultrafilter posted:

The w3schools SQL tutorial seems pretty good, and it'll take you through pretty much all of the standard features.

Thanks.

# ? Dec 17, 2017 18:34

Fergus Mac Roich: Nov 5, 2008; Soiled Meat

General_Failure posted:

Does anyone else use the forum post equivalent of rubber ducky debugging? Just curious.

Sometimes when I'm absolutely stuck on something and can't find the information I'm after I write a post explaining the nature of the issue. The vast majority of the time I don't hit the submit button because in trying to explain the problem to others I've made the solution obvious.
I just did that with my arch nemesis, the Makefile.

I used to do this but now I just use a notepad because it makes it easy to draw diagrams or flowcharts while I'm trying to put things into words.

# ? Dec 17, 2017 19:02

Seaside Loafer: Feb 7, 2012; Waiting for a train, I needed a shit. You won't bee-lieve what happened next

General_Failure posted:

Does anyone else use the forum post equivalent of rubber ducky debugging? Just curious.

Sometimes when I'm absolutely stuck on something and can't find the information I'm after I write a post explaining the nature of the issue. The vast majority of the time I don't hit the submit button because in trying to explain the problem to others I've made the solution obvious.
I just did that with my arch nemesis, the Makefile.

Yep for sure, once you have laid out the question you want to ask in a meaningful enough way to communicate to any medium/forum/group you can be sure you have properly thought it out and already have the answer before you have finished! Very useful approach I find. If you think about it, its just fully understanding what you want to know, and framing it in the form of a question answers it most of the time.

# ? Dec 17, 2017 19:29

Shy: Mar 20, 2010

What are the trial limitations on Sublime Text? I can't justify paying for it right now and will probably stick to VS and JetBrains anyway but it looks really neat. I installed it a few days ago and it's not asking for anything.

# ? Dec 18, 2017 18:28

Star War Sex Parrot: Oct 2, 2003

Shy posted:

What are the trial limitations on Sublime Text? I can't justify paying for it right now and will probably stick to VS and JetBrains anyway but it looks really neat. I installed it a few days ago and it's not asking for anything.

It'll just nag you occasionally. Otherwise I believe the trial is feature complete.

I could be wrong though since I have a license. I'm basing this off of classmates and coworkers who have used it without a license.

# ? Dec 18, 2017 18:32

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Shy posted:

What are the trial limitations on Sublime Text? I can't justify paying for it right now and will probably stick to VS and JetBrains anyway but it looks really neat. I installed it a few days ago and it's not asking for anything.

You can configure jetbrains ides to work and look very similar to ST...

# ? Dec 18, 2017 19:49

LLSix: Jan 20, 2010; The real power behind countless overlords

Shy posted:

What are the trial limitations on Sublime Text? I can't justify paying for it right now and will probably stick to VS and JetBrains anyway but it looks really neat. I installed it a few days ago and it's not asking for anything.

I've been using it for a couple months without a license. It nags be to get one periodically but I haven't yet found anything that I can't do with Sublime.

# ? Dec 18, 2017 22:12

The Fool: Oct 16, 2003

Obligitory "Have you tried VS Code yet?" post

https://code.visualstudio.com/

Supported Languages

Python

Javascript

# ? Dec 18, 2017 22:17

Shy: Mar 20, 2010

It's alright but I'm not a fan of "actually a browser" applications.

# ? Dec 18, 2017 22:21

The Fool: Oct 16, 2003

Shy posted:

It's alright but I'm not a fan of "actually a browser" applications.

Fair enough, however I feel like that is an over-simplification of what Electron is doing.

Of the Electron apps that I'm aware of, VS:Code is definitely the tightest.

It is currently taking up less RAM than Task Manager on my workstation.

Slack is taking up nearly 3 times that.

# ? Dec 18, 2017 22:26

Seventh Arrow: Jan 26, 2005

I'm working on a web scraping project and I want to capture certain posts from the "Big Data Jobs" subreddit.

I'm only looking for posts where someone is hiring and I noticed that these all start with some variation of "[Hiring]", "[HIRING]", "[hiring]", etc. Here's a typical title:

[HIRING] Democratic National Committee: Data Science Lead (jobs.lever.co)

So I thought a regex might do the trick...but I'm fairly new to regular expression and am not sure about some things. Here's what I've come up with so far:

\[[Hh][Ii][Rr][Ii][Nn][Gg]\][A-Za-z0-9\:] \(

So the idea is that I would start off by escaping the special character "[". Then I look for various case permutations of "hiring." Then I escape "]". The part after that can be any character, but I'm not sure how to get it to stop. I could get it to stop at "(jobs.level.co)" by including a "(" but I think that's in a different text field. Any suggestions?

# ? Dec 19, 2017 02:59

Shy: Mar 20, 2010

e: deleted, misunderstood

Shy fucked around with this message at 03:31 on Dec 19, 2017

# ? Dec 19, 2017 03:09

huhu: Feb 24, 2006

Seventh Arrow posted:

I'm working on a web scraping project and I want to capture certain posts from the "Big Data Jobs" subreddit.

I'm only looking for posts where someone is hiring and I noticed that these all start with some variation of "[Hiring]", "[HIRING]", "[hiring]", etc. Here's a typical title:

[HIRING] Democratic National Committee: Data Science Lead (jobs.lever.co)

So I thought a regex might do the trick...but I'm fairly new to regular expression and am not sure about some things. Here's what I've come up with so far:

\[[Hh][Ii][Rr][Ii][Nn][Gg]\][A-Za-z0-9\:] \(

So the idea is that I would start off by escaping the special character "[". Then I look for various case permutations of "hiring." Then I escape "]". The part after that can be any character, but I'm not sure how to get it to stop. I could get it to stop at "(jobs.level.co)" by including a "(" but I think that's in a different text field. Any suggestions?

You should learn your programming language's string methods. I feel like 99% of the time I can get away with using them to search a string instead of using regex. In Python you could do -

code:

seach_term = 'hiring'
search_string = '[HIRING] Democratic National Committee: Data Science Lead (jobs.lever.co)'


search_term in search_string.lower() # convert to lower case
# -> True

# ? Dec 19, 2017 03:31

Seventh Arrow: Jan 26, 2005

Actually, I am using python so thanks for the tip. Mainly I wanted BeautifulSoup to just search for the "[hiring]" part and whatever comes after that is irrelevant. Is that still possible when searching a string? What about issues with lower case vs. upper case?

# ? Dec 19, 2017 04:17

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

Seventh Arrow posted:

Actually, I am using python so thanks for the tip. Mainly I wanted BeautifulSoup to just search for the "[hiring]" part and whatever comes after that is irrelevant. Is that still possible when searching a string? What about issues with lower case vs. upper case?

Python strings have a ".lower()" function that converts the text in them to lowercase, which huhu used in their example.

# ? Dec 19, 2017 04:22

Seaside Loafer: Feb 7, 2012; Waiting for a train, I needed a shit. You won't bee-lieve what happened next

Seventh Arrow posted:

I'm working on a web scraping project and I want to capture certain posts from the "Big Data Jobs" subreddit.

I'm only looking for posts where someone is hiring and I noticed that these all start with some variation of "[Hiring]", "[HIRING]", "[hiring]", etc. Here's a typical title:

[HIRING] Democratic National Committee: Data Science Lead (jobs.lever.co)

So I thought a regex might do the trick...but I'm fairly new to regular expression and am not sure about some things. Here's what I've come up with so far:

\[[Hh][Ii][Rr][Ii][Nn][Gg]\][A-Za-z0-9\:] \(

So the idea is that I would start off by escaping the special character "[". Then I look for various case permutations of "hiring." Then I escape "]". The part after that can be any character, but I'm not sure how to get it to stop. I could get it to stop at "(jobs.level.co)" by including a "(" but I think that's in a different text field. Any suggestions?

I think most of the high level language string search/manipulation library's at their root use the same algorithms that have been worked out by people much cleverer than you or me (and quite possibly optimized by those same clever people) so just use them instead of trying to make your own special one imo.

e: what im saying is the functionality of regex is encapsulated in the high level languages string librarys if that makes sense.

Seaside Loafer fucked around with this message at 06:12 on Dec 19, 2017

# ? Dec 19, 2017 06:04

Seventh Arrow: Jan 26, 2005

TooMuchAbstraction posted:

Python strings have a ".lower()" function that converts the text in them to lowercase, which huhu used in their example.

Woops, you're right, thanks...I was looking right at it and somehow it didn't register :eng99:

# ? Dec 19, 2017 06:05

Adbot: ADBOT LOVES YOU

# ? May 30, 2024 16:02

TheresaJayne: Jul 1, 2011

Seaside Loafer posted:

I think most of the high level language string search/manipulation library's at their root use the same algorithms that have been worked out by people much cleverer than you or me (and quite possibly optimized by those same clever people) so just use them instead of trying to make your own special one imo.

e: what im saying is the functionality of regex is encapsulated in the high level languages string librarys if that makes sense.

Why not first toLower() the whole content and then the search would be easier?

# ? Dec 19, 2017 09:52

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »