Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
The Laplace Demon
Jul 23, 2009

"Oh dear! Oh dear! Heisenberg is a douche!"

hooah posted:

I have some 2-dimensional data in Matlab. I want to flip it vertically about the "center". That is, if the highest point's y-value is 6 and the lowest point's y-value is 1, a point which has an original y-value of 4 would have its value changed to 3. Is there a non-loop way to do this?

You want the data mirrored across the line y = (MAX(y) + MIN(y)) / 2 where MAX(y) is the maximum y value of your data and MIN(y) is defined similarly? Take every [x, y] and replace them with [x, (MAX(y) + MIN(y))/2 - y]

Adbot
ADBOT LOVES YOU

hooah
Feb 6, 2006
WTF?

The Laplace Demon posted:

You want the data mirrored across the line y = (MAX(y) + MIN(y)) / 2 where MAX(y) is the maximum y value of your data and MIN(y) is defined similarly? Take every [x, y] and replace them with [x, (MAX(y) + MIN(y))/2 - y]

That's pretty much what I've been doing, but in a loop. I was wondering if there's a non-looping way to do it.

Jabor
Jul 16, 2010

#1 Loser at SpaceChem

hooah posted:

That's pretty much what I've been doing, but in a loop. I was wondering if there's a non-looping way to do it.

You can pull out the column of y-values, apply your operation, then merge them back together:

code:
ys = data(:,2)
ys = (MAX + MIN) - ys
data = [data(:,1) ys]

ultrafilter
Aug 23, 2007

It's okay if you have any questions.


In this case, the transformation you're looking at is a permutation. All you need to do is left and right multiply by the appropriate permutation matrix, and you're good to go.

You may need a loop to initialize that permutation matrix, though. I don't know what facilities Matlab has for that.

lunar detritus
May 6, 2009


Nevermind, moving to web development thread!

lunar detritus fucked around with this message at 21:19 on Sep 17, 2015

Hughmoris
Apr 21, 2007
Let's go to the abyss!
I'm just now learning the basics of web scraping (for fun), and was hoping someone could answer this question at a high-level.

This link shows marine traffic activity at the Port of New Orleans, in a google maps-type window.
http://www.marinetraffic.com/en/ais/home/centerx:-90/centery:30/zoom:10

When the page loads, it has a map centered on New Orleans and it populates the map with ships that are in the area. Is it possible to scrape the ship information from the map? I'd like to pass my scraper a url with coordinates that I want, and it scrapes the information of any ships that display on the map. For this example, I ran a GET request to that URL from the python IDLE and I received a 403 Forbidden message.

Skandranon
Sep 6, 2008
fucking stupid, dont listen to me

Hughmoris posted:

I'm just now learning the basics of web scraping (for fun), and was hoping someone could answer this question at a high-level.

This link shows marine traffic activity at the Port of New Orleans, in a google maps-type window.
http://www.marinetraffic.com/en/ais/home/centerx:-90/centery:30/zoom:10

When the page loads, it has a map centered on New Orleans and it populates the map with ships that are in the area. Is it possible to scrape the ship information from the map? I'd like to pass my scraper a url with coordinates that I want, and it scrapes the information of any ships that display on the map. For this example, I ran a GET request to that URL from the python IDLE and I received a 403 Forbidden message.

It looks like the info mainly comes from the ViewportInfoService.GetViewportInfo, and it looks like it's sending along a token with the request, so you'll have to figure out how to get a token first. Just open up the page in Chrome, open up the Developer tools, and you can see all the network requests it makes.

Hughmoris
Apr 21, 2007
Let's go to the abyss!

Skandranon posted:

It looks like the info mainly comes from the ViewportInfoService.GetViewportInfo, and it looks like it's sending along a token with the request, so you'll have to figure out how to get a token first. Just open up the page in Chrome, open up the Developer tools, and you can see all the network requests it makes.

Using the developer tool, I started poking around. Looking in the network section, there is a file/link called "station:0". Double clicking on that took me to page that contains what might be ALL of the information I need, in json format:
http://www.marinetraffic.com/map/getjson/sw_x:-91.00000/sw_y:29.00000/ne_x:-88.00000/ne_y:32.00000/zoom:10/station:0

However, if I try to follow that link outside of my browser session, I'll receive the 403 error again. I'm stumped.

*Got it! I just copied/pasted all of my browser request header info into my Python request headers, and it works!

That will give me a nice JSON layout of the all the ship data (I think, still digging around). Man, it feels good to have solved a part of that. Been banging my head against it for two hours.

Hughmoris fucked around with this message at 00:49 on Sep 20, 2015

Tea Bone
Feb 18, 2011

I'm going for gasps.
I'm trying to do email verification for a small web app I wrote.

I've done what seems like the logical thing. Send an email with a link in like this: https://www.mysite.com/verify?uniquecode=qwerty

Where the 'uniquecode' is a unique string attributed to the e-mail address in the database.

The problem I'm having is my webserver seems to be following the links in the emails it sends out, so the addresses are automatically getting verified without the user having to read them/follow the link.

For the moment I'm just capturing the IP and if it's in the same range as my webserver the email isn't verified, but this feels hacky.

Has anybody else ever encountered something like this?

Tea Bone fucked around with this message at 22:35 on Sep 22, 2015

Gravity Pike
Feb 8, 2009

I find this discussion incredibly bland and disinteresting.
What do you mean, "your webserver"? What, briefly, does your stack look like? How are you sending the emails? Are you using a third-party service?

Hadlock
Nov 9, 2004

You mean like the spam filter (or whatever automated blah blah) is following the link and auto-verifying?

Two things I would do

1. Use cookies, the user that registered + the user that follows the link should use the same cookie (preferred)
2. Put some sort of delay in the system, i.e. email gets sent out, wait 90 seconds, then allow verification (very hacky but whatever). Put a message out there that says "we don't have your verification ready, yet. try back in 90 seconds" with some sort of live count-down and auto-refresh enabled

I'm not positive what the best practice is but #1 is probably loosely going down the correct path.

Tea Bone
Feb 18, 2011

I'm going for gasps.

Gravity Pike posted:

What do you mean, "your webserver"? What, briefly, does your stack look like? How are you sending the emails? Are you using a third-party service?

Yeah I'm just using a cheap third party hosting service, the server is running on LAMP and I'm sending the e-mails through PHP mail().

Hadlock posted:

You mean like the spam filter (or whatever automated blah blah) is following the link and auto-verifying?

Two things I would do

1. Use cookies, the user that registered + the user that follows the link should use the same cookie (preferred)
2. Put some sort of delay in the system, i.e. email gets sent out, wait 90 seconds, then allow verification (very hacky but whatever). Put a message out there that says "we don't have your verification ready, yet. try back in 90 seconds" with some sort of live count-down and auto-refresh enabled

I'm not positive what the best practice is but #1 is probably loosely going down the correct path.

Yeah that's exactly what's happening.

Thanks, as I said, for the moment I'm checking if the IP of whoever follows the link is in the same range as my site and ignoring it if so. It's a very small application (only expect max 30 users) so it's not likely I'll get any conflicts there. I considered the cookies but that would stop people for example signing up on their desktop then verifying on their phone.

I was just wondering if this is normal and there was some way of doing email verification I'm not aware of.

Impotence
Nov 8, 2010
Lipstick Apathy

Tea Bone posted:

I'm trying to do email verification for a small web app I wrote.

I've done what seems like the logical thing. Send an email with a link in like this: https://www.mysite.com/verify?uniquecode=qwerty

Where the 'uniquecode' is a unique string attributed to the e-mail address in the database.

The problem I'm having is my webserver seems to be following the links in the emails it sends out, so the addresses are automatically getting verified without the user having to read them/follow the link.

For the moment I'm just capturing the IP and if it's in the same range as my webserver the email isn't verified, but this feels hacky.

Has anybody else ever encountered something like this?

you're overthinking this, just make a button on the verification page that needs to be clicked and does a POST instead of GET

JamesieAB
Nov 5, 2005
I have a C# program that aggregates data from different sources into an array and I want to compare that data to a table in sql server.

Can I use LINQ to join my array with the sql server table? If so could someone point me to some example code?

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



JamesieAB posted:

I have a C# program that aggregates data from different sources into an array and I want to compare that data to a table in sql server.

Can I use LINQ to join my array with the sql server table? If so could someone point me to some example code?

C# code:
var things = new List<string>{"some","things","here"," - ","possibly","ints"};

//code for setting up database context elided - there are tons of examples online

var thingsInCommon = Table.Where(row => things.Contains(row.ColumnName));
Also, there's a .Net thread http://forums.somethingawful.com/showthread.php?threadid=3644791

hooah
Feb 6, 2006
WTF?
I have some 2-dimensional data which I want to remove outliers from in Matlab. The data isn't Gaussian at all, but it is symmetrical, so I figure that's a reasonable model to start out with, for this purpose. My process so far has been to find the mean and standard deviation of the data, then use that to discard points that are further than some number of standard deviations away from the mean. On the MathWorks website, I found a tutorial that does this for one-dimensional data and uses I = abs(fdata - ydata) > 1.5*std(ydata); to find the indices of outliers. I tried to extend this to two dimensions by doing I = (abs(gaussData(:, 2) - data(:, 2)) > 1.5 * sig(2)) & (abs(gaussData(:, 1) - data(:, 1)) > 1.5 * sig(1));, where data is the actual data and gaussData are the points generated calling mvnrnd using the estimated parameters. However, this removes some points in the denser part of the data and leaves points that visually are outliers. Where am I going wrong?

du -hast
Mar 12, 2003

BEHEAD THOSE WHO INSULT GENTOO
I'm not sure if this is the right place to ask this (if its not, lemme know).

I have a bunch of servers with Dell DRAC and Supermicro IPMI. Spefically we have 1600 servers with these remote access.

We are trying to set up all of them to Windows Active Directory. Is there any way to write a script to set up AD automatically? Doing it by hand is a loving bitch.

Gothmog1065
May 14, 2009
Going to put this in the general thread since this isn't Python specific, but more of a theory.

Okay, more of a theoretical question than a straight code question, but I'm still working on the Skynet game I started (Here is my ultimate revised code and Here is the game itself).

Now if you just play the "triple star" scenario, you'll see there's 3 gateways the "enemy" can go to. One of the achievements is to complete the game (trap the enemy so it can't get to any node) with 50 links available. I've gotten it close, but my current code leaves 41 links left, which means you have to trap the enemy on one of the stars and not let it out. My question is how do you predict where the enemy is going well enough to accomplish that? Playing out with my current code, it leaves three "wasted" link breakages, the rest stop them directly at the gateway.

I guess I'm trying to problem solve it now.

One way I could think of is to break certain links (the links on the rings that are only between two nodes) at the ends of the "stars", and when the virus goes around a star and has to come back, break the other side so it's trapped on the outer ring of a star, but I'm not sure how I'd go about determining where to break.

Here's some lovely psuedocode so you can kind of see my thought patterns:

code:
set loops = 0
for gateway in gateways:
    pull each node around the gateway (IE: pull all links with the gateway, and grab the nodes on the other side).
        put those in a tuple in a list called wheels or something 
    Go around and check the links to see which have more than one node (not the gateway) and set those as the "ends".
    verify the loop around the links. 
Other ideas while I'm thinking, is to just do a for loop over ALL The links (I do set a list of them as per my code above), and create the starting and ending node, and use one of my "free" breaks (the ones that just break the last link), to break one of the links on the circles themselves? am I just doing things the complicated way?

edit: Changing the way I'm working it, probably simpler.

Gothmog1065 fucked around with this message at 14:29 on Sep 24, 2015

ufarn
May 30, 2009
El Capitain is coming out, so that means making backups of all your packages and dotfiles. Can someone help me fill in the blanks for scripts to migrate the packages?

Python
code:
$ pip freeze > requirements.txt
:effort:

Node
code:
$ npm ls -g | grep -E "^(&#9500;|&#9492;)&#9472;" | cut -d" " -f2 |\
 awk -v quote='"' 'BEGIN { FS = "@" } ; { print quote $1 quote,":",quote $2 quote"," }' |\
 sed -e 's/ :/:/g' > package.json
Takes some rejiggering, but I think it suffices.

Ruby/rbenv
?

Homebrew
code:
$ brew list > brew-list.txt
It’s not a file you can install from like a requirements.txt, package.json, or Gemfile, but versioning shouldn’t be as important with Homebrew packages.

What do you normally do? Obviously I should hopefully be able to make things work out of Gemfiles and package.json's, but you never know what will break, and last upgrade to Yosemite was a nightmare.

ufarn fucked around with this message at 10:07 on Oct 1, 2015

Reik
Mar 8, 2004
I am not a programmer, I am an actuary, but I have a project I'm working on that involves automation of testing on a government website. The process we have in place from last year involved just running the site through Internet Explorer using VBA to fill out forms, push button, and pull text dumps from the pages in question. Unfortunately this year they updated the website and it appears our version of Internet Explorer, IE8, no longer works with this website. We do have access to Chrome thank god, but the VBA code we use only works in IE. I've looked up solutions to this problem and it looks like there are multiple simple solutions out there, but I'm trying to find one that meets the following restrictions:

1. Can't upgrade IE version.
2. Can't install any new software or plugins as these require approval from IT.

Is there any possible way to automate testing using VBA with Google Chrome? I know this isn't a hard problem, I just have some serious limitations due to Corporate America. I've recommended we request access for Firefox, the Selenium IDE plugin for Firefox, and the Selenium VBA plugin for Excel as a solution to this project, but I don't know if and when that can get approved.

Are there any solutions to this problem within my limitations? Is there some way to trick VBA in to thinking Chrome is actually Internet Explorer so that I can use the built-in objects and code stuff to automate Chrome?

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



So you can already use the VBA plugin or you're trying to get that approved? Because that seems like the only thing you'd need, other than also probably updating your VB to work with Selenium.

https://github.com/florentbr/SeleniumBasic is what you mean by the Selenium VBA plugin, right?

Skandranon
Sep 6, 2008
fucking stupid, dont listen to me

Reik posted:

I am not a programmer, I am an actuary, but I have a project I'm working on that involves automation of testing on a government website. The process we have in place from last year involved just running the site through Internet Explorer using VBA to fill out forms, push button, and pull text dumps from the pages in question. Unfortunately this year they updated the website and it appears our version of Internet Explorer, IE8, no longer works with this website. We do have access to Chrome thank god, but the VBA code we use only works in IE. I've looked up solutions to this problem and it looks like there are multiple simple solutions out there, but I'm trying to find one that meets the following restrictions:

1. Can't upgrade IE version.
2. Can't install any new software or plugins as these require approval from IT.

Is there any possible way to automate testing using VBA with Google Chrome? I know this isn't a hard problem, I just have some serious limitations due to Corporate America. I've recommended we request access for Firefox, the Selenium IDE plugin for Firefox, and the Selenium VBA plugin for Excel as a solution to this project, but I don't know if and when that can get approved.

Are there any solutions to this problem within my limitations? Is there some way to trick VBA in to thinking Chrome is actually Internet Explorer so that I can use the built-in objects and code stuff to automate Chrome?

Is there any way you can get a proper developer to do this? If you actually manage to get it working in VBA it will be an unholy abomination and will end up being a maintenance nightmare for whoever gets stuck with it. VBA macros are almost never the right answer.

Bruegels Fuckbooks
Sep 14, 2004

Now, listen - I know the two of you are very different from each other in a lot of ways, but you have to understand that as far as Grandpa's concerned, you're both pieces of shit! Yeah. I can prove it mathematically.

Reik posted:

I am not a programmer, I am an actuary, but I have a project I'm working on that involves automation of testing on a government website. The process we have in place from last year involved just running the site through Internet Explorer using VBA to fill out forms, push button, and pull text dumps from the pages in question. Unfortunately this year they updated the website and it appears our version of Internet Explorer, IE8, no longer works with this website. We do have access to Chrome thank god, but the VBA code we use only works in IE. I've looked up solutions to this problem and it looks like there are multiple simple solutions out there, but I'm trying to find one that meets the following restrictions:

1. Can't upgrade IE version.
2. Can't install any new software or plugins as these require approval from IT.

Is there any possible way to automate testing using VBA with Google Chrome? I know this isn't a hard problem, I just have some serious limitations due to Corporate America. I've recommended we request access for Firefox, the Selenium IDE plugin for Firefox, and the Selenium VBA plugin for Excel as a solution to this project, but I don't know if and when that can get approved.

Are there any solutions to this problem within my limitations? Is there some way to trick VBA in to thinking Chrome is actually Internet Explorer so that I can use the built-in objects and code stuff to automate Chrome?
You can't do OLE automation with Chrome.

Did you try playing with the compatibility view settings in IE to see if you could get the site to work, or changing the user agent string? It's extremely rare that something would have changed between IE versions that would break a site between IE6 and 8, yet work in Chrome at the same time - the site may just be checking browser versions and saying it doesn't work.

mystes
May 31, 2006

Bruegels Fuckbooks posted:

You can't do OLE automation with Chrome.

Did you try playing with the compatibility view settings in IE to see if you could get the site to work, or changing the user agent string? It's extremely rare that something would have changed between IE versions that would break a site between IE6 and 8, yet work in Chrome at the same time - the site may just be checking browser versions and saying it doesn't work.
It doesn't sound like Reik upgraded from IE 6 to 8 which broke the site. Rather, it sounds like the site now requires a version of IE newer than 8, which is why it also works in Chrome.

Jethro
Jun 1, 2000

I was raised on the dairy, Bitch!

Reik posted:

I am not a programmer, I am an actuary, but I have a project I'm working on that involves automation of testing on a government website. The process we have in place from last year involved just running the site through Internet Explorer using VBA to fill out forms, push button, and pull text dumps from the pages in question. Unfortunately this year they updated the website and it appears our version of Internet Explorer, IE8, no longer works with this website. We do have access to Chrome thank god, but the VBA code we use only works in IE.
If they updated the site such that it no longer works in IE8, I wouldn't be surprised to discover that they also broke your VBA script, even if you could get it to run.

MikeJF
Dec 20, 2003




Anyone know much about developing in Google App Engine? Python, specifically, although I don't think the language is particularly relevant.

I have to pull files off of Australia's Bureau of Meteorology for a webapp I've made on it. BoM's data services are 100% FTP-based. My attempt to pull files fails about, oh, 70% of the time. As far as I can tell, it's because (passive) FTP opens two sockets - one for control and one for data transfer. And thanks to the delightful app engine transparent behind-the-scenes balancing that you can't control, there's a chance they'll come from different IP addresses, in which case the FTP server will obviously reject the data connection attempt. I'm stumped for a workaround.

Hughmoris
Apr 21, 2007
Let's go to the abyss!
Anyone here have experience writing Medical Logic Modules (MLM) using Arden syntax?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
So in JIRA you can setup links between tickets. I don't think the terms we are using for the reciprocal relationship makes sense though, but I'm not sure what they should be instead.

XYZ-123 "is precedent for" XYZ-456
XYZ-456 "??" XYZ-123

XYZ-123 "depends on" XYZ-456
XYZ-456 "??" XYZ-123

What do guys think would make sense here?

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
We used "A is blocking B" to indicate that issue A must be resolved before meaningful progress can be made on B. We also use "B is blocked by A" for the regular depends-on relationship.

carry on then
Jul 10, 2010

by VideoGames

(and can't post for 10 years!)

Jabor posted:

We used "A is blocking B" to indicate that issue A must be resolved before meaningful progress can be made on B. We also use "B is blocked by A" for the regular depends-on relationship.

This is what our task management/source control system uses as well.

The MUMPSorceress
Jan 6, 2012


^SHTPSTS

Gary’s Answer
I feel like I need to start building up a portfolio of code that isn't proprietary to my company. What are some cool open source projects that are fairly accepting of newbie contributors? Preferably not something where my work is really just making money for some company offering the work as a product (looking at you Linux or Mozilla).

carry on then
Jul 10, 2010

by VideoGames

(and can't post for 10 years!)

Awful for Android?

The MUMPSorceress
Jan 6, 2012


^SHTPSTS

Gary’s Answer

carry on then posted:

Awful for Android?

I'm not sure if that would really improve my credibility :).

Gravity Pike
Feb 8, 2009

I find this discussion incredibly bland and disinteresting.

LeftistMuslimObama posted:

I feel like I need to start building up a portfolio of code that isn't proprietary to my company. What are some cool open source projects that are fairly accepting of newbie contributors? Preferably not something where my work is really just making money for some company offering the work as a product (looking at you Linux or Mozilla).

The easiest things to walk in and contribute, in my experience, are small bugfixes to library code. There are about a billion JavaScript libraries on github, and most of them don't work work particularly well. :P

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Jabor posted:

We used "A is blocking B" to indicate that issue A must be resolved before meaningful progress can be made on B. We also use "B is blocked by A" for the regular depends-on relationship.

carry on then posted:

This is what our task management/source control system uses as well.

Ok the blocking one makes sense, but what about the reciprocal of "is precedent for" ?

Shrimpy
May 18, 2004

Sir, I'm going to need to see your ticket.

Reik posted:

I am not a programmer, I am an actuary, but I have a project I'm working on that involves automation of testing on a government website. The process we have in place from last year involved just running the site through Internet Explorer using VBA to fill out forms, push button, and pull text dumps from the pages in question. Unfortunately this year they updated the website and it appears our version of Internet Explorer, IE8, no longer works with this website. We do have access to Chrome thank god, but the VBA code we use only works in IE. I've looked up solutions to this problem and it looks like there are multiple simple solutions out there, but I'm trying to find one that meets the following restrictions:

1. Can't upgrade IE version.
2. Can't install any new software or plugins as these require approval from IT.

Is there any possible way to automate testing using VBA with Google Chrome? I know this isn't a hard problem, I just have some serious limitations due to Corporate America. I've recommended we request access for Firefox, the Selenium IDE plugin for Firefox, and the Selenium VBA plugin for Excel as a solution to this project, but I don't know if and when that can get approved.

Are there any solutions to this problem within my limitations? Is there some way to trick VBA in to thinking Chrome is actually Internet Explorer so that I can use the built-in objects and code stuff to automate Chrome?

You could likely use the console in Chrome to do whatever you need to do in JavaScript.

https://developer.chrome.com/devtools/docs/authoring-development-workflow#snippets

You'd have to learn JavaScript, but considering you're just looking to manipulate some DOM elements, a good tutorial should be able to get you most of the way there.

carry on then
Jul 10, 2010

by VideoGames

(and can't post for 10 years!)

fletcher posted:

Ok the blocking one makes sense, but what about the reciprocal of "is precedent for" ?

Is preceded by?

1337JiveTurkey
Feb 17, 2005

fletcher posted:

Ok the blocking one makes sense, but what about the reciprocal of "is precedent for" ?

"Has precedent for"

JawnV6
Jul 4, 2004

So hot ...
How about antecedents?

Adbot
ADBOT LOVES YOU

Sex Tragedy
Jan 28, 2007

father of three with an extra large butt
Does anyone know about a virtual gamepad, controlled with python?

Sex Tragedy fucked around with this message at 21:42 on Oct 12, 2015

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply