Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Cingulate
Oct 23, 2012

by Fluffdaddy

Dominoes posted:

How does it relate and overlap with Tensorflow?
Scikit-learn (deliberately) has only very rudimentary support for neural networks, so not much. It's a unified framework for data preprocessing and applying standard machine learners like SVMs and trees.
The Tensorflow alternative in Python is Theano (and interfaces to Theano like Keras).

If at all, the Scikit-learn preprocessing tools, like one-hot encoding, should complement Tensorflow well.

Adbot
ADBOT LOVES YOU

Loezi
Dec 18, 2012

Never buy the cheap stuff

Nippashish posted:

You can use einsum to do it like this:
code:
sums = np.sign(np.einsum('ijk,ak->ija', weights, data)).sum(axis=1)
einsum is really powerful, but it tends to be slower than the standard BLAS ops so ymmv for speed.

Thanks for this. I've never encountered einsums before so I'll need to do some reading on those but that seems extremely useful.

chutwig
May 28, 2001

BURLAP SATCHEL OF CRACKERJACKS

What's the right way to express a complex type hint for a dictionary of the following format:
code:
{'command': 'some string', 'coord1': (0,0), 'coord2': (1,1)}
I tried rendering it as Dict[[str, str, str], [str, Tuple[int, int], Tuple[int, int]]], but the interpreter yakks on this with the error TypeError: Parameters to generic types must be types. Got [<class 'str'>, <class 'str'>, <class 'str'>]. PEP 0483/0484 are not clear on this (the only dict examples they give are a single key-value set) and using type hints is not widespread enough to turn anything up on Google.

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
There in none. In the complex type system, values of dictionaries are homogeneous. So you can't have two values with different types. Use a class if you want an object with named attributes.

chutwig
May 28, 2001

BURLAP SATCHEL OF CRACKERJACKS

Ah, I understand now. I re-rendered it as Tuple[str, Tuple[int,int], Tuple[int,int]], since the names weren't important.

QuarkJets
Sep 8, 2008

chutwig posted:

Ah, I understand now. I re-rendered it as Tuple[str, Tuple[int,int], Tuple[int,int]], since the names weren't important.

But you're basically creating a bunch of these tuples and then magic-indexing into them to get coord1 and coord2, right? You should really, really consider using a class instead; it'll be so, so much cleaner and easier to understand

Dominoes
Sep 20, 2007

QuarkJets posted:

But you're basically creating a bunch of these tuples and then magic-indexing into them to get coord1 and coord2, right? You should really, really consider using a class instead; it'll be so, so much cleaner and easier to understand
Or namedtuple

Flyndre
Sep 6, 2009
I'm trying to read a .csv-file into python, using the open-command. The file contains the Norwegian characters ÆØÅæøå.

When I print lines from this file containing these characters, they show up as question marks in the console, and string comparisons with variables created from this file obviously doesn't work either.

The code I'm trying to make work is basically something like:
code:
infile = open(filename, "r")
a, b = infile.readline().split()
if a ==  "Å":
	print True
This test never passes even though it should.


Any good tips on how to read this file properly into Python?

Edit: Solved the problem, first i determined the filetype using terminal "file <filename> -I", then I imported it correctly into Python using the codec module, before converting everything to utf-8 using .encode("utf-8")

Flyndre fucked around with this message at 19:53 on Dec 7, 2015

Space Kablooey
May 6, 2009


There may be other options that don't depend on an external lib, but :effort: . Did you try using unicodecsv? (Install it using pip install unicodecsv)

chutwig
May 28, 2001

BURLAP SATCHEL OF CRACKERJACKS

QuarkJets posted:

But you're basically creating a bunch of these tuples and then magic-indexing into them to get coord1 and coord2, right? You should really, really consider using a class instead; it'll be so, so much cleaner and easier to understand

It's a one-off script for advent of code, not something that anyone but me will ever look at. I know it would be cleaner to create an actual class and pass that around, but for this use case a tuple with magic indexes is fine.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

chutwig posted:

It's a one-off script for advent of code, not something that anyone but me will ever look at.

Why do you need to use type hinting then?

qntm
Jun 17, 2009

Flyndre posted:

I'm trying to read a .csv-file into python, using the open-command. The file contains the Norwegian characters ÆØÅæøå.

When I print lines from this file containing these characters, they show up as question marks in the console, and string comparisons with variables created from this file obviously doesn't work either.

The code I'm trying to make work is basically something like:
code:
infile = open(filename, "r")
a, b = infile.readline().split()
if a ==  "Å":
	print True
This test never passes even though it should.


Any good tips on how to read this file properly into Python?

What character encoding is the CSV file? Try specifying this encoding explicitly when you open() it in Python. Try tinkering with the encoding of your terminal to make them display correctly (in Windows I find myself needing to do something like `chcp 65001`). If that doesn't work, use `ord()` to get the Unicode code point, then you can look it up online. Make sure `a` has length 1. I don't know if the encoding of your Python script will affect anything here, it might.

Also the Norweigan text you gave doesn't start with "Å", but I assume that isn't the problem.

chutwig
May 28, 2001

BURLAP SATCHEL OF CRACKERJACKS

Hammerite posted:

Why do you need to use type hinting then?

Just practicing expressing things in it. Does there need to be a reason? It's different syntax from what I'm used to with Java/Go, so I figured I'd try to formulate the appropriate Python type hints.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

chutwig posted:

Just practicing expressing things in it. Does there need to be a reason?

I guess not. In that case go ahead I guess, although a reasonable answer to what you're asking might still be, "part of learning a tool is gaining an appreciation of when to use it and when not to use it, and this is a case where you should not use this tool but use a class instead".

I would give you an answer more helpful than that, but I don't really know anything about the type hinting feature, so I can't.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Type hinting has become second nature to me now so that I find myself using them in 20 line scripts for whatever reason.

In fact, I was providing some simple example code recently, and I realized I had some type hints in there. I took them out because they're not widely known (yet?) and I didn't want to confuse any potential users.

politicorific
Sep 15, 2007
I would like a gentle push in the right direction.

I document web interfaces at my job. I would like to automate capturing screen shots.

I've chosen Python as it's my strongest language. I've already checked this out:
https://automatetheboringstuff.com/chapter11/

Here are my problems:
  • Must work in Internet Explorer
  • Need to do rolling captures for extra long pages
  • Might need to keep track of sessions/logins/passwords
  • Need to avoid following "Save/Apply" pages which may reboot the router
  • I don't have a list of all the pages
  • The webui uses frames

For the first two points, I can use the screen capture program Greenshot to do rolling screen captures of Internet explorer and bind this to a short cut, or otherwise manipulate the program in the notification area. I also don't know how to pass passwords to a box using python.

Unfortunately the last point is beyond my css/html understanding. Our company uses frames on our web UIs to keep the menu options visable. The address bar simply reads "192.168.1.1" or whatever with the real 'page' hidden. I want to automatically scrape the the webpages for links, build a list, take a screenshot, and name them based on some text embedded in the page until finished. I've thought about using an offline page viewer application, but I'm not sure how that helps me.

If I had code that could do this, it would probably save me a few hundred hours of work per year, and perhaps a couple weeks of man-hours when multiplied out to my team.

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



Use Selenium's Python bindings and http://stackoverflow.com/questions/3422262/take-a-screenshot-with-selenium-webdriver

Part of their API lets you pick a frame to interact with http://stackoverflow.com/questions/13567128/selenium-frames

Or you can just record your own actions and replay them automagically if hand-coding the workflow is too much http://www.seleniumhq.org/projects/ide/

ButtWolf
Dec 30, 2004

by Jeffrey of YOSPOS
EDIT: Fixed

ButtWolf fucked around with this message at 18:40 on Dec 9, 2015

Risket
Apr 3, 2004
Lipstick Apathy
What is a good way to deal with a list of tuples? I'm just a touch above a beginner, so I feel like I'm missing something obvious

For context, I'm using pyserial to get the COM port name that contains a certain phrase. I couldn't figure out how to extract the COM port number using list comprehension or whatever, so I converted the list of tuples to a string and extracted the COM port
code:
import serial.tools.list_ports

#Get a list of ports that have .grep("Whatever") in the port description
#Returns a list of tuples like this:
#('COM8', 'Prolific USB-to-Serial Comm Port (COM8)', 'USB VID:PID=067B:2303 SNR=6')

temp =  list(serial.tools.list_ports.grep("Prolific"))

#Convert the list of tuples to a string
prPortFull = ''.join(str(temp)).strip('[]')

#Get the COM port number and decrement it
prComPortNum = (int( prPortFull[5])) -1

#Open the COM port
prSerPort = serial.Serial(prComPortNum)

"""
Do stuff....
"""
I know my method of getting the COM port from the string isn't very good. I can fix that, however my question is, is there a way of extracting the COM port number from the list of tuples without converting to a string? It seems like there should be a better way...

Emacs Headroom
Aug 2, 2003
You can index the tuple, so like

Python code:
for port_tuple in serial.tools.list_ports.grep("Prolific"):
    port_num = int(port_tuple[0][3:])
    print "Com port number:", port_num
but your life would probably be made a little easier by converting that data into a dictionary, with something like:
Python code:
def parse_port_data(port_tuple):
    port_num = int(port_tuple[0][3:])
    return {'com_port_num': port_num, 'description': port_tuple[1], 'info': port_tuple[2]}

for port_tuple in serial.tools.list_ports.grep("Prolific"):
    port_data = parse_port_data(port_tuple)
    print "Com port number:", port_data['com_port_num']
You can add some more parsing to pick out the things you might care about, like the SNR

Risket
Apr 3, 2004
Lipstick Apathy

Emacs Headroom posted:

but your life would probably be made a little easier by converting that data into a dictionary, with something like:
Python code:
def parse_port_data(port_tuple):
    port_num = int(port_tuple[0][3:])
    return {'com_port_num': port_num, 'description': port_tuple[1], 'info': port_tuple[2]}

for port_tuple in serial.tools.list_ports.grep("Prolific"):
    port_data = parse_port_data(port_tuple)
    print "Com port number:", port_data['com_port_num']
You can add some more parsing to pick out the things you might care about, like the SNR
Wow, this worked perfectly, thank you. I'm having some trouble figuring out what it's doing, but it's giving me a starting point for researching this. Thanks again!

politicorific
Sep 15, 2007

Munkeymon posted:

Use Selenium's Python bindings and http://stackoverflow.com/questions/3422262/take-a-screenshot-with-selenium-webdriver

Part of their API lets you pick a frame to interact with http://stackoverflow.com/questions/13567128/selenium-frames

Or you can just record your own actions and replay them automagically if hand-coding the workflow is too much http://www.seleniumhq.org/projects/ide/

Rats... selenium is Firefox only. Unless I learn this: https://www.youtube.com/watch?v=GxTHU_91Z1Q

I spent several hours researching possible solutions. This program webshot gave me some ideas:
http://www.websitescreenshots.com/usage.php#faq
But, unfortunately I will still need to take some screenshots by hand for certain dialogs which can't be automated and webshot isn't rendering the images to match the screenshots I took. I tried adjusting the registry to change Internet Explorer 11 to run in standards mode instead of quirks mode, but I didn't have any luck.

Here's what I came up with:

code:
from bs4 import BeautifulSoup
import webbrowser
import time
soup = BeautifulSoup(open("menu.html"))
linklist =[]
for link in soup.find_all('a'):
    print(link.get('href'))
    linklist.append(link.get('href'))
print(linklist)
    
for x in linklist:
        print(x)
        webbrowser.get("C:/Program Files/Internet Explorer/iexplore.exe %s").open(x)
        #time.sleep(2)
This scrapes a previously downloaded .html file, which I got using wget+cookie info, and then uses the webbrowser command to open the page in IE.

Next I want to use Greenshot to take a screenshot. I tried using SendKeys, but it hasn't been updated since 2003/2008 and doesn't work without visual studio installed.

Instead I found pywinauto.

https://github.com/pywinauto

However, I'm having trouble calling up a list of available windows, plus Greenshot runs in the taskbar.
https://pywinauto.googlecode.com/hg/pywinauto/docs/code/pywinauto.taskbar.html
pywinauto.taskbar.ClickSystemTrayIcon(button)

Anyone have any experience using this?

politicorific
Sep 15, 2007
Okay, I'm getting somewhere with pywinauto, I just need another kick in the right direction:

This clicks on the greenshot icon in the system tray/notification area. Greenshot produces a popup that disappears as soon as I click away, so wrote some code to try and find the window ID.
code:
from time import sleep
import pywinauto
from pywinauto.application import Application
from pywinauto import findwindows

app = pywinauto.application.Application().connect_(path = "explorer")
shelltraywnd = app.Shell_TrayWnd
shelltraywnd.Wait('ready')
toolbarwindow = shelltraywnd.Toolbar
toolbar_button = toolbarwindow.Button(u'Greenshot')
temp1=(findwindows.find_windows())
print(temp1)
toolbar_button.ClickInput()
sleep(2)
temp2 = (findwindows.find_windows())
print(temp2)
This produces two lists of open windows:
[262580, 525414, 265030, 722260, 132856, 1180792, 657050]
[66056, 461540, 262580, 525414, 265030, 722260, 132856, 1180792, 657050]

Which leads me to believe that 66056 is the greenshot popup window. Now all I need to do is tell it to click the 5th menu option down. But how do I do that? How do I connect to this window? How do I confirm that greenshot is 66056?

edit - found this text: "Often when you click/right click on an icon - you get a popup menu. The thing to remember at this point is that the popup menu is part of the application being automated not part of explorer." So it's possible I'm connected to the wrong thing.

politicorific fucked around with this message at 11:27 on Dec 10, 2015

Cingulate
Oct 23, 2012

by Fluffdaddy
How does iPython's ipyparallel compare to joblib?

OnceIWasAnOstrich
Jul 22, 2006

politicorific posted:

Rats... selenium is Firefox only. Unless I learn this: https://www.youtube.com/watch?v=GxTHU_91Z1Q


This is not the case. You just have to download the InternetExplorerDriverServer which is linked from the main Selenium download page. Similarly for Chrome/Safari you simply need to download and point your script to the correct executable for the driver executable.

Dominoes
Sep 20, 2007

How do I install MKL with the new Anaconda?

code:
Error: Unsatisfiable package specifications.
Generating hint:
[      COMPLETE      ]|##################################################| 100%


Hint: the following packages conflict with each other:
  - mkl
  - python 3.5*

Use 'conda info mkl' etc. to see the dependencies for each package.

Note that the following features are enabled:
  - mkl
  - vc14
Someone submissted this issue two weeks ago, with no followup. Here, someone thinks the 2.4 release of Anaconda will fix it; it doesn't.

SurgicalOntologist
Jun 17, 2004

What's vc14? I've had 3.5 and MKL in the same environments for a while now. But I don't have vc14.

OnceIWasAnOstrich
Jul 22, 2006

Dominoes posted:

How do I install MKL with the new Anaconda?

code:
Error: Unsatisfiable package specifications.
Generating hint:
[      COMPLETE      ]|##################################################| 100%


Hint: the following packages conflict with each other:
  - mkl
  - python 3.5*

Use 'conda info mkl' etc. to see the dependencies for each package.

Note that the following features are enabled:
  - mkl
  - vc14
Someone submissted this issue two weeks ago, with no followup. Here, someone thinks the 2.4 release of Anaconda will fix it; it doesn't.

Either don't install it into a 3.5 environment or wait until they build it for 3.5? This is how I interpreted it. It works in environments for every other version of Python, and that is what it looks like when any package is unavailable for a new version of Python.

BigRedDot
Mar 6, 2008

Anaconda 2.4.1 is out, seems to have the packages built (this is on OS X):
code:
bryan@0199-bryanv (git:server_host_arg) ~ $ conda create -n 35mkl python=3.5 numpy mkl
Remote site alias [url]https://api.anaconda.org[/url] does not exist in the config file
Fetching package metadata: ..............
Solving package specifications: .................
Package plan for installation in environment /Users/bryan/anaconda/envs/35mkl:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    mkl-rt-11.1                |               p0        66.6 MB
    mkl-service-1.1.0          |          py35_p0          18 KB
    numpy-1.10.1               |          py35_p0         2.6 MB
    scipy-0.16.0               |     np110py35_p1        10.7 MB
    scikit-learn-0.17          |     np110py35_p1         3.6 MB
    mkl-11.1                   |     np110py35_p1           4 KB
    ------------------------------------------------------------
                                           Total:        83.6 MB

The following NEW packages will be INSTALLED:

    mkl:          11.1-np110py35_p1  
    mkl-rt:       11.1-p0            
    mkl-service:  1.1.0-py35_p0      
    numpy:        1.10.1-py35_p0      [mkl]
    openssl:      1.0.2d-0           
    pip:          7.1.2-py35_0       
    python:       3.5.1-0            
    readline:     6.2-2              
    scikit-learn: 0.17-np110py35_p1   [mkl]
    scipy:        0.16.0-np110py35_p1 [mkl]
    setuptools:   18.5-py35_0        
    sqlite:       3.8.4.1-1          
    tk:           8.5.18-0           
    wheel:        0.26.0-py35_1      
    xz:           5.0.5-0            
    zlib:         1.2.8-0            

Proceed ([y]/n)? 
I don't know what that vcl4 feature up there is, though. I guess it's possible that's an issue.

BigRedDot fucked around with this message at 04:54 on Dec 13, 2015

pmchem
Jan 22, 2010


please bump mpi4py to 2.0.0 in anaconda 2.4.2 / 2.5, thanks~

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe
After someone in this thread posted about type annotations I took some time to read a little about them and thought they could help me solve a problem I've been mulling over to do with a personal project.

I started writing this personal project in Python but quickly found that it was getting to be quite hard work because I was wanting to enforce types in my methods all over the place; that's a lot of boilerplate and I realised after a while that I was trying to write statically typed code in Python. So I tried writing the same thing in C# but hit precisely the opposite problem where C# places all sorts of batty restrictions on what you can and can't do that are quite separate from being type-checked (what do you mean I can't have a method that returns an object that has the same type as "this" when called on a subclass object? what do you mean I can't have a constant value in my class that gets overridden by a subclass?) and found I wanted something more flexible again.

So, is anyone aware of any packages that enforce type annotations? A decorator seems like the natural thing. I found this but there are a couple of things about it that I'm wary of. It uses some funny annotations that don't match the ones shown off in PEP 0484 ({int: str}??) and I don't like all this talk of type coercion it does - if I gently caress up types I want to know about it, if I wanted the language to try that hard to fix it for me I'd use PHP.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Hammerite posted:

After someone in this thread posted about type annotations I took some time to read a little about them and thought they could help me solve a problem I've been mulling over to do with a personal project.

I started writing this personal project in Python but quickly found that it was getting to be quite hard work because I was wanting to enforce types in my methods all over the place; that's a lot of boilerplate and I realised after a while that I was trying to write statically typed code in Python. So I tried writing the same thing in C# but hit precisely the opposite problem where C# places all sorts of batty restrictions on what you can and can't do that are quite separate from being type-checked (what do you mean I can't have a method that returns an object that has the same type as "this" when called on a subclass object? what do you mean I can't have a constant value in my class that gets overridden by a subclass?) and found I wanted something more flexible again.

So, is anyone aware of any packages that enforce type annotations? A decorator seems like the natural thing. I found this but there are a couple of things about it that I'm wary of. It uses some funny annotations that don't match the ones shown off in PEP 0484 ({int: str}??) and I don't like all this talk of type coercion it does - if I gently caress up types I want to know about it, if I wanted the language to try that hard to fix it for me I'd use PHP.

mypy

The creator of MyPy, Jukka Lehtosalo, was pretty involved with the type annotations and stuff in Python 3.5, thus it's kind of the canonical type annotation consumer.

edit: Oh yeah, you can get stub files with type annotations for popular libraries at the typeshed.

Thermopyle fucked around with this message at 22:01 on Dec 13, 2015

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe
That's a very interesting project. It is all about statically analysing code, though. I suppose my post wasn't very explicit about what I have in mind. What I'm really thinking of is something that rather than try to analyse my code ahead of time, will check types when I call annotated functions. i.e. just as this code will throw when run:

code:
def f (x):
    pass

f()
so will this (where enforce_type_annotations is the decorator I'm envisaging):

code:
@enforce_type_annotations
def f (x: int):
    pass

f('Hello')
edit: To elaborate, I can see why someone would find value in static error checking, but I see a runtime check as be more in line with the flexible, dynamic model of execution that Python uses.

Hammerite fucked around with this message at 23:01 on Dec 13, 2015

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

I don't think anyone has anything like that out.

I guess what I'd ask is, what value do you envision getting out of adding type annotations and then using a library to parse those and check your types over just checking your types with isinstance or whatever logic?

I mean, there are advantages, but they're dependent upon the type of project and other factors.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Thermopyle posted:

I don't think anyone has anything like that out.

I guess what I'd ask is, what value do you envision getting out of adding type annotations and then using a library to parse those and check your types over just checking your types with isinstance or whatever logic?

I mean, there are advantages, but they're dependent upon the type of project and other factors.

If I have 10 methods with a parameter or two each, and I want to check that all of those parameters belong to classes I've approved, and raise an exception if not, then that quickly adds up to a lot of boilerplate. If there are several possible acceptable types for each parameter and I branch on the types, then including branches for the case where an unacceptable type was supplied adds noise to the code that impacts readability.

the talent deficit
Dec 20, 2003

self-deprecation is a very british trait, and problems can arise when the british attempt to do so with a foreign culture





i have a bunch of python scripts that i need to turn into a persistent service. flask is fine for what i am doing but i am clueless when it comes to running python in production. most of what i can find via google is aimed at people running on shared hosting or heroku or similar. is docker my best option here? do i need to reverse proxy from nginx/apache or can flask handle traffic directly?

Space Kablooey
May 6, 2009


If you can use a shared hosting, then I can't see why not use that.

Failing that, I use nginx + uWSGI. The biggest problem so far is that uWSGI prevents me from upgrading to Python 3. :( Otherwise, it works fine for me.

Space Kablooey fucked around with this message at 13:30 on Dec 14, 2015

Lysidas
Jul 26, 2002

John Diefenbaker is a madman who thinks he's John Diefenbaker.
Pillbug

HardDisk posted:

The biggest problem so far is that uWSGI prevents me from upgrading to Python 3. :( Otherwise, it works fine for me.

Why? I've been using uwsgi with Python 3 in production for almost two years, first with Python 3.2 and now 3.4.

Space Kablooey
May 6, 2009


That's nice, then.

I checked the WoS and uWSGI is listed as not supporting Python 3, and I thought that it couldn't run applications made with Python 3.

Adbot
ADBOT LOVES YOU

Coldstone Cream-my-pants
Jun 21, 2007
Is there a way to set variable type once and be done? Designating a variable as int(input(variable),10) every time I redefine it doesn't seem right.

If I don't do that it wants it to be a string, I guess because I'm using input but yeah.

  • Locked thread