Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
Well, what does the 2D numpy array contain? Information design is your friend here.

Adbot
ADBOT LOVES YOU

QuarkJets
Sep 8, 2008

Suspicious Dish posted:

Well, what does the 2D numpy array contain? Information design is your friend here.

Sorry; each 2D numpy array is an image.

Dominoes
Sep 20, 2007

QuarkJets posted:

I would like to create a GUI with PyQt that displays a 2D numpy array. Is this relatively straightforward? Is it better to convert the image to a QPixMap or a QImage? Or is it better to try to embed matplotlib into my PyQt GUI? I'm creating a slider so that the displayed image can be switched rapidly, so I was imagining that refreshing matplotlib over and over might be suboptimal, but maybe not?
Let me know if you figure this out. I've been looking for something similar as of yesterday (matPLotlib plot in a Qt Widget), but am only in the Google search stages. There are several articles, like this, but they seem to be out of date.

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe

QuarkJets posted:

Sorry; each 2D numpy array is an image.

So each element in the 2D array is a pixel? Just going to ask -- why did you use such an inefficient method of storage? You might be able to use scanData from PySide, I'm not sure.

Emacs Headroom
Aug 2, 2003

Suspicious Dish posted:

So each element in the 2D array is a pixel? Just going to ask -- why did you use such an inefficient method of storage? You might be able to use scanData from PySide, I'm not sure.

Probably because it's easy and intuitive?

I'd use either PIL or matplotlib, whichever is easier to embed (probably matlotlib if you're not doing color images).

Don't forget about numpy.flipud if your images end up upside-down (I think PIL and matplotlib imshow have opposite conventions on that).

QuarkJets
Sep 8, 2008

Suspicious Dish posted:

So each element in the 2D array is a pixel? Just going to ask -- why did you use such an inefficient method of storage? You might be able to use scanData from PySide, I'm not sure.

Yes, each element is a pixel value. I'm reading FITS files, mostly used in astronomy. PyFits reads FITS image data as a 2D numpy array.

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
OK, yeah. The issue is that image libraries, for efficiency reasons, like to align their buffers on native boundaries, so if you have an image that's 50x50, it will bump it up to 52x52 internally.

This means that the data has to be copied from the numpy buffer into the Qt buffer, and that it has to be copied line-by-line. I'm not exactly sure if you have any API for this other than "setPixel", and that's going to be horrifically slow. As I said, I'm not sure if "scanData" is available from Python, but that seems to be the API that exists for low-level pixel manipulation.

QuarkJets
Sep 8, 2008

Suspicious Dish posted:

OK, yeah. The issue is that image libraries, for efficiency reasons, like to align their buffers on native boundaries, so if you have an image that's 50x50, it will bump it up to 52x52 internally.

This means that the data has to be copied from the numpy buffer into the Qt buffer, and that it has to be copied line-by-line. I'm not exactly sure if you have any API for this other than "setPixel", and that's going to be horrifically slow. As I said, I'm not sure if "scanData" is available from Python, but that seems to be the API that exists for low-level pixel manipulation.

There are a few other techniques that I've reviewed, I just wasn't sure which was best. Embedding matplotlib directly obviously means that I don't have to use any of the Qt image methods directly, but there are some other Qt features that I'd like to use with the image data so this is not something that I'll be doing. QImage supports a 2D numpy array as an input in the constructor, and I could also create a QPixMap from a QImage, I just really don't understand the differences between a QImage and a QPixMap and why I would use one or the other. The documentation on these classes was unclear and difficult to understand. It sounds like QPixMap is better if you plan to not manipulate the image directly because it gets cached? Are there other speed advantages for one vs the other?

QuarkJets fucked around with this message at 00:40 on Jul 17, 2013

accipter
Sep 12, 2003

QuarkJets posted:

There are a few other techniques that I've reviewed, I just wasn't sure which was best. Embedding matplotlib directly obviously means that I don't have to use any of the Qt image methods directly, but there are some other Qt features that I'd like to use with the image data so this is not something that I'll be doing. QImage supports a 2D numpy array as an input in the constructor, and I could also create a QPixMap from a QImage, I just really don't understand the differences between a QImage and a QPixMap and why I would use one or the other. The documentation on these classes was unclear and difficult to understand. It sounds like QPixMap is better if you plan to not manipulate the image directly because it gets cached? Are there other speed advantages for one vs the other?

QImage is used for reading/writing of files, but before any QImage is shown it has to be converted to a QPixmap. Here is the summary from the Qt documentation:

quote:

Qt provides four classes for handling image data: QImage, QPixmap, QBitmap and QPicture. QImage is designed and optimized for I/O, and for direct pixel access and manipulation, while QPixmap is designed and optimized for showing images on screen. QBitmap is only a convenience class that inherits QPixmap, ensuring a depth of 1. The isQBitmap() function returns true if a QPixmap object is really a bitmap, otherwise returns false. Finally, the QPicture class is a paint device that records and replays QPainter commands.

QuarkJets
Sep 8, 2008

Thank you, that helps a lot; it sounds like a QPixmap is what I want to be messing with. Thanks!

Bohemian Cowabunga
Mar 24, 2008

I have a script that I have (reluctantly) been asked to instruct some semi-technical users to use and that have presented some issues.

My main problem is that impatient users will just hammer away on their keyboard without reading the progress info the script outputs which often results in wrong info being passed to an input().

My initial thought on how to solve this, is to push the loop into its own thread and block the main thread until it is done, but was wondering if there was a better way of blocking/redirecting user input during the loop?

Mrs. Wynand
Nov 23, 2002

DLT 4EVA

Bohemian Cowabunga posted:

I have a script that I have (reluctantly) been asked to instruct some semi-technical users to use and that have presented some issues.

My main problem is that impatient users will just hammer away on their keyboard without reading the progress info the script outputs which often results in wrong info being passed to an input().

My initial thought on how to solve this, is to push the loop into its own thread and block the main thread until it is done, but was wondering if there was a better way of blocking/redirecting user input during the loop?

So when the script is invoked they get an input() prompt and basically just call arbitrary functions to use it? Yikes.

Why is input() still reading when stuff is happening?

BeefofAges
Jun 5, 2004

Cry 'Havoc!', and let slip the cows of war.

Why are you using input() at all? Why not just have a config file, or not let users customize the script at all?

Bohemian Cowabunga
Mar 24, 2008

That is my main issue, when the loop is running you can hammer away on the keyboard and it will get buffered (probably not the right term) and read by input() when it is called after the loop has finished.

The script running will need input from the user as its more of tool to simplify a work task that I initially just made for myself.

Without going too much into detail, the script retrieves some info from an external SOAP interface where the user has to provide a username and password and then be able perform a custom search on the retrieved data.

It is during the retrieval of the data I need to block user input because certain people cannot read and just wait for it to finish.

For me it is not an issue to just read the input on the screen and wait, but I have been told that it should be "fixed".

Bohemian Cowabunga fucked around with this message at 23:42 on Jul 18, 2013

Mrs. Wynand
Nov 23, 2002

DLT 4EVA
I still don't quite get how input() is running at all during the retrieval unless you are already using a multithreaded or at least async lib. If you are then there is usually a way to control tasks and make them dependent on one of them finishing. I mean you said something about it being in a loop, so... stop the loop?

If you really really want to do this via input streams, sure, I mean you could just use something other then raw_input/input to which you can explicitly pass the input stream. This strikes me as a terribly awkward way of doing things though when you can just as easily control the thing reading from that stream (i.e. the input call) - at least from where I'm standing.

I'm sure there is a god awful way of suppressing stdin globally but that should almost certainly be your last option.

Dominoes
Sep 20, 2007

Hoping someone can help with understanding HTTP streaming. I have a stock market program that periodically checks for certain market conditions. I do this using a loop and a sleep timer; I make a request using the Requests module, analyze the data, then time.sleep for 5 minutes. My broker has a streaming API. I think this means I can use a single request that stays open, and constantly display or analyze it, without triggering rate limiting.

However, the way I'm using it seems to do the same thing as the non-streaming api call. Both the API and Requests' documentation on this is terse.

Paraphrase of my attempt:

Python code:
s = requests.Session()
def streaming(symbols):
    url = 'market/quotes.json'
    payload = {'symbols': ','.join(symbols)}
    return s.get(HOST + url, params=payload, stream=True).json()

stream = streaming(['GOOG', 'MSFT'])
while True:
    print(stream)
    time.sleep(1)
This prints the same feed repeatedly, instead of a new feed whenever updated. I have no idea what this structure should look like or what streaming HTTP requests are supposed to do.

Dominoes fucked around with this message at 00:15 on Jul 19, 2013

Bohemian Cowabunga
Mar 24, 2008

Oh don't get me wrong, I don't really wanna do this at all but made the mistake of showing a coworker how to do a 10 min task in 1 min.
I am not a programmer but doing admin / 2nd level support and just use python to make my life easier.
If it makes more sense about the behaviour

I will have a look at it tomorrow at work.

Mrs. Wynand
Nov 23, 2002

DLT 4EVA

Dominoes posted:

Hoping someone can help with understanding HTTP streaming. I have a stock market program that periodically checks for certain market conditions. I do this using a loop and a sleep timer; I make a request using the Requests module, analyze the data, then time.sleep for 5 minutes. My broker has a streaming API. I think this means I can use a single request that stays open, and constantly display or analyze it, without triggering rate limiting.

That's kind of weird for HTTP.

Does the response have a "Transfer-Encoding: Chunked" header? I guess they could be sending down a coherent blob of json each chunk, but how do you tell it to end?

I mean, there are ways, but they should be documented in the API docs otherwise you are just blindly guessing.

quote:

However, the way I'm using it seems to do the same thing as the non-streaming api call. Both the API and Requests' documentation on this is terse.

Paraphrase of my attempt:

Python code:
s = requests.Session()
def streaming(symbols):
    url = 'market/quotes.json'
    payload = {'symbols': ','.join(symbols)}
    return s.get(HOST + url, params=payload, stream=True).json()

stream = streaming(['GOOG', 'MSFT'])
while True:
    print(stream)
    time.sleep(1)
This prints the same feed repeatedly, instead of a new feed whenever updated. I have no idea what this structure should look like or what streaming HTTP requests are supposed to do.


Yeah the http lib is a loving mess, and so are its docs. It certainly can do streaming, though usually that is done when receiving a massive response (e.g. a large file) and you don't want it all in memory. Even for that though, the way you do is weird and arcane. I ended up having to just code http processing from scratch out of sockets, for this one weird cases where I needed streaming request processing. It really is a mess. (I think python 3 is cleaning it up, but whatever, that is star trek future poo poo)

What you are doing otoh is none of the above. You aren't actually doing any http whatsoever between the loops - all that is already done in stream = streaming(['GOOG', 'MSFT']). All you're doing is printing out the same result every second.

If you figure out what their actual API is for "streaming" quotes I can probably give you something more concrete.

Mrs. Wynand
Nov 23, 2002

DLT 4EVA

Bohemian Cowabunga posted:

Oh don't get me wrong, I don't really wanna do this at all but made the mistake of showing a coworker how to do a 10 min task in 1 min.
I am not a programmer but doing admin / 2nd level support and just use python to make my life easier.
If it makes more sense about the behaviour

I will have a look at it tomorrow at work.

No good deed goes unpunished.

Just add more exclamation marks in the warnings :v:

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



Mr. Wynand posted:

I still don't quite get how input() is running at all during the retrieval unless you are already using a multithreaded or at least async lib. If you are then there is usually a way to control tasks and make them dependent on one of them finishing. I mean you said something about it being in a loop, so... stop the loop?

If you really really want to do this via input streams, sure, I mean you could just use something other then raw_input/input to which you can explicitly pass the input stream. This strikes me as a terribly awkward way of doing things though when you can just as easily control the thing reading from that stream (i.e. the input call) - at least from where I'm standing.

I'm sure there is a god awful way of suppressing stdin globally but that should almost certainly be your last option.

stdin is a pipe. Pipes are buffered. Input is being buffered during the long-running operation and handed to raw_input after it gets done. This is easy to demonstrate:
Python code:
from time import sleep

print "Hello and good night"

sleep(20)

print raw_input("What was that? ")
print "Oh"
code:
$ ./input_test.py
Hello and good night
blah blah blah I'm an impatient rear end in a top hat with the self-control of a three year old
What was that? blah blah blah I'm an impatient rear end in a top hat with the self-control of a three year old
Oh
I can't quite figure out what to do about it, though. Reading from stdin will block no matter whether anything is there unless the user enters Ctrl + D (Z on Windows), so you can't just 'read until EOF' to clear it out. Truncating it breaks raw_input. And you can't write to it to inject your own EOF (which might not work anyway thanks to the vagaries of how EOF is handled on different OSes).

E: oh, ⏬ ⏬ well there you go ⏬ ⏬.

Munkeymon fucked around with this message at 18:04 on Jul 19, 2013

BeefofAges
Jun 5, 2004

Cry 'Havoc!', and let slip the cows of war.

Read from stdin in a 2nd thread, put the output in a queue, then do a nonblocking read on that queue in your main thread.

Mrs. Wynand
Nov 23, 2002

DLT 4EVA

Munkeymon posted:

stdin is a pipe. Pipes are buffered. Input is being buffered during the long-running operation and handed to raw_input after it gets done. This is easy to demonstrate:

Ohhhhhhhhhh right. Well, I feel foolish.

Is the script on a linux of sorts? If so, the termios module provides explicit control over terminal input, and is probably most appropriate here.

evensevenone
May 12, 2001
Glass is a solid.
Have you tried sys.stdin.flush()? I can't test it right now and terminals seem to defy all common sense but feel like that might work if you put it immediately before the input command.

Mrs. Wynand
Nov 23, 2002

DLT 4EVA

evensevenone posted:

Have you tried sys.stdin.flush()? I can't test it right now and terminals seem to defy all common sense but feel like that might work if you put it immediately before the input command.

That occurred to me too, but I think flush is for writing.

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



evensevenone posted:

Have you tried sys.stdin.flush()? I can't test it right now and terminals seem to defy all common sense but feel like that might work if you put it immediately before the input command.

Tried it. Doesn't do anything.

QuarkJets
Sep 8, 2008

Munkeymon posted:

Tried it. Doesn't do anything.

Is it possible to tell your students not to pound on the keyboard inbetween stdin reads?

An alternative is to use PyQt or something to create a text box for input. Text box pops up, students put something in and hit Okay, text box input gets used instead of stdin.

Marsol0
Jun 6, 2004
No avatar. I just saved you some load time. You're welcome.
Are you on a *nix system? If so, you could try using select to read and discard everything from stdin before prompting for input. Basically poll stdin to see if there is any data to read, read all of it, discard that stuff, and then prompt for actual user input.

Dominoes
Sep 20, 2007

Mr. Wynand posted:

That's kind of weird for HTTP.

Does the response have a "Transfer-Encoding: Chunked" header? I guess they could be sending down a coherent blob of json each chunk, but how do you tell it to end?

I mean, there are ways, but they should be documented in the API docs otherwise you are just blindly guessing.



Yeah the http lib is a loving mess, and so are its docs. It certainly can do streaming, though usually that is done when receiving a massive response (e.g. a large file) and you don't want it all in memory. Even for that though, the way you do is weird and arcane. I ended up having to just code http processing from scratch out of sockets, for this one weird cases where I needed streaming request processing. It really is a mess. (I think python 3 is cleaning it up, but whatever, that is star trek future poo poo)

What you are doing otoh is none of the above. You aren't actually doing any http whatsoever between the loops - all that is already done in stream = streaming(['GOOG', 'MSFT']). All you're doing is printing out the same result every second.

If you figure out what their actual API is for "streaming" quotes I can probably give you something more concrete.
There's no "Transfer-Encoding: Chunked" listed when running .headers on the requests object. Here's what the API docs (TradeKing) says about it: "We’ve enabled a streaming endpoint to for requesting both quote and trade data utilizing a persistent HTTP socket connection. Streaming data from the API consists of making an Authenticated HTTP request and leaving the HTTP socket open to continually receive data." I assumed "streaming" was a known HTTP request type that I don't understand. It seems like Requests' "stream" property may actually be for iterating over a response, not for streaming APIs, based on the docs.

Houston Rockets
Apr 15, 2006

Read the docs? You need to use iter_lines()

code:
import requests

r = requests.get('http://httpbin.org/stream/20', stream=True)

for line in r.iter_lines():
    print line
http://docs.python-requests.org/en/latest/user/advanced.html#streaming-requests

Mrs. Wynand
Nov 23, 2002

DLT 4EVA
There must be more to it then that. Without chunked encoding the response length has to be set ahead of time, in the Content-Size header, and if you are reciveing continiously updated data, they can't possibly know that.

Or are they just abusing http and closing the socket whenever they feel like it, without either content-size or transfer-encoding set?

The APIs ought to provide more details.

IMlemon
Dec 29, 2008
snip

IMlemon fucked around with this message at 19:44 on Jul 12, 2015

QuarkJets
Sep 8, 2008

Is there any reason that you're doing this with a lambda instead of a function?

code:
    def alive_neighbors(self):
        return len(filter(lambda c: c.is_alive(), self.neighbours))
And maybe it's just me, but I think that this looks weird just because I don't normally use _ as a variable name

code:
grid = [[Cell(True) for _ in xrange(size)] for _ in xrange(size)]
This is a fine use of lists, but you might consider using numpy arrays instead, as an exercise. They're faster and more useful for some things. Lists are nice in that they can contain anything at all

Overall I think your code looks pretty good

evensevenone
May 12, 2001
Glass is a solid.
This surprised some folks at work:

Python code:
foo = 'bar'

def printfoo():
    print foo

def printfoo2():
    if False:
        foo = 'foo'
    print foo
Python code:
In [1]: import foo

In [2]: foo.printfoo()
bar

In [3]: foo.printfoo2()
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-3-981b642201ab> in <module>()
----> 1 foo.printfoo2()

~/foo.py in printfoo2()
      8     if False:
      9         foo = 'foo'
---> 10     print foo

UnboundLocalError: local variable 'foo' referenced before assignment

Telarra
Oct 9, 2012

IMlemon posted:

Would anyone be willing to take a quick look at this small program https://github.com/astanzys/GameOfLife/ and point out anything that looks bad to you? I want to learn some Python but I have never done any programming in dynamically typed languages so this is new territory for me.

I decided to tidy up Game.get_neighbours() for you:
code:
def get_neighbours(self, row, col):
    NEIGHBOURS = [[-1, -1], [ 0, -1], [+1, -1],
                  [-1,  0],           [+1,  0],
                  [-1, +1], [ 0, +1], [+1, +1]]
    return [self.grid[row+x][col+y] for x, y in NEIGHBOURS if self.in_grid(row+x, col+y)]
(Warning, drycoded: test before committing)

Now you can get rid of all those ugly <direction>() functions, and the also ugly Game.add() that was for some reason part of the class and not def'd inside get_neighbours(), the only place it was used.

You seem to be doing just fine learning Python. The code's a bit class-heavy compared to what I would write, but that's not a big deal. Plus, you taught me you can use tuples in function argument definitions. Don't know how I missed that handy little bit of syntax.

Haystack
Jan 23, 2005





QuarkJets posted:

And maybe it's just me, but I think that this looks weird just because I don't normally use _ as a variable name

_ is the standard name for a variable you have to declare, but never use, like in "foo for _ in xrange(9)." Most IDEs are configured to exclude the name from static analysis.


evensevenone posted:

This surprised some folks at work:

....

Yeah, this one gets a lot of people. Basically, when you assign to a variable anywhere within a scope (such as the one a function creates), it overwrites any outer variable, regardless of whether or not the logical flow assigns anything to the variable. It's generally considered a flaw in the language, and python 3 added a keyword that lets you work around it.

Haystack fucked around with this message at 02:50 on Jul 21, 2013

ahmeni
May 1, 2005

It's one continuous form where hardware and software function in perfect unison, creating a new generation of iPhone that's better by any measure.
Grimey Drawer

evensevenone posted:

This surprised some folks at work:

Python code:

foo = 'bar'

def printfoo():
    print foo

def printfoo2():
    if False:
        foo = 'foo'
    print foo

Python code:
In [1]: import foo

In [2]: foo.printfoo()
bar

In [3]: foo.printfoo2()
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-3-981b642201ab> in <module>()
----> 1 foo.printfoo2()

~/foo.py in printfoo2()
      8     if False:
      9         foo = 'foo'
---> 10     print foo

UnboundLocalError: local variable 'foo' referenced before assignment

The local/global scope can be a bit tricky, but from my understanding at compile time foo is being declared global but the interpreter can't determine if foo is locally or globally scoped in the second function so it throws the error rather than being unpredictable.

coaxmetal
Oct 21, 2010

I flamed me own dad
I've always thought that problem occured because the parser sees a local assignment, and so puts it in the local namespace, but it has no way to tell whether or not the assignment is executed (which is good, otherwise it couldn't be statically parsed). The result is slightly confusing though.

evensevenone
May 12, 2001
Glass is a solid.

Ronald Raiden posted:

I've always thought that problem occured because the parser sees a local assignment, and so puts it in the local namespace, but it has no way to tell whether or not the assignment is executed (which is good, otherwise it couldn't be statically parsed). The result is slightly confusing though.

Python code:
def printfoo3():
    print foo
    if False:
        foo = 'foo'
This will also throw an UnboundLocalError, for what its worth.

But yeah, it's all due to compilation.

Dominoes
Sep 20, 2007

I spent some time figuring out how global variables work last week as well. Some things I learned:

-If you assign a value to a variable, ie you use '=' or '+=' on a global variable anywhere in a function, you must declare it as global within that function, even if the assignment never executes, like in evenseven's example.

-If you don't directly assign a value using an equal sign, you may access the global variable. For example, you don't have to declare a dictionary as global if you assign a key/value to it. dict[key] = value works, but dict = {key: value} doesn't, without a global declaration.

-If not assigning a value anywhere in a function, the function can access the variable without error.

-If you assign a value before the global declaration, you get "SyntaxWarning: name ' ' is assigned to before global declaration."


Bottom line: When using global variables in functions, write "global x" at the top of the function, where x is the variable, if anything in the function could assign a value to the variable.

Haystack posted:

Yeah, this one gets a lot of people. Basically, when you assign to a variable anywhere within a scope (such as the one a function creates), it overwrites any outer variable, regardless of whether or not the logical flow assigns anything to the variable. It's generally considered a flaw in the language, and python 3 added a keyword that lets you work around it.
What's the keyword?

Dominoes fucked around with this message at 11:06 on Jul 21, 2013

Adbot
ADBOT LOVES YOU

coaxmetal
Oct 21, 2010

I flamed me own dad

evensevenone posted:

Python code:
def printfoo3():
    print foo
    if False:
        foo = 'foo'
This will also throw an UnboundLocalError, for what its worth.

But yeah, it's all due to compilation.

yeah that checks out, i'm assuming the parsing is what is determining variable scope and that happens before runtime, so the assignment is a different token than print and print, at runtime, is told that foo is a local (or rather, python thinks foo is a local and tries to give that to print). That is assuming i'm right about the parsing. Either way, it's not intuitive or ideal, but its better than the alternative.

  • Locked thread