Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Dominoes: Sep 20, 2007

How do you run unit tests? I get errors about import directories and such, both from PyCharm, and from the terminal.

I nav to the directory, then run in terminal: 'python -m unittest tests.py'. I receive ImportError: cannot import name 'my module', for any imports within my program.

I can fix this by changing my imports in all files from from module import file to import file, but this breaks actually using the module vice testing it. Pycharm's unit test Run feature breaks in the same way. Any ideas?

Using Django's unittests introduces a whole other world of DB errors... Not even going there!

It looks like most projects on Github use third party tools, but I'm trying to do this straight from Python's docs.

# ? May 7, 2016 14:51

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 06:19

GameCube: Nov 21, 2006

Have you marked all your source directories as such in the PyCharm project? If you do, PyCharm will add the source directories to the Python path when running your tests.

It could be a legitimate issue with your package structure and how you import modules from one another. Is this source posted anywhere so we can see how you've arranged things?

Also, by "file" do you mean you're importing some class/object/whatever called "file" from a module, or that you're importing a file from a module? Because if it's the latter, you may need to back up a bit.

# ? May 9, 2016 14:52

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

Loving Africa Chaps posted:

It wasn't and might explain my cool new errors. Got some more reading to do i guess

The method probably looks something like

Python code:

def execute(self, command, args):
    #... probably some important stuff going on up here
    for i, arg in enumerate(args):
        self._bindArgToCommand(i, arg)

Which strikes me as an odd way to handle it in Python because there's great language support for named arguments and argument list unpacking, but hey it's their party.

Anyway, both string and tuple are enumerable, so that for loop will happily consume either, member by member, so it will try to bind every character in a naked string (or code point in Python 3, I guess). The reason you need the (thing,) construct to pass a tuple instead of a string is that that's the special case syntax to create a one-element tuple. An empty tuple would just be empty matching parens. I would be a little surprised if a list didn't work just as well and the syntax there is a little more intuitive: [thing].

# ? May 9, 2016 15:27

Dominoes: Sep 20, 2007

GameCube posted:

Have you marked all your source directories as such in the PyCharm project? If you do, PyCharm will add the source directories to the Python path when running your tests.

It could be a legitimate issue with your package structure and how you import modules from one another. Is this source posted anywhere so we can see how you've arranged things?

Also, by "file" do you mean you're importing some class/object/whatever called "file" from a module, or that you're importing a file from a module? Because if it's the latter, you may need to back up a bit.

I didn't have it marked in PyCharm, but marking it doesn't appear to help. I think I have it arranged in a normal way. Github of an example. By file - let's say I have a normal python file. It imports another, with 'from module import file' or 'from . import file'. This is the line that breaks the unit tests. Changing it to 'import file' fixes it for tests, but breaks it for normal use.

# ? May 9, 2016 16:17

GameCube: Nov 21, 2006

Dominoes posted:

I didn't have it marked in PyCharm, but marking it doesn't appear to help. I think I have it arranged in a normal way. Github of an example. By file - let's say I have a normal python file. It imports another, with 'from module import file' or 'from . import file'. This is the line that breaks the unit tests. Changing it to 'import file' fixes it for tests, but breaks it for normal use.

First, some info on importing:

A "module" is a .py file. A "package" is a folder with an __init__.py file.

Say I want to use your saturn package. I can import the whole package:

pre:

import saturn
x = saturn.today()

Or import today from the package.

pre:

from saturn import today
x = today()

This only works because the saturn package's __init__.py imports today. If it didn't, I'd have to either import the module within the package:

pre:

import saturn.saturn
x = saturn.saturn.today()

Or import today itself from that module:

pre:

from saturn.saturn import today
x = today()

So by 'from module import file' do you mean 'from package import module'? It sounds to me like you're either missing something in your __init__.py, or you need to add something to your Python path (which the PyCharm suggestion might have helped with, if that were the case).

# ? May 9, 2016 17:34

Dominoes: Sep 20, 2007

GameCube posted:

So by 'from module import file' do you mean 'from package import module'? It sounds to me like you're either missing something in your __init__.py, or you need to add something to your Python path (which the PyCharm suggestion might have helped with, if that were the case).

Yep. Thanks for clarifying how this stuff works. Most of the repos I look at on Github rely on a third-party test like nose, and structure their imports like mine.

Dominoes fucked around with this message at 18:01 on May 9, 2016

# ? May 9, 2016 17:55

GameCube: Nov 21, 2006

The top answer here should be what you need, then. The build-in Python unittest module also does the sys.path fuckery, because your unit tests should be written like any other code that might use your package.

# ? May 9, 2016 19:42

Dominoes: Sep 20, 2007

Thanks; you solved it! The solution was, as described in that SO answer, to move tests.py to a separate tests folder. Running the tests from that folder (either by just running the script, or using 'python -m unittest') works; still can't get Pycharm to run them for me though.

Dominoes fucked around with this message at 19:55 on May 9, 2016

# ? May 9, 2016 19:53

Eela6: May 25, 2007; Shredded Hen

Hi. I'm trying to learn NumPy as a replacement for MATLAB. I'm implementing an image processing routine I wrote that detects copy-paste forgeries.

image is an mxnx3 array of unsigned 8-bit integers.
mask is an mxn array of booleans.
separator is an mx8x3 array that's BLUE (i.e, each pixel is [0, 0, 255])

I want to create a masked image that is the original image on the left, the blue separator, and a psuedo-grayscale version on the right, eg:

But I'm still having some trouble with basic array manipulation. Here's what I did in MATLAB. How would I do this in numpy?

code:

RED = 1
GREEN = 2 
BLUE = 3

maskedImage = rgb2gray(image)			% set to grayscale
maskedImage = repmat(maskedImage, 1, 1, 3)	% make 'psuedo-rgb'

% I need help with this:
maskedImage(:, :, RED) =  maskedImage + mask*255	
maskedImage(:, :, BLUE) = maskedImage + mask*-255	
maskedImage(:, :, GREEN) = maskedImage + mask*-255
maskedImage(maskedImage <= 0) = 0
maskedImage(maskedImage >= 255) = 255
% keep in uint8 space. this might not be necessary
% but i don't understand numpy's typing very well yet

% and this:
outputImage = [image, separator, maskedImage]	
%bmat('image', 'separator', 'maskedImage') seems like the right command,	
%except it seems to only work specifically on matrices, not n-dimensional numpy arrays.

Thanks for your time!

Eela6 fucked around with this message at 20:38 on May 10, 2016

# ? May 10, 2016 19:31

Dominoes: Sep 20, 2007

sympy, numpy, and toolz have 3 very different functions all called diff!

# ? May 10, 2016 21:50

QuarkJets: Sep 8, 2008

Eela6 posted:

Hi. I'm trying to learn NumPy as a replacement for MATLAB. I'm implementing an image processing routine I wrote that detects copy-paste forgeries.

image is an mxnx3 array of unsigned 8-bit integers.
mask is an mxn array of booleans.
separator is an mx8x3 array that's BLUE (i.e, each pixel is [0, 0, 255])

I want to create a masked image that is the original image on the left, the blue separator, and a psuedo-grayscale version on the right, eg:

But I'm still having some trouble with basic array manipulation. Here's what I did in MATLAB. How would I do this in numpy?
code:
RED = 1
GREEN = 2 
BLUE = 3

maskedImage = rgb2gray(image)			% set to grayscale
maskedImage = repmat(maskedImage, 1, 1, 3)	% make 'psuedo-rgb'

% I need help with this:
maskedImage(:, :, RED) =  maskedImage + mask*255	
maskedImage(:, :, BLUE) = maskedImage + mask*-255	
maskedImage(:, :, GREEN) = maskedImage + mask*-255
maskedImage(maskedImage <= 0) = 0
maskedImage(maskedImage >= 255) = 255
% keep in uint8 space. this might not be necessary
% but i don't understand numpy's typing very well yet

% and this:
outputImage = [image, separator, maskedImage]	
%bmat('image', 'separator', 'maskedImage') seems like the right command,	
%except it seems to only work specifically on matrices, not n-dimensional numpy arrays.
Thanks for your time!

What problems are you running into, specifically? Can you post your Python code?

First thing I can point out is that Python (and most other languages) starts numbering from 0, rather than 1. So RED, GREEN, and BLUE should be 0, 1, 2 rather than 1, 2, 3

Matlab also tends to treat array shapes oppositely from Python; check the shape of your arrays by printing maskedImage.shape or inspecting it with a debugger. It really depends on what modules you're using to read your data

Python uses square brackets [ ] instead of parens ( ) for array indexing.

Numpy will keep things in uint8 if you're dealing with just uint8 arrays, whereas Matlab sometimes converts things to double unexpectedly. Be careful that you're not accidentally creating an overflow

There are several equivalent functions for rgb2gray (there are many equally valid ways to convert from rgb to grayscale) but if you just want something quick and easy you can just take the average along the rgb axis: maskedImage_gs = maskedImage_rgb.mean(axis=0) or axis=2 depending on how the array was created

I've never used bmat but it looks like it deals primarily with matrices, like you said. Numpy actually distinguishes between matrices and arrays, and you almost definitely want an array. If you're just trying to build a rectangular image of 3 inputs you can use concatenate, vstack, or hstack (all part of numpy).

Python code:

import numpy as np
from matplotlib import pyplot as plt
image = np.zeros((100, 100), np.uint8)  #100x100 array of zeros
separator = np.ones((100, 20), np.uint8)*2  # 100x20 array of twos
maskedImage = np.ones((100, 100), np.uint8)  #100x100 array of ones
# Eh let's set some of the elements to 3
maskedImage[:,0:10] = 3

# Stack the arrays together into one combined array
outputImage = np.hstack((image, separator, maskedImage))

# Show the results
plt.imshow(outputImage, cmap='gray')
plt.show()

Note here that (100, 100) is just a tuple that I'm passing to np.zeros, functions don't normally need double-parens. The np.uint8 specifies that I want the arrays to be uint8, but you can leave that off and you'll get float64 arrays instead.

QuarkJets fucked around with this message at 00:19 on May 11, 2016

# ? May 11, 2016 00:04

QuarkJets: Sep 8, 2008

Here's also a helpful cheat sheet for converting between Matlab and Python syntax:

http://mathesaurus.sourceforge.net/matlab-numpy.html

# ? May 11, 2016 00:06

Eela6: May 25, 2007; Shredded Hen

QuarkJets posted:

What problems are you running into, specifically? Can you post your Python code?

QuarkJets posted:

Here's also a helpful cheat sheet for converting between Matlab and Python syntax:

http://mathesaurus.sourceforge.net/matlab-numpy.html

Thank you for your advice. I am aware of the MATLAB-numPy cheat sheet & looked there first.

Edit: No I didn't. I was aware of ~a~ MATLAB-numPy cheat sheet. Let me do some digging in this one.

I seem to have the hang of 2d arrays in numpy (though I am still making the occasional off-by-one error adjusting to the zero-based indexing).

My problem is that I don't seem to understand how to deal with 3+ dimension arrays.
EG,

code:

import numpy as np
from skimage import io
RED = 0
# i want an image where the top left corner is RED
# and the rest is black; just as a toy case.

redMask = np.zeros((128, 128, 3), 'uint8')
redMask[0:63][0:63][:, RED] = 255
redMask = np.uint8(redMask)
io.imshow(redMask)
# but this does not do what I want it to do.
# ... :(

A snippet of the relevant part of my real code:

code:

import numpy as np
from skimage import color
def create_mask(blocks, img, init):
    # create mask: this is a 2d boolean matrix of the same dimensions as the 
    # original image, set to True in blocks that are connected to other blocks
    # (i.e, considered possibly copy-paste forged) and false elsewhere
    size = np.shape(img)
    rows = size[0]
    cols = size[1]
    mask = np.zeros(rows, cols)
    for block in blocks:
        row = block.row
        col = block.col
        rowEnd = row +init.blockSize-1
        colEnd = col + init.blockSize-1
        mask[row:rowEnd][:, col:colEnd] = True        
    return mask
    
def write_mask(mask, img, init):
    # we create an image image_out which is the original image on the left, a
    # 8-row BLUE (0, 0, 255) barrier, and then a MASKED image on the right.
    # The MASKED image is a grayscale version of the original image, except that
    # and set to RED (255, 0, 0) where the MASK is True 
    # (i.e, blocks considered modified)
    cols = np.shape[1]
    # we want a 'color' grayscale version of the original image:
    imgGray = color.gray2rgb(color.rgb2gray(img))


    """
create separator, maskedImage
we want to add maskedImage*255 to the red channel of imgGray
maskedImage*(-255) to the green channel
maskedImage*(-255) to the blue channel
then cast back to a uint8
    """
# this should work?
    imgOut =  np.hstack((image, separator, maskedImage))
    return imgOut

And the relevant code in MATLAB:

code:

function [imgMasked, imgOut] = write_mask(mask, imgIn)
% use a mask (matrix of same size comprised of natural numbers)
% to 'write over' image: 

%imgMasked is the image in naive grayscale except
% RED   (255, 0, 0) where mask >= 1

%imgOut is the original m x n image (on LEFT)
% a m x 8 separation of BLUE (0, 255, 0)
% and then imageMasked on RIGHT

% create RED mask
mask = repmat(mask, 1, 1, 3);
to_red = (mask > 0);
mask(:, :, 1) = to_red*512;
mask(:, :, 2) = to_red*(-1024);
mask(:, :, 3) = to_red*(-1024);

% every element with a 1 will be > 256        
imgGray = rgb2gray(imgIn);
imgMasked = zeros([size(mask), 3]);
imgMasked(:, :, 1) = imgGray;
imgMasked(:, :, 2) = imgGray;
imgMasked(:, :, 3) = imgGray;
imgMasked = imgMasked + mask;
% every element in imgMasked will have
% blue and green channel < 0
separation = zeros([size(mask, 1), 8, 3]);
separation(:, :, 3) = 255;

imgOut = uint8([imgIn, separation, imgMasked]);
% every element in imgMasked will be grayscale except where mask has nonzeros
end

Thank you again for your time. Sorry if these are silly questions. I am very new to numPy, but trying to get better: I enjoy working in python!

Eela6 fucked around with this message at 03:54 on May 11, 2016

# ? May 11, 2016 03:40

SurgicalOntologist: Jun 17, 2004

Eela6 posted:

code:

import numpy as np
from skimage import io
RED = 0
# i want an image where the top left corner is RED
# and the rest is black; just as a toy case.

redMask = np.zeros((128, 128, 3), 'uint8')
** redMask[0:63][0:63][:, RED] = 255
redMask = np.uint8(redMask)
io.imshow(redMask)
# but this does not do what I want it to do.
# ... :(

The line I marked with ** needs to be

code:

redMask[0:63, 0:63, RED] = 255

# ? May 11, 2016 04:00

QuarkJets: Sep 8, 2008

Eela6 posted:

Thank you again for your time. Sorry if these are silly questions. I am very new to numPy, but trying to get better: I enjoy working in python!

Happy to help. Your questions aren't silly, it's good to get syntax help when learning a new language

I think what SurgicalOntologist posted should be enough to get you where you're going, just a small syntax change fixes your toy code. But I also noticed this:

Python code:

redMask = np.zeros((128, 128, 3), 'uint8')
** redMask[0:63][0:63][:, RED] = 255
redMask = np.uint8(redMask)
io.imshow(redMask)

It looks like you're creating a uint8 array, setting some values in it, and then converting it to uint8. You can skip that last step. Cleaning that up and following along with SurgicalOntologist's change, you can replace the above code chunk with this:

Python code:

redMask = np.zeros((128, 128, 3), np.uint8)
redMask[0:63, 0:63, RED] = 255
io.imshow(redMask)

Does numpy really let you use 'uint8' as the datatype? That's pretty cool, I've never tried passing in a string for that argument.

# ? May 11, 2016 07:28

rvm: May 6, 2013

Dominoes posted:

How do you run unit tests? I get errors about import directories and such, both from PyCharm, and from the terminal.

I nav to the directory, then run in terminal: 'python -m unittest tests.py'. I receive ImportError: cannot import name 'my module', for any imports within my program.

I can fix this by changing my imports in all files from from module import file to import file, but this breaks actually using the module vice testing it. Pycharm's unit test Run feature breaks in the same way. Any ideas?

Using Django's unittests introduces a whole other world of DB errors... Not even going there!

It looks like most projects on Github use third party tools, but I'm trying to do this straight from Python's docs.

Put your tests in a sibling folder of your main project source folder and run `python -m unittest <your_test_dir> discover` from the root folder.

Edit: beaten.

# ? May 11, 2016 07:53

Eela6: May 25, 2007; Shredded Hen

Thank you for your help, everyone! I think I've got it.

code:

import numpy as np
from skimage import io
RED = 0
GREEN = 1
BLUE = 2
ADD_CHANNEL = 8
REMOVE_CHANNEL = 16
testImg = io.imread('test.png')
mask = np.zeros((256, 256, 3), 'uint8')

""" set mask """
mask[0:64, 0:64, RED] = ADD_CHANNEL
mask[0:64, 0:64, GREEN] = REMOVE_CHANNEL
mask[0:64, 0:64, BLUE] = REMOVE_CHANNEL

mask[0:64, 64:128, RED] = REMOVE_CHANNEL
mask[0:64, 64:128, BLUE] = ADD_CHANNEL
mask[0:64, 64:128, GREEN] = REMOVE_CHANNEL

mask[64:128, 0:64, RED] = REMOVE_CHANNEL
mask[64:128, 0:64, BLUE] = REMOVE_CHANNEL
mask[64:128, 0:64, GREEN] = ADD_CHANNEL

mask[64:128, 64:128, RED] = ADD_CHANNEL
mask[64:128, 64:128, BLUE]= ADD_CHANNEL
mask[64:128, 64:128, GREEN] = REMOVE_CHANNEL

mask[32:96, 32:96, RED] = ADD_CHANNEL
mask[32:96, 32:96, GREEN] = ADD_CHANNEL
mask[32:96, 32:96, BLUE] = ADD_CHANNEL

mask[48:80, 48:80, RED] = REMOVE_CHANNEL
mask[48:80, 48:80, BLUE] = REMOVE_CHANNEL
mask[48:80, 48:80, GREEN] = REMOVE_CHANNEL

""" mask over image """
testImg[mask==ADD_CHANNEL] = 255
testImg[mask==REMOVE_CHANNEL]= -255
io.imshow(testImg)

Edit: I was able to fully re-implement my program. Thanks again for the help!

Eela6 fucked around with this message at 20:34 on May 12, 2016

# ? May 11, 2016 17:28

IT BEGINS: Jan 15, 2009; I don't know how to make analogies

Does anyone happen to know any good resources for asynchronous programs in Python?

I'm working in a command line twitter client and I'd like to add some features that exist on the web version - for example, I'd like the be able to load tweets from various timelines and lists and read user input to the terminal as well as render any changes to the state (such as adding new tweets). I currently process things by loading tweets at the start and when a certain key combination is pressed. However, this is slow and not particularly user friendly. I'd love to be able to push tweets into my process but I don't know how to handle doing that AND responding to user input at the same time.

My thought is to have a few processes running that all pipe any 'events' they receive into a queue that the main terminal can read from. However, the tutorials I've found on Queues and async in general for Python have been extremely simplistic and don't take into account this sort of continuous reading. Maybe I'm just missing something?

# ? May 14, 2016 07:35

Stringent: Dec 22, 2004; image text goes here

IT BEGINS posted:

Does anyone happen to know any good resources for asynchronous programs in Python?

I'm working in a command line twitter client and I'd like to add some features that exist on the web version - for example, I'd like the be able to load tweets from various timelines and lists and read user input to the terminal as well as render any changes to the state (such as adding new tweets). I currently process things by loading tweets at the start and when a certain key combination is pressed. However, this is slow and not particularly user friendly. I'd love to be able to push tweets into my process but I don't know how to handle doing that AND responding to user input at the same time.

My thought is to have a few processes running that all pipe any 'events' they receive into a queue that the main terminal can read from. However, the tutorials I've found on Queues and async in general for Python have been extremely simplistic and don't take into account this sort of continuous reading. Maybe I'm just missing something?

Sounds like you're looking for gevent:
http://www.gevent.org

Alternatively there's Celery but that sounds like overkill.

# ? May 14, 2016 13:00

Hughmoris: Apr 21, 2007; Let's go to the abyss!

Any recommended 'Pandas 101' books/tutorials? Doing some basic data manipulation in Excel, curious to see how Pandas matches up against it.

# ? May 15, 2016 05:33

hooah: Feb 6, 2006; WTF?

I'm working on creating a project for the class I TA, and I'm not quite understanding what's going on with my dictionaries. My code reads in some files of EPA fuel test data, creating individual dictionaries where the keys are the manufacturers and the values are a list of [city_mpg, highway_mpg, combined_mpg, num_vehicles] and combines them into one larger dictionary:

Python code:

def update_dict(target, source):
    """
    Updates one dictionary with the contents of the second. Dictionaries
    must be of the format generated by read_file.
    :param target: The dictionary to be updated.
    :param source: The dictionary used to update.
    :return: None
    """
    for key, value in target.items():
        if key in source:
            source_city = source[key][0]
            source_highway = source[key][1]
            source_combined = source[key][2]
            source_count = source[key][3]
            target_city = target[key][0]
            target_highway = target[key][1]
            target_combined = target[key][2]
            target_count = target[key][3]
            target[key][0] = (target_city * target_count + source_city *
                              source_count) / (target_count + source_count)
            target[key][1] = (target_highway * target_count + source_highway
                              * source_count) / (target_count + source_count)
            target[key][2] = (target_combined * target_count + source_combined
                              * source_count) / (target_count + source_count)
            target[key][3] = target_count + source_count

eighties_file = open_file()
nineties_file = open_file()
eighties_dict = read_file(eighties_file)
master_dict = {}
master_dict.update(eighties_dict)
nineties_dict = read_file(nineties_file)
update_dict(master_dict, nineties_dict)

The problem is that when the last line runs, eighties_dict gets changed. I've verified that master_dict and eighties_dict are different objects by both printing their IDs and printing the result of master_dict is eighties_dict. Why is this happening?

Edit: This only happens in PyCharm; I tried using Spyder (which I hate), and it worked fine. Fantastic.

Edit 2: The problem also happens on the command line. Super weird.

hooah fucked around with this message at 14:25 on May 15, 2016

# ? May 15, 2016 14:09

IT BEGINS: Jan 15, 2009; I don't know how to make analogies

Stringent posted:

Sounds like you're looking for gevent:
http://www.gevent.org

Alternatively there's Celery but that sounds like overkill.

Thanks for the help. I ended up going towards the threads route for now, but the docs for gevent were very useful for figuring out the right architecture.

# ? May 15, 2016 14:27

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

hooah posted:

The problem is that when the last line runs, eighties_dict gets changed. I've verified that master_dict and eighties_dict are different objects by both printing their IDs and printing the result of master_dict is eighties_dict. Why is this happening?

Edit: This only happens in PyCharm; I tried using Spyder (which I hate), and it worked fine. Fantastic.

Edit 2: The problem also happens on the command line. Super weird.

When you run master_dict.update(eighties) it doesn't copy the values as new objects, it adds a reference to the original object

Python code:

>>> master.update(eighties)
>>> master
{'Toyota': [60, 70, 80], 'Ford': [10, 20, 30]}
>>> eighties
{'Toyota': [60, 70, 80], 'Ford': [10, 20, 30]}
>>> eighties is master
False
>>> eighties["Ford"] is master["Ford"]
True

So your actual dictionaries are different objects, but they share the same list objects, which is what you're poking at.

As for why it works in one thing and not another, that would come down to the environment right? Python version and any libraries you're using. Maybe one of them is making copies somewhere along the line

# ? May 15, 2016 16:55

SurgicalOntologist: Jun 17, 2004

Instead of passing objects to functions in order to be mutated, have them return a new object.

# ? May 15, 2016 17:09

hooah: Feb 6, 2006; WTF?

Ah, I wasn't aware that update() is a shallow copy; thanks.

# ? May 15, 2016 19:10

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

IT BEGINS posted:

Does anyone happen to know any good resources for asynchronous programs in Python?

I'm working in a command line twitter client and I'd like to add some features that exist on the web version - for example, I'd like the be able to load tweets from various timelines and lists and read user input to the terminal as well as render any changes to the state (such as adding new tweets). I currently process things by loading tweets at the start and when a certain key combination is pressed. However, this is slow and not particularly user friendly. I'd love to be able to push tweets into my process but I don't know how to handle doing that AND responding to user input at the same time.

My thought is to have a few processes running that all pipe any 'events' they receive into a queue that the main terminal can read from. However, the tutorials I've found on Queues and async in general for Python have been extremely simplistic and don't take into account this sort of continuous reading. Maybe I'm just missing something?

Python 3 has asyncio built-in.

# ? May 15, 2016 22:19

the talent deficit: Dec 20, 2003; self-deprecation is a very british trait, and problems can arise when the british attempt to do so with a foreign culture

i know python hates functional programming, but is there really no filtermap in the standard library?

# ? May 17, 2016 01:43

tef: May 30, 2004; -> some l-system crap ->

the talent deficit posted:

i know python hates functional programming, but is there really no filtermap in the standard library?

x = [f(y) for y in c if y < 2] ?

# ? May 17, 2016 01:53

Tigren: Oct 3, 2003

the talent deficit posted:

i know python hates functional programming, but is there really no filtermap in the standard library?

filter() map() reduce()???

# ? May 17, 2016 02:04

OnceIWasAnOstrich: Jul 22, 2006

tef posted:

x = [f(y) for y in c if y < 2] ?

And you can even do this lazily with a generator comprehension.

Tigren posted:

filter() map() reduce()???

And these with the itertools versions!

Not to mention getting into the toolz stuff.

# ? May 17, 2016 17:40

Jose Cuervo: Aug 25, 2004

I am now trying to package my GUI as an .exe using the py2exe package. However I have encountered the following error:

Python code:

C:\Users\ME\Desktop\my_gui\dist>my_gui.exe
[Error 3] The system cannot find the path specified: 'C:\\Users\\ME\\Deskt
op\\my_gui\\dist\\library.zip\\dateutil\\zoneinfo/*.*'
Traceback (most recent call last):
  File "my_gui.py", line 29, in <module>
  File "run_sim_gui.pyc", line 3, in <module>
  File "tfm.pyc", line 2, in <module>
  File "init_cond.pyc", line 3, in <module>
  File "policy.pyc", line 1, in <module>
  File "pandas\__init__.pyc", line 7, in <module>
  File "pandas\tslib.pyc", line 12, in <module>
  File "pandas\tslib.pyc", line 10, in __load
  File "pandas\tslib.pyx", line 44, in init pandas.tslib (pandas\tslib.c:79879)
  File "dateutil\zoneinfo\__init__.pyc", line 35, in <module>
  File "dateutil\zoneinfo\__init__.pyc", line 28, in getzoneinfofile
WindowsError: [Error 3] The system cannot find the path specified: 'C:\\Users\\ME\\Desktop\\tfsp_gui\\dist\\library.zip\\dateutil\\zoneinfo/*.*'

The error message seems to indicate that there is NO folder located at C:\Users\ME\Desktop\tfsp_gui\dist\library.zip\dateutil\zoneinfo. However, I the folder does exist and when I navigate to the folder located at C:\Users\ME\Desktop\tfsp_gui\dist\library.zip\dateutil\zoneinfo it contains a single file __init__.pyc (i.e. is non-empty). Anyone have thoughts on what this error message actually means and how I can solve this error?

# ? May 17, 2016 19:39

QuarkJets: Sep 8, 2008

I don't know the answer to your quesiton, but my hunch is that the zip file is the problem; try extracting library.zip and pointing to the contents rather than stuff inside of the zip file

e: unless you're supposed to create a .zip, or maybe the module did that? This exe creation business is so weird

# ? May 17, 2016 19:43

Jose Cuervo: Aug 25, 2004

QuarkJets posted:

I don't know the answer to your quesiton, but my hunch is that the zip file is the problem; try extracting library.zip and pointing to the contents rather than stuff inside of the zip file

e: unless you're supposed to create a .zip, or maybe the module did that? This exe creation business is so weird

The py2exe package creates the library.zip folder as part of the dist folder (where to my understanding the dist folder contains all the files necessary to run the exe).

# ? May 17, 2016 20:42

GameCube: Nov 21, 2006

The folder on the desktop changes name in your output from my_gui to tfsp_gui. Is that in the original output or did you change that when copying/pasting ?

# ? May 17, 2016 22:40

Jose Cuervo: Aug 25, 2004

GameCube posted:

The folder on the desktop changes name in your output from my_gui to tfsp_gui. Is that in the original output or did you change that when copying/pasting ?

No, that's my fault when copy pasting, it should read:

Python code:

C:\Users\ME\Desktop\my_gui\dist>my_gui.exe
[Error 3] The system cannot find the path specified: 'C:\\Users\\ME\\Deskt
op\\my_gui\\dist\\library.zip\\dateutil\\zoneinfo/*.*'
Traceback (most recent call last):
  File "my_gui.py", line 29, in <module>
  File "run_sim_gui.pyc", line 3, in <module>
  File "tfm.pyc", line 2, in <module>
  File "init_cond.pyc", line 3, in <module>
  File "policy.pyc", line 1, in <module>
  File "pandas\__init__.pyc", line 7, in <module>
  File "pandas\tslib.pyc", line 12, in <module>
  File "pandas\tslib.pyc", line 10, in __load
  File "pandas\tslib.pyx", line 44, in init pandas.tslib (pandas\tslib.c:79879)
  File "dateutil\zoneinfo\__init__.pyc", line 35, in <module>
  File "dateutil\zoneinfo\__init__.pyc", line 28, in getzoneinfofile
WindowsError: [Error 3] The system cannot find the path specified: 'C:\\Users\\ME\\Desktop\\my_gui\\dist\\library.zip\\dateutil\\zoneinfo/*.*'

# ? May 18, 2016 00:30

Reformed Pissboy: Nov 6, 2003

py2exe has been temperamental for me when packaging anything but really simple libraries. It's inelegant, but you can pass zipfile=None to your setup() call and it'll bundle everything into your exe itself instead of a library.zip archive.

# ? May 18, 2016 19:03

Jose Cuervo: Aug 25, 2004

Reformed Pissboy posted:

py2exe has been temperamental for me when packaging anything but really simple libraries. It's inelegant, but you can pass zipfile=None to your setup() call and it'll bundle everything into your exe itself instead of a library.zip archive.

I found this suggestion in a StackOverflow post as well and tried it, but I still get the same error message. I guess I will try using cx_freeze instead as I cannot find anything online that points me to how to solve this issue.

EDIT:
I have managed to get cx_freeze to work, but I have another non-related general question. On this page, there is the line

Python code:

cxfreeze hello.py --target-dir dist

Where am I meant to type this line in? In Idle, in the cmd window?

Jose Cuervo fucked around with this message at 21:57 on May 18, 2016

# ? May 18, 2016 19:25

IT BEGINS: Jan 15, 2009; I don't know how to make analogies

Thermopyle posted:

Python 3 has asyncio built-in.

Thanks for the link. I've been spending the last couple of days understanding it, helped in particular by this code.

Is there a way to somewhat easily convert regular non-asyncio code to asyncio? Posts like this one lead me to believe there's not much I can do other than convert most everything in my codebase into a coroutine if I want to go in that direction.

Maybe I'm tackling the wrong problem overall? I've started looking into this because I have a program that redraws the screen a bunch and adding a thread to control some extra stuff like making push notifications to my main thread has caused my screen rendering to drop in speed significantly (to the point where I can visually identify the re-render). Should I just be using processes to handle this problem?

# ? May 19, 2016 07:41

Dominoes: Sep 20, 2007

Why does Pycharm inspect spelling by default? Doesn't make sense to me!

# ? May 19, 2016 17:20

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 06:19

QuarkJets: Sep 8, 2008

Yeah I hate that, I always disable spellchecking in both code and comments (because sometimes my comments contain tiny bits of code mid-sentence!)

# ? May 19, 2016 18:57

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »