Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Houston Rockets: Apr 15, 2006

Why not use requests or urllib2?

Also the docs state that:

quote:

The urlopen function from the previous version, Python 2.6 and earlier, of the module urllib has been discontinued as urlopen can return the file-object as the previous

So if you were on <= Python 2.6, that could account for the difference.

# ? Mar 23, 2013 00:14

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 23:06

QuarkJets: Sep 8, 2008

Dominoes posted:

Thanks for the GUI advice dudes - I'm going to use QT.

IDLE is the only program I know of that runs uncompiled python scripts, other than a command line. What do you recommend instead? I'll try PyNPP.

Spyder 2 does that also, I think

# ? Mar 23, 2013 01:29

spankweasel: Jan 4, 2006

Mad Pino Rage posted:

I made a post in here about two weeks ago asking about a cryptography assignment that I was having some major problems with. Now I've come to an assignment that's due in four hours, and I don't know what I'm doing. I don't even know how to go about it. AT ALL. I think I need to buy some beer and read the provided online book all weekend until I figure it out. Maybe I can submit it for partial credit before Monday.

http://www.openbookproject.net/thinkcs/python/english2e/index.html

Um. What's the assignment?

# ? Mar 23, 2013 02:30

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Houston Rockets posted:

Why not use requests

Just stop what you're doing and use requests.

# ? Mar 23, 2013 02:58

DoggesAndCattes: Aug 2, 2007

I was doing pretty good as in I was writing something. The assignment is to write a main function to call functions for A) list a deck of cards in 52 rows and 3 columns: a number given to each card(0-51), the card itself, and show the location of the card(deck player's hand, computer's hand) B)clear the deck so all cards start in the deck C)assign five cards to each the player and computer D) show the cards in the player's hand and the computer's hand

I was able to write something that showed the list, and then I got lost in writing a way to assign cards to the player.

code:

""" cardGame.py
    basic card game framework
    keeps track of card locations for as many hands as needed
"""
from random import *
import random
random.seed()

NUMCARDS = 52
DECK = 0
PLAYER = 1
COMP = 2

#cardLoc = [0] * NUMCARDS was code given and creates an array of [0,0,0...]
cardLoc = [i for i in range(NUMCARDS)]
suitName = ("hearts", "diamonds", "spades", "clubs")
rankName = ("Ace", "Two", "Three", "Four", "Five", "Six", "Seven",
            "Eight", "Nine", "Ten", "Jack", "Queen", "King")
playerName = ("deck", "player", "computer")

def clearDeck():
    print None

def assignCard():
    populateRanks = 0
    populateSuits = 0
    cardPosition = 0
    while populateSuits < 4:
        while populateRanks < 13:
            cardLoc[cardPosition] = rankName[populateRanks] + suitName[populateSuits]
            cardPosition += 1
            populateRanks += 1
        populateRanks = 0
        populateSuits += 1
    cards = [random.randint(0,51)]
    print cards
    return cards

def showDeck():
    populateRanks = 0
    populateSuits = 0
    cardPosition = 0
    print "Number \t Card \t\t Location"
    while populateSuits < 4:
        while populateRanks < 13:
            print str(cardLoc[cardPosition]), '\t' , rankName[populateRanks], 'of',suitName[populateSuits],'\t  ',playerName[0]
            cardPosition += 1
            populateRanks += 1
        populateRanks = 0
        populateSuits += 1


def showHand():
    print cards

def main():
  #clearDeck()

  for i in range(5):
    assignCard()
    #assignCard(PLAYER)
    #assignCard(COMP)

  showDeck()
  #showHand()
  #showHand(PLAYER)
  #showHand(COMP)

main()
print cards

# ? Mar 23, 2013 03:16

spankweasel: Jan 4, 2006

Assigning a card from the deck to a player is akin to moving one element from a list to another.

# non random version
deck = [2H, 3H, 4H, 5H]
player.draw_card() # pseudo function...
deck = [3H, 4H, 5H]
player = [2H]

Right?

So, you just need to manipulate the lists.

You could use random.choice to pick one of the cards. Then use deck.remove(choice) followed by player.append(choice).

# ? Mar 23, 2013 04:06

Lurchington: Jan 2, 2003; Forums Dragoon

I posted this a LONG time ago when the question of GUI toolkits kept up, and honestly, I'm more sure of myself than ever:

Lurchington posted:

honestly, projects like http://sickbeard.com/ and http://sabnzbd.org/ show how good/usable GUI apps can be using javascript and a thin server like cherrypy. That'd be my recommendation for a new GUI project anyway. Especially cross platform.

here's a good follow-up post that had some stuff about designing UI's in JS:

uncleTomOfFinland posted:

I am hardly the best qualified person for this but first I'd watch this lecture before starting with JS to avoid the most obvious shittiness of it:
http://www.youtube.com/watch?v=hQVTIJBZook
.. and after that get the book as well.

Good reference tool
Good online book for getting to know the new fancy HTML5 features
Better than W3Schools for HTML/CSS tutorials

# ? Mar 23, 2013 04:07

Dominoes: Sep 20, 2007

accipter posted:

Why don't you just surround the urlopen in a try/catch statement on urllib.error.HTTPError?

Thank you - I got a working solution using this. More elegant than checking the file size. At first I was trying to enter the type of exception after 'except', but couldn't figure it out, so I tried leaving that parameter blank, and it worked. Here's an excerpt from the working code:

code:

try:
            urlopen(url)
            print ("Version: " + str(i))
            return i
        except:
            i -= 1
            if i % 100 == 0:
                i -= 88

Houston Rockets posted:

Why not use requests or urllib2?

Also the docs state that:

So if you were on <= Python 2.6, that could account for the difference.

I was using Python 2.7. My understanding from reading some Python docs is that 'urllib' in Python 3 is urllib 2 from python 2. I ended up downloading something called urllib 3 in the process; not sure what it does. The syntax in Python 3 is a bit different, as reflected in the two code snippets I posted; Python 3 uses 'urllib.request.urlopen' instead of 'urllib.urlopen'.

BeefofAges posted:

Just stop what you're doing and use requests.

I'll look into that as an alternate solution.

It's interesting how I don't need to (am no longer allowed to) convert the iterator 'i' to a str in Python 3, when it was required in Python 2.

Dominoes fucked around with this message at 05:07 on Mar 23, 2013

# ? Mar 23, 2013 04:51

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Dominoes posted:

code:

try:
            urlopen(url)
            print ("Version: " + str(i))
            return i
        except:
            i -= 1
            if i % 100 == 0:
                i -= 88

Don't do this. This will catch every exception.

You want 'except urllib.error.HTTPError as e:'

# ? Mar 23, 2013 05:58

evilentity: Jun 25, 2010

Mad Pino Rage posted:

I was doing pretty good as in I was writing something. The assignment is to write a main function to call functions for A) list a deck of cards in 52 rows and 3 columns: a number given to each card(0-51), the card itself, and show the location of the card(deck player's hand, computer's hand) B)clear the deck so all cards start in the deck C)assign five cards to each the player and computer D) show the cards in the player's hand and the computer's hand

I was able to write something that showed the list, and then I got lost in writing a way to assign cards to the player.

I was bored so I did this for you. Try to use this as guidelines for your own work and dont just copy verbatim. At least try to understand why and how it does things. Ive commented things that I thought might be hard. There is no error checking so you could add that.

Python code:

import random

DECK_ID = 0
PLAYER_ID = 1
COMP_ID = 2

SUITS = ("hearts", "diamonds", "spades", "clubs")
RANKS = ("Ace", "Two", "Three", "Four", "Five", "Six", "Seven",
            "Eight", "Nine", "Ten", "Jack", "Queen", "King")

def create_deck():
    deck = []
    for suit in SUITS:
        for rank in RANKS:
            deck.append([rank, suit, DECK_ID])
    return deck

def clear_deck(deck):
    # change id in card to deck
    for card in deck:
        card[2] = DECK_ID
   
def print_deck(deck):
    # print simple header
    print ' ID  CARD               LOCATION'
    for i, card in enumerate(deck):
        # create string with cards name
        card_name = card[0] + ' of ' + card[1] 
        # create columns, :>3 pads string to len 3
        print '{id:>3}  {card:<17}  {deck}'.format(id=i, card=card_name, deck=card[2])
    # footer
    print ' ID  CARD               LOCATION'

def new_hand(deck, id):
    # create list with cards in deck that are not in other hands
    available_cards = [x for x in deck if x[2] == 0]
    # get 5 random cards from them
    hand = random.sample(available_cards, 5)
    # cards int hand are references for cards in deck
    # so changing them affects deck as well
    for card in hand:
        card[2] = id
    return hand

def print_hand(hand):
    for card in hand:
        print card
    
def main():
    deck = create_deck()
    player_hand = new_hand(deck, PLAYER_ID)
    print_hand(player_hand)
    comp_hand = new_hand(deck, COMP_ID)
    print_hand(comp_hand)
    
    print_deck(deck)
    clear_deck(deck)
    print_deck(deck)
        
if __name__ == '__main__':
    main()

Feel free to improve this.

# ? Mar 23, 2013 12:41

Dominoes: Sep 20, 2007

BeefofAges posted:

Don't do this. This will catch every exception.

You want 'except urllib.error.HTTPError as e:'

Thanks. Was trying to figure out how to do this, but it wouldn't take the error type. I think I was missing the 'as e'. It works now, with

code:

from urllib.error import HTTPError

and

code:

except HTTPError as e:

# ? Mar 23, 2013 16:15

Stabby McDamage: Dec 11, 2005; Doctor Rope

What's the preferred way to serialize tuples of simple scalars (strings, ints, floats, and bools)? Pickle has all those security concerns because it can serialize whole objects with methods, but I don't want to waste time building a boilerplate serializer.

# ? Mar 24, 2013 17:04

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

Stabby McDamage posted:

What's the preferred way to serialize tuples of simple scalars (strings, ints, floats, and bools)? Pickle has all those security concerns because it can serialize whole objects with methods, but I don't want to waste time building a boilerplate serializer.

Protocol Buffers or Thrift if you want a schema, JSON if you don't

# ? Mar 24, 2013 17:09

Stabby McDamage: Dec 11, 2005; Doctor Rope

TOO SCSI FOR MY CAT posted:

Protocol Buffers or Thrift if you want a schema, JSON if you don't

Of course, JSON, duhhhhhh. Thanks.

# ? Mar 24, 2013 17:11

Haystack: Jan 23, 2005

There's also msgpack if performance is a factor.

# ? Mar 24, 2013 18:24

Popper: Nov 15, 2006

Mad Pino Rage posted:

I was doing pretty good as in I was writing something. The assignment is to write a main function to call functions for A) list a deck of cards in 52 rows and 3 columns: a number given to each card(0-51), the card itself, and show the location of the card(deck player's hand, computer's hand) B)clear the deck so all cards start in the deck C)assign five cards to each the player and computer D) show the cards in the player's hand and the computer's hand

I was able to write something that showed the list, and then I got lost in writing a way to assign cards to the player.

This is basically how people are thought classes.
Think like this:

Python code:


class Deck(object):
	
	def __init__(self):
		self.suits = #suit names
		self.ranks = #rank names
		self.cards = []

	def build(self):
		# Cards have three attributes, suit, rank, and the player who owns them. Here they all belong to "Dealer"
		[[self.cards.append(Card(suit, rank, "Dealer")) for rank in self.ranks] for suit in self.suits]

etc...

Card should be a class too.

There are faster ways to do this with dicts and namedtuples but get a class implementation working first.

# ? Mar 25, 2013 00:23

QuarkJets: Sep 8, 2008

Popper posted:

This is basically how people are thought classes.
Think like this:

etc...

Card should be a class too.

There are faster ways to do this with dicts and namedtuples but get a class implementation working first.

The assignment makes it sound like he hasn't been told about classes yet (specifically, it says to use functions), so he shouldn't use a class (even if it would be objectively better and was my first thought, too).

# ? Mar 25, 2013 00:45

Popper: Nov 15, 2006

QuarkJets posted:

The assignment makes it sound like he hasn't been told about classes yet (specifically, it says to use functions), so he shouldn't use a class (even if it would be objectively better and was my first thought, too).

Yeah you're probably right, I forget how ludicrous college is for this stuff.

# ? Mar 25, 2013 00:58

Qwertyiop25: Jan 19, 2011; D is for Dank
It's Hammerin' Hank
Green in his name
And Green in his bank.

I'm working on assignment right now that involves creating a canvas and radio buttons to adjust the background color of the canvas. The code I have so far is:

code:

class Text(Frame):
    def __init__(self):
        Frame.__init__(self)
        self.master.title("Controlling Text")
        self.grid()
        
        self.canvas = Canvas(self, height=100, width=300)
        self.canvas.grid(row = 0, column = 0)

        frame = Frame(self)
        frame.grid(row = 1, column = 0)

        Red = Radiobutton(frame, text = "Red", command = self.redCanvas)
        Yellow = Radiobutton(frame, text = "Yellow", command = self.yellowCanvas)
        White = Radiobutton(frame, text = "White", command = self.whiteCanvas)
        Gray = Radiobutton(frame, text = "Gray", command = self.grayCanvas)
        Green = Radiobutton(frame, text = "Green", command = self.greenCanvas)

        
        Red.grid(row = 0, column = 0)
        Yellow.grid(row = 0, column = 1)
        White.grid(row = 0, column = 2)
        Gray.grid(row = 0, column = 3)
        Green.grid(row = 0, column = 4)

        def redCanvas(self):
            self.canvas.configure(background = "Red")

        def whiteCanvas(self):
            self.canvas.configure(background = "White")

        def yellowCanvas(self):
           self.canvas.configure(background = "Yellow")

        def grayCanvas(self):
            self.canvas.configure(background = "Gray")

        def greenCanvas(self):
            self.canvas.configure(background = "Green")


def main():
    Text().mainloop()

main()

This gives me a "text instance has no attribute 'redCanvas' error. I'm under the impression that canvas.configure would allow me to adjust the attributes of the canvas. Do I need to update the canvas as well in my "color"Canvas methods? Also what do I need to change to align my radio buttons above the canvas instead of below it?

# ? Mar 25, 2013 02:19

Modern Pragmatist: Aug 20, 2008

Qwertyiop25 posted:

I'm working on assignment right now that involves creating a canvas and radio buttons to adjust the background color of the canvas. The code I have so far is:

code:

class Text(Frame):
    def __init__(self):
        Frame.__init__(self)
        self.master.title("Controlling Text")
        self.grid()
        
        self.canvas = Canvas(self, height=100, width=300)
        self.canvas.grid(row = 0, column = 0)

        frame = Frame(self)
        frame.grid(row = 1, column = 0)

        Red = Radiobutton(frame, text = "Red", command = self.redCanvas)
        Yellow = Radiobutton(frame, text = "Yellow", command = self.yellowCanvas)
        White = Radiobutton(frame, text = "White", command = self.whiteCanvas)
        Gray = Radiobutton(frame, text = "Gray", command = self.grayCanvas)
        Green = Radiobutton(frame, text = "Green", command = self.greenCanvas)

        
        Red.grid(row = 0, column = 0)
        Yellow.grid(row = 0, column = 1)
        White.grid(row = 0, column = 2)
        Gray.grid(row = 0, column = 3)
        Green.grid(row = 0, column = 4)

        def redCanvas(self):
            self.canvas.configure(background = "Red")

        def whiteCanvas(self):
            self.canvas.configure(background = "White")

        def yellowCanvas(self):
           self.canvas.configure(background = "Yellow")

        def grayCanvas(self):
            self.canvas.configure(background = "Gray")

        def greenCanvas(self):
            self.canvas.configure(background = "Green")


def main():
    Text().mainloop()

main()

The reason that you are getting the attribute error is because your methods are indented one level too far and are thus considered nested subfunctions of your __init__ method.

Modern Pragmatist fucked around with this message at 03:18 on Mar 25, 2013

# ? Mar 25, 2013 03:13

Qwertyiop25: Jan 19, 2011; D is for Dank
It's Hammerin' Hank
Green in his name
And Green in his bank.

Modern Pragmatist posted:

The reason that you are getting the attribute error is because your methods are indented one level too far and are thus considered nested subfunctions of your __init__ method.

Bah, thank you. I knew it had to be something stupid like that.

# ? Mar 25, 2013 03:46

QuarkJets: Sep 8, 2008

I have something like this:

code:

#myfile.py
class file(dict):
    #some file attributes
    #and for fun let's say that the first 10MB of the file is stored in memory, 
    #so this becomes relatively time-consuming with multiple files

##

#mydir.py
import myfile
class dir(dict):
    #some dir attributes
    #Also contains a bunch of file objects such that self[filename] = file()

##

#myhost.py
import mydir
class host(dict):
    #some dir attributes
    #Also contains a bunch of dir objects such that self[dirname] = dir()

#example usage:
#x = myhost("127.0.0.1") 
#y = x.get("127.0.0.1")
#if x is not None:
#	z = y.get("/home/someuser/somedir/")
#	if z is not None:
#		fi = z.get("somefile.txt")
#		#do something

These are three separate files; myhost is a directory of mydir objects, mydir is a directory of myfile objects. This isn't the actual project, it's just an example analogous to some project that I'm working on.

So here's my question: are there any dangers to making each class threaded? I'm new to threading and haven't experienced the difficulties of creating a threaded program before. So for instance, class host creates a list of dir() objects, and class dir creates a list of file() objects, so I could theoretically thread each of these and experience a speed increase so long as I'm careful about waiting for for the dictionary-filling and class-creating operations to complete before declaring the thread complete (ie use join() to make sure that everything is filled before accessing any of the dictionaries?)

But since this is a lot of file I/O, will this not benefit as much from threading?

# ? Mar 25, 2013 23:19

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

You seem to have a fundamental misunderstanding of threading. I'd lay it to rest until you pick up more of the basics of CS. Threading is a fairly complex subject in and of itself, so it's something to tackle after you understand more of Python and more about memory and processes and everything.

# ? Mar 26, 2013 00:11

FoiledAgain: May 6, 2007

I have an object that I want "globally" available, meaning that I want to be able to write PBase.features in any arbitrary class method and get the list of features for the PBase object. So I set up a seperate module, PBase.py, and imported it into my main.py module. Before I go any further, if this was a bad idea, please let me know. A bit of googling around (actually reading a post on stackoverflow) suggested this was the best way to make an object "global" in the sense that I mean. But I'm not an advanced programmer.

Once I did this, I ran into a problem that PBase needs to know about another class in main.py, called Segment, e.g. there are lines that say things like self.segments.append(Segment()). A quick fix would be to copy the Segment class into PBase.py, but the Segment class makes reference to four other objects defined in main.py, which make reference to other classes there, which eventually renders moot the idea of importing PBase in the first place.

What kind of solution am I looking for here? I don't completely grasp how namespaces work (in general I mean, not just how how it works in Python) so there might be something about how that works which would clear up the mess. Or is this kind of situation that would call for a Singleton?

# ? Mar 26, 2013 01:02

Haystack: Jan 23, 2005

I usually consider it something of a bad habit to import instantiated objects into other parts of code. You see, objects very often are mutated, or have side effects. An imported object is essentially a singleton shared by everything that imports it, whether you want it to be or not. Down the road, that makes the code that uses imported objects very tricky to unit test, or to use in multiple processes.

Ideally, the things you import should be stateless: classes and functions, constants, etc. If a function or method needs an object, that object should be passed in to it as (or in) an argument. You may need to restructure your application a bit to do this, but I can assure you that it makes things easier in the long run. For starters, it will help prevent those circular imports that have been bugging you.

I recommend not putting anything stateful (ie, any of your data) in the global scope. State belongs in the if __name__ = '__main__': block you should have at the bottom your your main.py file.

As an aside, I do not think you mean what you think you mean when you say "class method." In python, you are almost always calling a method from an object, and a class method is a fairly advanced tool largely used for API tomfoolery.

# ? Mar 26, 2013 01:59

Emacs Headroom: Aug 2, 2003

FoiledAgain posted:

So I set up a seperate module, PBase.py, and imported it into my main.py module. Before I go any further, if this was a bad idea, please let me know.

This seems reasonable enough to me

quote:

Once I did this, I ran into a problem that PBase needs to know about another class in main.py, called Segment, e.g. there are lines that say things like self.segments.append(Segment()). A quick fix would be to copy the Segment class into PBase.py, but the Segment class makes reference to four other objects defined in main.py, which make reference to other classes there, which eventually renders moot the idea of importing PBase in the first place.

What kind of solution am I looking for here?

I would think about splitting off all the classes into their own files, maybe inside of a package. For instance you could have

code:

LanguageFuckery/
    init.py
    Segment.py
    Nobule.py
    GlottalFricative.py
    PBase/
        init.py
        Morphemes.py
        Phonemes.py
        Monotremes.py

Then you could put all this in your path, and in your main you would be importing stuff like

Python code:

import LanguageFuckery.Segment as Segment
from LanguageFuckery import PBase

my_monotremes = PBase.Monotremes.global_monotreme_list()

or whatever.

# ? Mar 26, 2013 02:02

FoiledAgain: May 6, 2007

Haystack posted:

Ideally, the things you import should be stateless: classes and functions, constants, etc. If a function or method needs an object, that object should be passed in to it as (or in) an argument. You may need to restructure your application a bit to do this, but I can assure you that it makes things easier in the long run. For starters, it will help prevent those circular imports that have been bugging you.

Thanks for the suggestions. This is a long term project (part of a dissertation) so I definitely appreciate tips like this.

Haystack posted:

As an aside, I do not think you mean what you think you mean when you say "class method." In python, you are almost always calling a method from an object, and a class method is a fairly advanced tool largely used for API tomfoolery.

I actually am aware of the difference, but you're right that was a little sloppy. When I said "write it in an arbitrary class method" I meant "write it inside of any method inside of any class".

Emacs Headroom posted:

I would think about splitting off all the classes into their own files, maybe inside of a package.

I thought about this, but put it off because it seemed like too much work. But it would seem I've reached the point where more work now = less work later so I think that I will give this a try. Thanks! By the way, your variable names made me giggle. (edit: file names, not variable names.)

FoiledAgain fucked around with this message at 02:50 on Mar 26, 2013

# ? Mar 26, 2013 02:47

Omnomnomnivore: Nov 14, 2010; I'm swiftly moving toward a solution which pleases nobody! YEAGGH!

Howdy Python goons. Is there a way to take a slice of a numpy array along an arbitrary axis? I'm looking for something like numpy.take that takes a slice instead of a list of integers, i.e.:

code:

In [1]: a = numpy.random.rand(3, 4, 5)

In [2]: take_slice(a, slice(1, -1), axis=2).shape
(3, 4, 3)

In [3]: take_slice(a, slice(1, -1), axis=1).shape
(3, 2, 5)

This would be useful if the axis is only known at runtime or you want to iterate over all of them. As far as I can tell nothing like that exists and you have to either construct a sequence of slices yourself or do something with swapaxis or rollaxis.

e: vvv Guess I wasn't clear, the key word here is "arbritary". Right now I'm doing:

code:

def take_slice(a, slc, axis):
	key = [slc if i == axis else slice(None) for i in xrange(a.ndim)]
	return a[key]

'axis' is a parameter, I don't know it ahead of time, so I can't just stick the right number of colons in. I'm just surprised this doesn't already exist.

Omnomnomnivore fucked around with this message at 05:33 on Mar 26, 2013

# ? Mar 26, 2013 05:09

Emacs Headroom: Aug 2, 2003

code:

a = np.random.randn(1000)
a = np.reshape(a, [10, 10, 10])
slice1 = a[3:5, :, :]
slice2 = a[:, 4:8, :]
slice3 = a[:, :, 2:]

# ? Mar 26, 2013 05:25

QuarkJets: Sep 8, 2008

Suspicious Dish posted:

You seem to have a fundamental misunderstanding of threading. I'd lay it to rest until you pick up more of the basics of CS. Threading is a fairly complex subject in and of itself, so it's something to tackle after you understand more of Python and more about memory and processes and everything.

I'm self-taught and have been coding for nearly a decade now (scientific coding, a means to an end), but I don't know anything about threading aside from whatever I've read on the Internet. I'm well-versed in dealing with memory, but I haven't really learned any actual computer science

# ? Mar 26, 2013 12:19

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

So, you can't make objects threaded. Threads are like additional programs that run in the same memory space (in Linux, that's literally the only difference from forked processes — they get their own PID and everything)

The issue with this is that if one thread reads memory and then another thread writes memory, you end up with inconsistency issues between the two processes. Since Python's objects are refcounted, this can be disasterous: one object's refcount is at 1, Thread 1 reads it, Thread 2 increments it, and then Thread 1 decrements it to 0 and destroys the object, leaving Thread 2 with some freed memory. Boom.

To account for this, Python introduced something called the Global Interpreter Lock, or GIL, and a thread must take it when it wants to deal with Python objects. All Python threads wait on this lock until it becomes available, and the performance of the GIL is quite poor when combined with how kernel scheduling works.

Note that the GIL is only taken when something wants to interact with Python, so a thread can be waiting on I/O (read, poll), and another thread can come in and do its stuff. Some libraries like NumPy also release the GIL when some kinds of calculations go on, since they've been extremely careful to make all their special calculation objects thread-safe and such.

So, your directory system, it depends on what method you want to thread. If you want to thread the filling of each host object, perhaps through readdir (through os.walk or os.listwhatever), then each thread is going to be hanging in I/O, waiting to grab the lock, and it's unlikely you'll see much of a speedup, but try it out and profile. Computers are so complex that I can make a guess as to some performance thing based on everything I know, but I can't be 100% sure at all, and have been proven wrong before.

Note that you might have bugs when you start to thread! That's OK. Note that there still can be race conditions and thread bugs in your Python code — the GIL doesn't prevent those.

Some people use what's called a job thread model, where they have one thread dedicated to Graphics, one thread dedicated to disk I/O, one thread dedicated to Sound, etc. This is usually a good model for video games because most APIs used there, especially ones like OpenGL, are in no way thread-safe.

I tend to use what's called a worker thread model, where threads are spawned off to do a very specific task (read this PNG file from disk, decode the bytes) — where I share pretty much nothing except some starting task data, and the result data when nothing is complete. It's effectively a separate process without the overhead of IPC.

Oh boy, that was probably a bit too much. If you read all this and have any questions on it, feel free to ask more questions. It's complex stuff.

# ? Mar 26, 2013 14:46

geonetix: Mar 6, 2011

To add to that; if you want to do stuff in parallel, it's often (at least in my circles) considered to be wiser to use multiprocessing instead of threading. Multiprocessing doesn't have the GIL thing; but requires a bit more thought on getting stuff done safely and nicely.

This is treading into "Quite Advanced" territory though, so if you're not completely comfortable with Python or the OS you're using it on, stay away until you grasp the related concepts better.

e: vvvvv Very true. I'll stress it a bit: stay away until you completely know what you're doing and what you're getting yourself into.

geonetix fucked around with this message at 15:48 on Mar 26, 2013

# ? Mar 26, 2013 15:23

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Multiprocessing is a giant pile of magic. If it works, it's p. cool. And ifwhen it breaks (and it will break), you are up poo poo's creek without a paddle.

For a start, it's not truly cross-platform. You can write a multiprocessing script that will work fine on Linux (minus some special caveats), but will break on Windows, even without using any OS-level features, because of how it marshals data across process boundaries.

# ? Mar 26, 2013 15:38

Thern: Aug 12, 2006; Say Hello To My Little Friend

You can release the GIL? I need to investigate this further as I have something that is very I/O bound. And Multiprocessing is a bit of hack I feel.

# ? Mar 26, 2013 18:36

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

You can't release the GIL from Python. Any time that you into a C function, the GIL gets released, as there's no Python objects being manipulated. A C function that wants to manipulate Python objects has to take the GIL itself. Think of the GIL as something that's taken when you want to manipulate some Python object, not something that one thread has all the time. If you have one thread that's blocked somewhere in I/O, nobody has the GIL.

# ? Mar 26, 2013 18:52

wwb: Aug 17, 2004

I'm starting to get into python a bit. I've done a few tutorials, learned the hard way, etc. But I'm running into an issue my google-fu fails at -- how does one properly structure a non-trivial python project.

For example, lets say you decided to roll your own RSS aggregator after Google told you go GFY. I come from a .NET background, given that I would structure it something like:

-- RSSReader.Core -> class library that provides most of the core functionality to the project, no UI just a .DLL
-- RSSReader.Core.Tests -> unit tests for said library
-- RSSReader.Web -> Web app dependent upon MyProject.Core
-- RSSReader.Cli -> Command line app for cron jobs, etc.

Is this just insane to approach this way in Python? Or if it isn't how would I build this?

If it helps I'm using jetbrains PyCharm as my IDE.

# ? Mar 26, 2013 18:55

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

Suspicious Dish posted:

Any time that you into a C function, the GIL gets released, as there's no Python objects being manipulated. A C function that wants to manipulate Python objects has to take the GIL itself. Think of the GIL as something that's taken when you want to manipulate some Python object, not something that one thread has all the time. If you have one thread that's blocked somewhere in I/O, nobody has the GIL.

This is not correct. When a C procedure is called from Python, the GIL is held by the current thread. The procedure can release the GIL if it knows that it won't be doing any manipulation of the interpreter state, but the programmer has to do that themselves and most don't bother.

# ? Mar 26, 2013 19:10

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Whoops, I'm wrong. I went into the Python source to refresh my memory, but missed some calls to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS.

# ? Mar 26, 2013 19:27

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

I have a code that presents a graph, and when I like it, I'll use the nice little GUI matplotlib provides to save it. I specifically do not use something like plt.savefig() because more times than not, I don't like the plot for some reason and need to change the data in some way.

When I try to overwrite an existing graph (usually a pdf file) I get the following error in the console. The save is successful (or at least looks like it is), so I don't understand what is going on in the back end. Any comments?

Tue Mar 26 13:19:54 <JETSGUY>.local Python[46054] <Error>: kCGErrorIllegalArgument: _CGSFindSharedWindow: WID 6614
Tue Mar 26 13:19:54 <JETSGUY>.local Python[46054] <Error>: kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged.
Tue Mar 26 13:19:54 <JETSGUY>.local Python[46054] <Error>: kCGErrorIllegalArgument: CGSGetWindowTags: Invalid window 0x19d6

Popper posted:

This is basically how people are thought classes.
Think like this:
Python code:
class Deck(object):
	
	def __init__(self):
		self.suits = #suit names
		self.ranks = #rank names
		self.cards = []

	def build(self):
		# Cards have three attributes, suit, rank, and the player who owns them. Here they all belong to "Dealer"
		[[self.cards.append(Card(suit, rank, "Dealer")) for rank in self.ranks] for suit in self.suits]
etc...

Card should be a class too.

There are faster ways to do this with dicts and namedtuples but get a class implementation working first.

Why are you inheriting from Object? I've seen this done before and I don't see why it is needed when declaring a top-level (sort of) class? Am I misunderstanding what you're doing?

# ? Mar 26, 2013 19:32

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 23:06

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

JetsGuy posted:

Why are you inheriting from Object? I've seen this done before and I don't see why it is needed when declaring a top-level (sort of) class? Am I misunderstanding what you're doing?

Inheriting from object in Python 2 gives you a new-style class. It's a fancy system, but if you're using Python 2, always inherit from object.

# ? Mar 26, 2013 19:33

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »