Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

I set out to make a Python program that interacts with the user through the command line and gives a (more-or-less) friendly interface allowing them to make changes to the contents of a database, or dump the database contents. I wasn't sure quite how to do it, but what I came up with is this: I have a class which defines a __call__() method. When the module is loaded, an object of this class is instantiated and assigned to main. As a result, the program can be launched by calling modulename.main(). The __call__() method looks like this:

Python code:

def __call__ (self, autocommit = False):
    '''See the class's docstring.'''

    self.autocommit = bool(autocommit)
    # Code goes here in which the "self" object is prepared.
    handlers = {
        self._mode_connect: self._handler_connect,
        self._mode_quit: self._handler_quit,
        self._mode_select_trainer: self._handler_select_trainer,
        ... (more) ...,
        self._mode_write: self._handler_write,
    }
    mode = self._mode_connect
    while mode:
        # When each handler returns, it tells us which mode to go to next,
        # or it returns a False-like value, in which case we are done.
        mode = handlers[mode]()

So the class also has a bunch of methods with names that follow the pattern _handler_*(), and for each one a corresponding constant _mode_*. Each one returns one of the _mode_* constants when it is done, except for _handler_quit(), which returns None.

The module also has a global dictionary "messages" (text to be output to the user) and a global dictionary "_queries" (SQL queries). Having the messages and queries inline was proving to be unwieldy.

Is this how something like this would ordinarily be done, or is it unusual? Is there a better way to implement this kind of behaviour? I found that having a set of individual handler functions let me strike a good balance between being able to keep some global state (by assigning to attributes of self) and keeping the logic of a particular handler localised. I did end up with one quite hairy handler function that is 570 lines long now that I have written a docstring and added comments to it, but the states it has to manage are so intricate I think that I would have had difficulty not making it huge.

# ? Sep 12, 2012 06:06

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 16:32

Don Mega: Nov 26, 2005

I am developing a multiplayer game and I have a question about using pickle. Is there any pros/cons between using the TrustedPickle module, which includes a signature to verify the data when unpickling, and overriding the pickle's find_class method to only allow it to unpickle classes you specify?

Edit: The clients will be connecting to each other, which is important to take into consideration.

Don Mega fucked around with this message at 16:57 on Sep 12, 2012

# ? Sep 12, 2012 15:47

Cat Plus Plus: Apr 8, 2011

Hammerite posted:

Is this how something like this would ordinarily be done, or is it unusual? Is there a better way to implement this kind of behaviour? I found that having a set of individual handler functions let me strike a good balance between being able to keep some global state (by assigning to attributes of self) and keeping the logic of a particular handler localised. I did end up with one quite hairy handler function that is 570 lines long now that I have written a docstring and added comments to it, but the states it has to manage are so intricate I think that I would have had difficulty not making it huge.

It's a pretty straightforward implementation of a state machine. If you have problem with particular state having too much logic, you can try splitting it up into substates. If you're following a strict naming convention, you can get rid of your handlers dict completely, if you make your _mode_ constants strings. Hell, I'd probably get rid of variables and just used literals everywhere, but that's a matter of preference.

Python code:

_mode_connect = 'connect'
# ...
mode = self._mode_connect
while mode:
    handler = getattr(self, '_handler_' + mode)()

Don Mega posted:

I am developing a multiplayer game and I have a question about using pickle. Is there any pros/cons between using the TrustedPickle module, which includes a signature to verify the data when unpickling, and overriding the pickle's find_class method to only allow it to unpickle classes you specify?

Edit: The clients will be connecting to each other, which is important to take into consideration.

pickle is good for quick and dirty serialisation, but Protocol Buffers (or something similar) is better suited to situations where compatibility matters and serialised data cannot be thrown out and recreated. You will invariably end up in a situation when two clients with two different versions of your game will try to talk to each other. You need an established protocol that doesn't change on a whim, and that's much easier to do when external data format is not intimately tied to internals.

I've never heard about TrustedPickle, but it's still pickle.

# ? Sep 12, 2012 18:20

Don Mega: Nov 26, 2005

Thanks, I looked into Protocol Buffers and it is exactly what I am looking for.

# ? Sep 12, 2012 19:58

Pythagoras a trois: Feb 19, 2004; I have a lot of points to make and I will make them later.

Sylink posted:

Alright, so I kept working on my scrabble program and ran into a problem.

I'm working on this problem, but I'm slowly learning not to type fresh code into a reply window.

edit: Where'd you pull your word list from? I've written enough code that I'm interested to see if it works.

code:

space_check = ['','','c','d','','','','h','','','k','','','','o']
space = space_check[:]
words_that_fit = []
start = 0
words = ["arc", "den", "hi", "kola", "kol", "heck", "kolao", "abcd"]
#make a list of positions that are already filled
filled_positions = []
for position in range(len(space[start:])):
	if space[position+start]:
		filled_positions.append(position+start)
			
while start < 15:
	
	#rewrite a version of your board with a word in place
	for word in words:
		match = 0
		space = space_check[:]
		try:
			for i in range(len(word)):
				space[i+start] = word[i]
		except:
			break
			
		# check that your begin and end
		# positions are open with a wild
		# if statement
		if (start == 0 or not space[start-1]   
		) and (len(word)+start == 15 or 
		space[start+len(word)]==''):
			
			# check that filled positions
			# are satisfied and append the
			# word to a white list
			for i in filled_positions:
				if space[i] == space_check[i]:
					match += 1
				else:
					match = 0
					break
			if match:
				words_that_fit.append(word)
	
	#increment start position
	start += 1

print words_that_fit
>>> ['abcd', 'hi', 'heck', 'kol', 'kolao']

Pretty much a cpu buster, but it works with the phoney made up words I included. basically it makes a copy of your gameboard, puts each word down on each starting location, and then checks to see if it broke game rules, existing tiles, board width, etc.

I'd recommend putting some sort of progress bar into your code before you search through a real dictionary, but this code should get there eventually.

Assuming the length of the dictionary is n, it should run in bigO(n) time.

Pythagoras a trois fucked around with this message at 13:45 on Sep 13, 2012

# ? Sep 12, 2012 23:10

Sylink: Apr 17, 2004

This is actually what I was about to try and its pretty much bruteforcing.

I think its the only way to do it and get the "ideal" or "maximum" answer because I looked up some other scrabble AI and they all act very fast by kind of avoiding an optimal answer and getting a decent one based on prefixes and weird tree methods of finding words.

Though, I don't know that it would be too hard to search on modern PCs. Loading an entire dictionary into memory is trivial (I'm using one I just found randomly online that is about 80k words and it takes less than a second to sort into a list).

My next step was to try something like what you came up with and brute force every option.

Then I would order by maximum points and fill in the board and confirm that adding the word doesn't create secondary words that aren't actually words (which should be easy).

# ? Sep 15, 2012 00:54

Pangolin Poetry: Jan 1, 2008

I'm an idiot and brand new at python and this is so basic that I am literally just staring at it and not understanding anything please help

code:

var = [list,of,etc.]
def fun(x)
   var.dostuff

And yet whenever fun is called it claims var is undefined in its scope. I thought that python would search the next closest scope for variables??? (both var and fun are inside a class). Why can't fun see var :smith:

I feel like I've done this before and it's worked I can't understand why it isn't here.

# ? Sep 16, 2012 22:52

ufarn: May 30, 2009

Pangolin Poetry posted:

I'm an idiot and brand new at python and this is so basic that I am literally just staring at it and not understanding anything please help
code:
var = [list,of,etc.]
def fun(x)
   var.dostuff
And yet whenever fun is called it claims var is undefined in its scope. I thought that python would search the next closest scope for variables??? (both var and fun are inside a class). Why can't fun see var I feel like I've done this before and it's worked I can't understand why it isn't here.

Can you be slightly more specific about what you want to do? Give us an example case of something to go through.

# ? Sep 16, 2012 23:04

FoiledAgain: May 6, 2007

Pangolin Poetry posted:

I'm an idiot and brand new at python and this is so basic that I am literally just staring at it and not understanding anything please help
code:
var = [list,of,etc.]
def fun(x)
   var.dostuff
And yet whenever fun is called it claims var is undefined in its scope. I thought that python would search the next closest scope for variables??? (both var and fun are inside a class). Why can't fun see var I feel like I've done this before and it's worked I can't understand why it isn't here.

Maybe you want this?

code:

def fun(x):
    x.dostuff

var = [list,of,stuff]
fun(var)

Although I don't really know what is in var or if it has a dostuff attribute, so this could easily fail for other reasons. (And did you mean var.dostuff()?)

# ? Sep 16, 2012 23:43

tef: May 30, 2004; -> some l-system crap ->

Pangolin Poetry posted:

code:
var = [list,of,etc.]
def fun(x)
   var.dostuff
(both var and fun are inside a class)

variables defined inside a class are only accessible via self or the class name. (rather than inside an __init__ ...)

if the function is inside a class, it should be taking self as the first arg.

code:

class Foo:
    baz = 1
    def bar(self):
        return self.baz

You can access it from Foo.baz or Foo().bar() - nb in this instance, baz is a class variable, rather than instance.

# ? Sep 16, 2012 23:44

Pangolin Poetry: Jan 1, 2008

I'll try... Hopefully this is clearer. The code isn't complete but it should give an idea of what's up

code:

class foo:
� ��
� � var = [ [ ] for x in xrange(3)] #a list of 3 lists which are filled with objects from another module
� ��
� � def fun(arg): �#here's the function that cant see var when its called
� � � var[0].method1()
� � � var[1][arg].method2()
� � ��
� � for count in range(1,104,10): �#method3 creates an instance of an object, which i then put into var
� � � x = method3(arg, arg2, arg3=partial(fun,x))
� � � var[0].append(x)
� � ��
� � for x in range(len(var[0])):
� � � var[0][x].method

Basically this all works fine and what it does is create some objects in Tkinter and places them in my gui. arg3 of the objects in var is the function called when the objects are interacted with. when I interact with the objects, it gives me the "global name var is not defined" error

Edit: wow posting on a phone is hard!!! If I give fun self as the first arg, do I have to pass it self when I call it in the same class? Or if not what do I pass in arg 1? I know it gets filled if I call it from outside the class but that doesn't seem to be the case here (since it still asks for two parameters if I only pass it one)

Pangolin Poetry fucked around with this message at 23:56 on Sep 16, 2012

# ? Sep 16, 2012 23:47

OnceIWasAnOstrich: Jul 22, 2006

Pangolin Poetry posted:

I'll try... Hopefully this is clearer. The code isn't complete but it should give an idea of what's up

..code..

Basically this all works fine and what it does is create some objects in Tkinter and places them in my gui. arg3 of the objects in var is the function called when the objects are interacted with. when I interact with the objects, it gives me the "global name var is not defined" error

self.var

Like tef said when you define a function inside a class the first argument should be self, if you don't put that it puts the object that you are calling it from in the first argument anyway, even if it isn't named self. Then you would just use self.var to access inside of a method.

When you call a method of an object with args (self, a) you only need to pass it a, Python automatically passes the object to the function, whether you are calling it as foo.f() or just f() from within the object.

OnceIWasAnOstrich fucked around with this message at 00:00 on Sep 17, 2012

# ? Sep 16, 2012 23:54

Pangolin Poetry: Jan 1, 2008

When I do that (give self, a as arguments and only pass a) it gives me a "fun takes exactly two arguments (1 given)" error

# ? Sep 17, 2012 00:06

Don Mega: Nov 26, 2005

The function should be defined with 'self' as the first argument.

# ? Sep 17, 2012 00:28

etcetera08: Sep 11, 2008

http://amoffat.github.com/sh/index.html

This looks cool and elegantly solves a problem that has come up in this thread more than once (executing subprocesses). I haven't used it yet but if it works as advertised I'll probably be using it sometime.

e: looking at this more and it seems like you could write some really powerful scripts with it.

etcetera08 fucked around with this message at 03:23 on Sep 17, 2012

# ? Sep 17, 2012 03:18

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

What's hard about the subprocess module?

# ? Sep 17, 2012 03:30

Plorkyeran: Mar 22, 2007; To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed

It involves far too much code for the use-case of one-off shell scripts written in a programming language that isn't a retarded mess. I wouldn't use the sh module for anything I'm not going to delete a few minutes after running, but I've come to quite like it for that sort of thing.

# ? Sep 17, 2012 03:45

etcetera08: Sep 11, 2008

Suspicious Dish posted:

What's hard about the subprocess module?

Nothing, but it's not nearly as nice as sh.py imo.

# ? Sep 17, 2012 04:41

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

etcetera08 posted:

Nothing, but it's not nearly as nice as sh.py imo.

Why not?

See, to me, sh.py looks terrible. I don't like those sort of generated interfaces. If I do from sh import nonexistant_program, what does it fail with? How do I run a program with a - in the name? If I have a program that's not on PATH, what do I do? Can I run programs with the same name in different locations, or am I limited to a merged PATH namespace?

Most other subprocess modules I've seen are open to shell injection for "convenience". Is this one? What happens if I do ls("-lh *.wav") instead of the proper ls("-lh", "*.wav")? Will it work? How does it construct and parse the string?

I wonder what happens if you do from sh import *.

Does it block execution before continuing? How can I do parallel execution of four subprocesses, and get notified when all four are done?

Plorkyeran posted:

It involves far too much code for the use-case of one-off shell scripts written in a programming language that isn't a retarded mess. I wouldn't use the sh module for anything I'm not going to delete a few minutes after running, but I've come to quite like it for that sort of thing.

For one-off scripts, it seems like it might be handy. But I'm deathly afraid of this thing.

# ? Sep 17, 2012 04:47

etcetera08: Sep 11, 2008

Suspicious Dish posted:

Why not?

See, to me, sh.py looks terrible. I don't like those sort of generated interfaces. If I do from sh import nonexistant_program, what does it fail with? How do I run a program with a - in the name? If I have a program that's not on PATH, what do I do? Can I run programs with the same name in different locations, or am I limited to a merged PATH namespace?

All of these are clearly addressed in the documentation.

Suspicious Dish posted:

Most other subprocess modules I've seen are open to shell injection for "convenience". Is this one? What happens if I do ls("-lh *.wav") instead of the proper ls("-lh", "*.wav")? Will it work? How does it construct and parse the string?

I wonder what happens if you do from sh import *.

Passing arguments is also clearly addressed. Not sure about the wildcard import though.

Suspicious Dish posted:

Does it block execution before continuing? How can I do parallel execution of four subprocesses, and get notified when all four are done?

http://amoffat.github.com/sh/index.html#iterating-over-output
Seems to indicate you can do non-blocking monitoring?

Suspicious Dish posted:

For one-off scripts, it seems like it might be handy. But I'm deathly afraid of this thing.

There is definitely an element of trust in anything like this and I think this falls into the 'convenience module' category as opposed to a 'robust & failsafe module' category, but I also think there is room for both. Almost all of your usability questions are easily answered by skimming the concise documentation and the examples provided are really useful. That alone makes it valuable for quick and dirty scripts.

I have no horse in this race, I just think it looks like it could be useful for people and it's a topic that has come up in this thread. I was simply providing the link for others who may find use in it.

# ? Sep 17, 2012 07:56

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

Pangolin Poetry posted:

When I do that (give self, a as arguments and only pass a) it gives me a "fun takes exactly two arguments (1 given)" error

Basically: When you define a function within a class, that function becomes an attribute of your class. So if you do this:

Python code:

    class cat:
        def meow ():
            print('Meow')

you can then call the "meow" method by typing cat.meow() (obviously from a suitable scope). That is how it behaves if there is no actual instantiation of the class involved, only the class object itself ("cat" here).

When you create an instantiation of the class, by typing something like fluffy = cat(), you can then access the "meow" method through "fluffy" (an object of class cat) as well as through cat (a class object). However, when you type fluffy.meow(), that is treated as though you had typed cat.meow(fluffy). That is, what really happens is that the "meow" method of "cat" is called and the relevant cat object is prepended to the list of arguments that you supplied.

That is why you always include "self" as the first parameter when you define a class method, because if you don't then you cannot call it through an instantiation of the class without potentially odd things happening. This is also why the "self" object is available in the object method's scope.

(You do not have to call it "self", but it's established convention to do so.)

Hammerite fucked around with this message at 08:01 on Sep 17, 2012

# ? Sep 17, 2012 07:59

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

etcetera08 posted:

All of these are clearly addressed in the documentation.

They're not. The only one addressed was the dash replacement, but it doesn't specify what happens in the edge case of "if I have google-chrome and google_chrome, which one wins? What about /usr/bin/google-chrome and /usr/local/bin/google_chrome"? That is, will it exhaust the PATH before trying fallbacks replacements or will it check all replacements at once?

I can specify an absolute path to a binary with sh.Command, but can I set up my own magic context? Something like my_sh = sh.DoYourMagicPlease(path=["/opt/mything/bin", "/opt/mything/local/bin"])?

etcetera08 posted:

Passing arguments is also clearly addressed. Not sure about the wildcard import though.

My question of how arguments are constructed and how vulnerable something like this is to shell injection was not answered.

etcetera08 posted:

http://amoffat.github.com/sh/index.html#iterating-over-output
Seems to indicate you can do non-blocking monitoring?

Is EWOULDBLOCK seriously the only interface I get? The subprocess module is non-blocking by default.

Suspicious Dish fucked around with this message at 08:09 on Sep 17, 2012

# ? Sep 17, 2012 08:07

Stabby McDamage: Dec 11, 2005; Doctor Rope

Fairly dumb question...I'm trying to add a simple console "throbber" (a slash going back and forth). I want to use an inline function for this, but I can't get the inline function to maintain state in the outer function without obscene trickery (abusing mutable objects in the parameter list). Code:

Python code:

import sys,time

def monitor_process1():
	sl = True
	def slash():
		sys.stdout.write("\r" + '/\\'[sl])
		sys.stdout.flush()
		sl = not sl
	while True:
		# ... do stuff ...
		slash()
		time.sleep(1)

def monitor_process2():
	def slash(sl=[True]):
		sys.stdout.write("\r" + '/\\'[sl[0]])
		sys.stdout.flush()
		sl[0] = not sl[0]
	while True:
		# ... do stuff ...
		slash()
		time.sleep(1)

monitor_process1 is the naive solution I thought should work, but it fails with:

code:

Traceback (most recent call last):
  File "mon.py", line 24, in <module>
    monitor_process1()
  File "mon.py", line 11, in monitor_process1
    slash()
  File "mon.py", line 6, in slash
    sys.stdout.write("\r" + '/\\'[sl])
UnboundLocalError: local variable 'sl' referenced before assignment

monitor_process2 is the gross hack that exploits that mutable parameters are defined with the function and can be modified after that.

What's the elegant way to do this?

EDIT: I could do this, but it's a bit wordy:

Python code:

class Throb():
	def __init__(self):
		self.sl=True
	def __call__(self):
		sys.stdout.write("\r" + '/\\'[self.sl])
		sys.stdout.flush()
		self.sl = not self.sl

def monitor_process3():
	slash = Throb()
	while True:
		# ... do stuff ...
		slash()
		time.sleep(1)

Stabby McDamage fucked around with this message at 16:24 on Sep 17, 2012

# ? Sep 17, 2012 16:21

Cat Plus Plus: Apr 8, 2011

Stabby McDamage posted:

What's the elegant way to do this?

EDIT: I could do this, but it's a bit wordy:

Using a class with __call__ is a correct way to do stateful callables.
You could make it an attribute on a normal function, but that makes it a global state and hurts reusability. Maybe won't matter in this case, but it's definitely not elegant.

# ? Sep 17, 2012 16:50

Stabby McDamage: Dec 11, 2005; Doctor Rope

PiotrLegnica posted:

Using a class with __call__ is a correct way to do stateful callables.
You could make it an attribute on a normal function, but that makes it a global state and hurts reusability. Maybe won't matter in this case, but it's definitely not elegant.

Cool.

Semi-related question: Is there a way to get a reference to a function inside the function? I'm thinking of using threading.Timer to make a periodic timer, which involves re-scheduling the next call at the end of this one. I'd like to be able to say __this__func__ or something so I can rename the function later without breaking things. It's a minor ask -- more of my preference to avoid needless repetition.

# ? Sep 17, 2012 17:18

Cat Plus Plus: Apr 8, 2011

Stabby McDamage posted:

Semi-related question: Is there a way to get a reference to a function inside the function?

Technically it's possible to get the name by inspecting current frame, but it's ugly and possibly not portable across implementations.

A class is better in this case, too.

# ? Sep 17, 2012 17:32

Pangolin Poetry: Jan 1, 2008

Hammerite posted:

explanation

This makes sense, but I'm still confused about implementation, because when I do this:

code:

class Cat:
� �def meow(self, args):
� � � things

foo = Cat()
bar = partial(foo.meow(args))

I get this error:�unbound method meow() must be called with Cat instance as first argument (got args instance instead)

If I understand what you wrote correctly, since writing foo.meow() is the same as cat.meow(foo), then writing foo.meow(args) should be the same as cat.meow(foo,args), and I am calling meow with an instance of Cat as the first argument. Why is this not the case? I feel retarded for not grasping this since it seems really, really basic. Thanks for the help everybody :downs:

# ? Sep 17, 2012 18:28

The Gripper: Sep 14, 2004; i am winner

Pangolin Poetry posted:

This makes sense, but I'm still confused about implementation, because when I do this:
code:
class Cat:
� �def meow(self, args):
� � � things

foo = Cat()
bar = partial(foo.meow(args))
I get this error:�unbound method meow() must be called with Cat instance as first argument (got args instance instead)

Probably because you're some kind of crazy person, why are you using partial there? Running that code without it works fine on my end, and gives a completely different error to yours with it:

Python code:

class Cat:
   def meow(self, args):
      print "I AM DOING A THING"

foo = Cat()
foo.meow([])

#E:\code>python test.py
#I AM DOING A THING

~~I'm guessing you've accidentally written foo = Cat.meow() instead of foo = Cat(), and the error you're getting is on that line and not the bar one (which is weird for the reason above).~~ I have no idea how you managed to get that error with the code you provided.

If you really want to use partial() (maybe you don't) you'd need to change it to this:

Python code:

bar = partial(foo.meow,args="whatever")
bar()

I actually completely forgot partial existed and it'll come in handy so thanks for the reminder, I guess!

The Gripper fucked around with this message at 19:09 on Sep 17, 2012

# ? Sep 17, 2012 18:47

Pangolin Poetry: Jan 1, 2008

I'm using partial because if I just stick a regular function call inside my object (a Tkinter button), it calls the function when the object is instantiated and turns the function into a return value, which is no good to me. Partial lets me get around that (or so claims StackOverflow - I haven't actually been able to test this yet). The code does work fine if I don't use partial so that's good, I guess, but now I don't understand why partial is making things go weird! There's a line in the partial docs about how objects defined in classes don't behave like static methods and don't turn into bound methods during instancing. o_o this merits more investigation. I'd like to use classes if I can but it seems that my current way of addressing this problem isn't going to let me... We'll see

Edit: What the gently caress! I just un-commented everything after testing if it worked without partial and now everything works. I didn't change a thing!!! There's some black sorcery going on here. Or I'm dumb and changed something without realizing it.

Pangolin Poetry fucked around with this message at 19:07 on Sep 17, 2012

# ? Sep 17, 2012 19:01

Civil Twilight: Apr 2, 2011

The Gripper posted:

I have no idea how you managed to get that error with the code you provided.

I do. That's not how functools.partial works.

Python code:

from functools import partial

class Cat:
    def meow(self, args):
        print "mrow (args was '%s')" % args

foo = Cat()
print '-- calling foo.meow'
foo.meow('blah')

bar = partial(foo.meow, 1)
print '--calling bar'
bar()

Shell session code:

% python cat.py
-- calling foo.meow
mrow (args was 'blah')
--calling bar
mrow (args was '1')

in Pangolin's code, he calls the meow function instead of just referring to it:

Python code:

bar = partial(foo.meow(args))  # wrong
bar = partial(foo.meow, args)  # right, assuming 'args' is defined in this scope

Edit: by the way, the use of 'args' has me concerned that you're seeing method definitions like
def something(self, *args)
and don't understand what the asterisk is for. You should look into that.

Civil Twilight fucked around with this message at 19:09 on Sep 17, 2012

# ? Sep 17, 2012 19:04

The Gripper: Sep 14, 2004; i am winner

Pangolin Poetry posted:

I'm using partial because if I just stick a regular function call inside my object (a Tkinter button), it calls the function when the object is instantiated and turns the function into a return value, which is no good to me. Partial lets me get around that (or so claims StackOverflow - I haven't actually been able to test this yet). The code does work fine if I don't use partial so that's good, I guess, but now I don't understand why partial is making things go weird! There's a line in the partial docs about how objects defined in classes don't behave like static methods and don't turn into bound methods during instancing. o_o this merits more investigation. I'd like to use classes if I can but it seems that my current way of addressing this problem isn't going to let me... We'll see

In that case maybe you do want to use partial, but you'd need to use it the way my above edit has it (don't put foo.meow() in the call to partial as that will call it immediately, put foo.meow).

Civil Twilight posted:

I do. That's not how functools.partial works.

in Pangolin's code, he calls the meow function instead of just referring to it:
Python code:
bar = partial(foo.meow(args))  # wrong
bar = partial(foo.meow, args)  # right, assuming 'args' is defined in this scope

The first line there throws a TypeError: the first argument must be callable since the return value of the function isn't callable, I couldn't reproduce any combo of calls that would announce that an instance first argument was missing.

The Gripper fucked around with this message at 19:08 on Sep 17, 2012

# ? Sep 17, 2012 19:05

Civil Twilight: Apr 2, 2011

The Gripper posted:

The first line there throws a TypeError: the first argument must be callable since the return value of the function isn't callable, I couldn't reproduce any combo of calls that would announce that an instance first argument was missing.

You're right; I swear I managed to reproduce it when I was setting up my example above, but I've closed that window now and I don't know what I did.

# ? Sep 17, 2012 19:12

Pangolin Poetry: Jan 1, 2008

I actually had it as partial(fun,args) and not partial(fun(args)) in my code - when I transcribed it onto my phone I did so incorrectly. Thanks for the reading material though! I need all the resources I can get. Thanks for the help guys, it's really useful to have more experienced people to go for when google searches fail.

# ? Sep 17, 2012 19:14

The Gripper: Sep 14, 2004; i am winner

Civil Twilight posted:

You're right; I swear I managed to reproduce it when I was setting up my example above, but I've closed that window now and I don't know what I did.

As Pangolin Poetry put it, black sorcery. If something makes no sense, a wizard did it.

# ? Sep 17, 2012 19:14

Comrade Gritty: Sep 19, 2011; This Machine Kills Fascists

Suspicious Dish posted:

They're not. The only one addressed was the dash replacement, but it doesn't specify what happens in the edge case of "if I have google-chrome and google_chrome, which one wins? What about /usr/bin/google-chrome and /usr/local/bin/google_chrome"? That is, will it exhaust the PATH before trying fallbacks replacements or will it check all replacements at once?

I can specify an absolute path to a binary with sh.Command, but can I set up my own magic context? Something like my_sh = sh.DoYourMagicPlease(path=["/opt/mything/bin", "/opt/mything/local/bin"])?

Using _env from the Documentation: my_sh = sh.DoYourMagcPlease(_env={"PATH": ":".join(["/opt/mything/bin", "/opt/mything/local/bin"])})

Suspicious Dish posted:

My question of how arguments are constructed and how vulnerable something like this is to shell injection was not answered.

It mentions globing, specifically that globbing is not done by default. One can reasonably assume that they didn't go out of their way to disable shell globbing and instead they are not using a shell. A quick perusal of the source code confirms this.

Suspicious Dish posted:

Is EWOULDBLOCK seriously the only interface I get? The subprocess module is non-blocking by default.

_bg=True is hard? Subprocess might be nonblocking by default but sh isn't. Neither one is the right choice. It's a design call and subprocess chose to make it non blocking by default whereas sh chose to make it blocking by default.

# ? Sep 17, 2012 21:59

raminasi: Jan 25, 2005; a last drink with no ice

I have a question that might be more appropriate for the scientific computing megathread, but I think I should start here.

I'm pretty new to Python. I whipped a short little thing up for a project I'm working on. It's pretty simple - it just calculates some simple normal distributions, samples from them, fiddles around with the samples a little, and plots the result.

I didn't think this would be that complicated, but the performance is much worse than I'd expected. It takes over a minute to run. Is there some glaring thing I'm doing wrong? Like I said, I'm pretty new to Python in general and this is my first time doing anything with NumPy or SciPy. (Also, if there are Python conventions I'm not following it'd be great to hear about those too.)

# ? Sep 18, 2012 00:37

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

You could just toss in some timestamps to see if there's one particular part of your script that is sucking up all the computation time. Shouldn't take more than a few minutes to narrow it down.

# ? Sep 18, 2012 00:45

Emacs Headroom: Aug 2, 2003

GrumpyDoctor posted:

I have a question that might be more appropriate for the scientific computing megathread, but I think I should start here.

I'm pretty new to Python. I whipped a short little thing up for a project I'm working on. It's pretty simple - it just calculates some simple normal distributions, samples from them, fiddles around with the samples a little, and plots the result.

I didn't think this would be that complicated, but the performance is much worse than I'd expected. It takes over a minute to run. Is there some glaring thing I'm doing wrong? Like I said, I'm pretty new to Python in general and this is my first time doing anything with NumPy or SciPy. (Also, if there are Python conventions I'm not following it'd be great to hear about those too.)

You should definitely profile, but style-wise you should stop using map so much and try to use vectorized operations as much as possible (with numpy arrays), and list comprehensions anywhere else. For instance, line 53 should be hourly_availability_samples = demand_samples * flex_samples, where all three are 1d numpy arrays.

Also, this is more minor, but numpy.random already has a randn function, so you don't need to build the distributions from scipy.stats to sample from them.

# ? Sep 18, 2012 01:30

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Steampunk Hitler posted:

Using _env from the Documentation: my_sh = sh.DoYourMagcPlease(_env={"PATH": ":".join(["/opt/mything/bin", "/opt/mything/local/bin"])})

Why would the environment of the process that you're trying to exec affect path resolution in the current process? Are they double forking and doing path resolution in a subchild in that environment?

And I think you read my API issue wrong. What I'd like is:

Python code:

my_sh = sh.DoYourMagicPlease(path=["/opt/apache/bin", "/opt/apache/sbin"])
my_sh.apachectl("restart")

Steampunk Hitler posted:

It mentions globing, specifically that globbing is not done by default. One can reasonably assume that they didn't go out of their way to disable shell globbing and instead they are not using a shell. A quick perusal of the source code confirms this.

They could still be building a string for convenience, so sh.sudo("sync /my-directory/") will work along with sh.sudo("sync", "/my-directory/").

Steampunk Hitler posted:

_bg=True is hard? Subprocess might be nonblocking by default but sh isn't. Neither one is the right choice. It's a design call and subprocess chose to make it non blocking by default whereas sh chose to make it blocking by default.

How do I read output from a "background process"? Send input?

Suspicious Dish fucked around with this message at 03:49 on Sep 18, 2012

# ? Sep 18, 2012 03:47

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 16:32

Pangolin Poetry: Jan 1, 2008

Wouldn't mind having somebody take a look at my code / logic for this operation and letting me know if there's a more Pythonic way of doing it, since I'm new to the language. This feels kind of hamfisted and like I'm missing some function of Python data types! Thanks in advance.

So the idea is this: I want a dictionary where a shared data type between two sets of data is used to cross-reference between them. Both data sets are entered into dictionaries already with the keys being their data types - so they share a key. The data is arranged in tables so if the shared "columns" match I want to get the entire row (e.g all the rest of the data for that entry) and pair it with that shared datum value. Hopefully this makes sense but it's early so I'm not entirely lucid.

code:

dict_keys = [ ] #it's a lot of work to iteratively create a tuple so I just make a list and turn it into a tuple with tuple()
� for x in range(len(data1_dict["key"])): dict_keys.append(data1_dict["key"][x]) #make a list of all the keys in data set 1 that I want to use
� dict_keys = tuple(dict_keys)
��
� dict_values = [ ]
��
� for x in range(len(data1_dict["key"])): #we go through this loop once per key
� � � dataX = data1_dict["key"][x] #get the particular key value to search for this time thru the loop
� � � temp_value_list = [ ]
� � � for y in range(len(data2_dict["key"])): #search through the entirety of the 2nd data set each time
	 �if dataX == data2_dict["key"][y]: #if we have a match...
	 � � �even_tempier_list = []
#since the data is arranged in tabular form, we grab the entirety of each entry that matches and make a list
	 � � �for z in range(len(data2_columns)): even_tempier_list.append(data2_dict["%s" % (db.data2_columns[z])][y])�
	 � � �temp_value_list.append(even_tempier_list) #and stick this list into our other list...
� � � dict_values.append(temp_promotion_list) #and then stick the list of lists into our list of values
��
#once that loop is all finished and we have all our data, we make the dictionary
� dict = { }
� for x in range(len(dict_keys)): dict[dict_keys[x]] = dict_values[x]

I like the way that the data ends up being organized (I can access all the data associated with a particular value just by calling dict["key"], or one set in particular with dict["key"][n], one value from that set with dict["key"][n][m], etc. etc.) but looping through every possible entry searching for a match for each value seems like a lot of wasted CPU cycles since there are only a few matches for each value. Is there a more efficient way to do this? Any other criticisms of my code would be greatly appreciated as well. Trying my best to learn... Thanks!

# ? Sep 18, 2012 06:42

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »