Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

tef: May 30, 2004; -> some l-system crap ->

Senso posted:

I never really used threads before, with the reputation of being more complex and hard to debug. I'll give threads a look tomorrow. Is sharing objects between threads much easier than with multiprocessing?

multiprocessing is a drop in replacement for threading that uses processes instead. you should have little problem using similar constructs (threading.Queue iirc)

# ? Nov 1, 2011 18:33

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 09:54

Senso: Nov 4, 2005; Always working

tef posted:

multiprocessing is a drop in replacement for threading that uses processes instead. you should have little problem using similar constructs (threading.Queue iirc)

True, I was able to switch to threads in a few minutes. I don't even need a Queue since, for now, I have one thread writing and one thread reading and global items can be accessed by both threads.
It's dead simple like that, should have looked into that in the first place...

# ? Nov 2, 2011 09:45

Lurchington: Jan 2, 2003; Forums Dragoon

I don't normally go out of my way to mention this, but for what you're talking about (multiple threads accessing the same global items) multiprocessing is NOT a drop in replacement. Don't worry about the specifics for now, but the subtleties definitely tripped me up when going from threads to processing.

# ? Nov 2, 2011 15:55

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:12 on Mar 13, 2017

# ? Nov 4, 2011 01:25

Met48: Mar 15, 2009

It'd be a lot easier to use a dictionary for that, or even better, the Counter class in the collections module. You can pass it a list and it will count each element's occurrences.

Although it's not very thoroughly tested, this should work. It uses a list comprehension to create a list of number pairs and the Counter class to get each pair's occurrence.

code:

import collections

def gapTest(gap, numbers):
	nums = numbers
	pairs = ((nums[x], nums[x+gap+1]) for x in range(0, len(nums) - 1 - gap))
	return collections.Counter(pairs)

if __name__=='__main__':
	#Prints Counter({(1, 2): 2, (2, 3): 2, (3, 4): 1, (3, 2): 1, (2, 1): 1})
	print gapTest(0, [1, 2, 1, 2, 3, 2, 3, 4])

# ? Nov 4, 2011 01:57

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:25 on Mar 13, 2017

# ? Nov 4, 2011 02:00

Lurchington: Jan 2, 2003; Forums Dragoon

well Counter is new in 2.7/3.1 I believe, so that may be why? Not sure if you mentioned all the particulars of your environment

# ? Nov 4, 2011 02:08

Met48: Mar 15, 2009

GregNorc posted:

Cool, thanks. What is the if name == main bit doing?

Also, if you know what was causing the error in the original code, could you let me know? I'd like to try and learn from the mistake.

Last part is kind of irrelevant to the issue, just habit. Executes that section only if you're running the file itself, rather than importing it.

I think your code needs a slight change on line 42: follow = data[i + gap + 1]

Not sure on the error.

# ? Nov 4, 2011 02:10

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:25 on Mar 13, 2017

# ? Nov 4, 2011 02:12

Met48: Mar 15, 2009

Ok, that makes sense. If you don't have the Counter class you can get the same effect using dictionaries:

code:

def count_elements(aList):
	counts = {}
	for item in aList:
		if item not in counts:
			counts[item] = 0
		counts[item] += 1
	return counts

# ? Nov 4, 2011 02:18

FoiledAgain: May 6, 2007

Metroid48 posted:

Ok, that makes sense. If you don't have the Counter class you can get the same effect using dictionaries:

This is a touch shorter, and defaultdict should be available in GregNorc's version of Python.

code:


from collections import defaultdict

def count_elements(aList):
	counts = defaultdict(int)
	for item in aList:
		counts[item] += 1
	return counts

# ? Nov 4, 2011 02:34

Scaevolus: Apr 16, 2007

FoiledAgain posted:

This is a touch shorter, and defaultdict should be available in GregNorc's version of Python.

Yet another alternative, using dict.get(key, default):

code:

def count_elements(aList):
	counts = {}
	for item in aList:
		counts[item] = counts.get(item, 0) + 1
	return counts

# ? Nov 4, 2011 02:36

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:25 on Mar 13, 2017

# ? Nov 4, 2011 02:37

Scaevolus: Apr 16, 2007

GregNorc posted:

What I need to do is store how often X follows Y with a gap of N. So if the gap is 0, then how often does X directly follow y?

Should 233 have two entries for 2->3 or just one?

# ? Nov 4, 2011 02:41

Met48: Mar 15, 2009

GregNorc posted:

This is a cool idea, but it doesn't do exactly what I'm trying to do.

What I need to do is store how often X follows Y with a gap of N. So if the gap is 0, then how often does X directly follow y?

All that would store is how often each digit comes up... I need how often each number followed another number.

It's a replacement for the collections.Counter part from the first code snippet

Using the defaultdict edit FoiledAgain posted:

code:

from collections import defaultdict

def count_elements(aList):  # FoiledAgain
	counts = defaultdict(int)
	for item in aList:
		counts[item] += 1
	return dict(counts)

def gapTest(gap, numbers):
	nums = numbers
	pairs = ((nums[x], nums[x+gap+1]) for x in range(0, len(nums) - 1 - gap))
	return count_elements(pairs)
	
#Prints {(1, 2): 2, (4, 1): 1, (3, 4): 1, (2, 3): 1}
#Ex. 2 followed 1 twice
print gapTest(0, [1, 2, 3, 4, 1, 2])

# ? Nov 4, 2011 02:54

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:25 on Mar 13, 2017

# ? Nov 4, 2011 03:04

Met48: Mar 15, 2009

GregNorc posted:

If we had 1234567890 with a gap of 2

3 follows 1 once. Increment the appropriate variable
4 follows 2. Increment the appropriate variable
5 follows 3 once. Increment the appropriate variable
...

Looks like a gap of 1!

GregNorc posted:

Edit: And I ran the code Metroid48 posted, and I still get an error:

...
NameError: global name 'defaultdict' is not defined

Make sure you have "from collections import defaultdict" at the top of the file. If it's still giving that error even with the line, replace count_elements with the first version I posted since it's the simplest (albeit least concise) version:

code:

def count_elements(aList):
	counts = {}
	for item in aList:
		if item not in counts:
			counts[item] = 0
		counts[item] += 1
	return counts

# ? Nov 4, 2011 03:26

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:25 on Mar 13, 2017

# ? Nov 5, 2011 23:53

Met48: Mar 15, 2009

The error in your code is line 42, which should read: follow = data[i+gap+1]
The +1 is missing from your version, so with a gap size of 0 you are always getting the same number for prior and follow. Note that you'll also need to add a +1 to the loop condition on line 37 to avoid an index out of range error.

As for the code sample I posted:

Metroid48 posted:

If it's still giving that error even with the line, replace count_elements with the first version I posted since it's the simplest (albeit least concise) version:

That isn't the full file, that's just an adjustment to make it work on your version of python. Here's the full file you should have:

code:

def count_elements(aList):
	counts = {}
	for item in aList:
		if item not in counts:
			counts[item] = 0
		counts[item] += 1
	return counts

def gapTest(gap, numbers):
	nums = numbers
	pairs = ((nums[x], nums[x+gap+1]) for x in range(0, len(nums) - 1 - gap))
	return count_elements(pairs)
	
#Prints {(1, 2): 2, (4, 1): 1, (3, 4): 1, (2, 3): 1}
#Ex. 2 followed 1 twice
print gapTest(0, [1, 2, 3, 4, 1, 2])

I've just tested this and gapTest does what you want.

# ? Nov 6, 2011 08:56

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:25 on Mar 13, 2017

# ? Nov 6, 2011 23:22

FreakyMetalKid: Nov 23, 2003

I've recently been attempting to jump into Python to script my webservice testing. I've run into a BadStatusLine error that I just can't get my head around. I created a question over at StackOverflow: http://stackoverflow.com/questions/8031636/python-badstatusline-error-on-certain-delete-requests. Any thoughts?

# ? Nov 7, 2011 04:01

maskenfreiheit: Dec 30, 2004

Edit: Double Post

maskenfreiheit fucked around with this message at 20:36 on Mar 13, 2017

# ? Nov 10, 2011 02:10

dirby: Sep 21, 2004; Helping goons with math

GregNorc posted:

Is there anything I can do? Maybe use some special, larger datatype?

The numbers involved in that code are way too large to be computed by anything. That formula also won't help you with your original question.

I recommend you follow the advice in ultrafilter's post, or, if you're desperate for a crash course in precisely what you need to solve your problem make a thread in SAL for your original problem and people (possibly even me) might be willing to help you out.

Edit: Fixed link to ultrafilter's post.

dirby fucked around with this message at 03:01 on Nov 10, 2011

# ? Nov 10, 2011 02:51

ryo: Jan 15, 2003

When writing modules, where is it best to put the exception handling?
For example, I've written three modules;
One that describes a Publication class with a title, date, author etc,
One that handles database connectivity and CRUD operations,
One that connects to an API and queries for XML documents and creates a Publication object.

I then have a main script that imports the modules, and does what it needs to do (i.e. get the publications and put them in the database).

Should I be putting all the exception handling within the functions in the modules themselves? Or should I let the exceptions fall through to the main script and handle them all there?

A module that doesn't have any exception handling seems a bit strange to me, but then when unit testing you expect invalid data to be raising exceptions.

What's the "pythonic" way of going about this?

# ? Nov 10, 2011 12:48

duck monster: Dec 15, 2004

A module is just python code like any other.

Catch the errors inside your code, then push an appropriate exception upwards towards whatevers calling it, so it can deal with it, unless you want to handle it yourself then dont.

Just define the exceptions you want to describe at the start of your code, then raise your FooBarException where it needs to be, and then when your unit testing , make sure your code raises that exception you defined where the failure conditions are met.

Exceptions are a pain in the rear end, but at least we're not doing that @who_gives_a_fuck_if_our_code_sucks() poo poo that the php people do. Every time I see code that just lazily swallows exceptions in PHP I want to pound some coders loving head in.

duck monster fucked around with this message at 14:23 on Nov 10, 2011

# ? Nov 10, 2011 14:20

fnordcircle: Jul 7, 2004; PTUI

What's the best way to remove items with a certain character in them from a list?

What I've tried so far has been weird with one of the elements not getting deleted.

code:

        print("Sorted sources list before dash removal:\n",sorted_sources_list,"\n\n")            
        for each in sorted_sources_list:
            print("Processing",each)
            if "-" in each:
                sorted_sources_list.remove(each)
        print("\n\nWith dashes removed:\n",sorted_sources_list,"\n\n")

Results:

code:

Sorted sources list before dash removal:
 ['10.16.20.40 255.255.255.0', '145.95.200.55 255.255.0.0', '192.168.67.20 255.255.255.0', 
'140.95.8.180-140.95.8.180', '140.95.8.233-140.95.8.233', '140.95.8.70-140.95.8.71', '140.95.8.180 
255.255.255.255', '140.95.8.233 255.255.255.255', '140.95.8.70 255.255.255.255', '140.95.8.71 
255.255.255.255'] 


Processing 10.16.20.40 255.255.255.0
Processing 145.95.200.55 255.255.0.0
Processing 192.168.67.20 255.255.255.0
Processing 140.95.8.180-140.95.8.180
Processing 140.95.8.70-140.95.8.71
Processing 140.95.8.233 255.255.255.255
Processing 140.95.8.70 255.255.255.255
Processing 140.95.8.71 255.255.255.255


With dashes removed:
 ['10.16.20.40 255.255.255.0', '145.95.200.55 255.255.0.0', '192.168.67.20 255.255.255.0', 
'140.95.8.233-140.95.8.233', '140.95.8.180 255.255.255.255', '140.95.8.233 255.255.255.255',
 '140.95.8.70 255.255.255.255', '140.95.8.71 255.255.255.255']

I just can't seem to get that one value, '140.95.8.233-140.95.8.233', to come out and it's driving me bonkers.

Still pretty new at this but my searches have yielded stuff that I don't quite grasp yet as far as answers so it's hard to tell if that's what I need or not.

Edit: Well, this seemed to work instead of all that stuff above, though I'm not quite sure what it does but I can figure that out in more depth on my own:

sorted_sources_list = [x for x in sorted_sources_list if "-" not in x]

fnordcircle fucked around with this message at 16:45 on Nov 10, 2011

# ? Nov 10, 2011 16:39

taqueso: Mar 8, 2004

I don't think you can modify the list you are looping through.

# ? Nov 10, 2011 16:47

spankweasel: Jan 4, 2006

rule number 1 for working with lists: never ever delete elements of the list while you are iterating over it. It will do unpredictable things because you're changing the list as you're iterating.

quote:

Edit: Well, this seemed to work instead of all that stuff above, though I'm not quite sure what it does but I can figure that out in more depth on my own:

sorted_sources_list = [x for x in sorted_sources_list if "-" not in x]

This is a list comprehension. You are applying logic statements to construct a new list while iterating over the original list.

It is exactly the same as:

code:

new_list = []
for x in sorted_sources_list:
    if "-" not in x:
       new_list.append(x)

Just written in a single, compact statement.

# ? Nov 10, 2011 16:48

No Safe Word: Feb 26, 2005

taqueso posted:

I don't think you can modify the list you are looping through.

You can but I don't know of a good reason ever to do so.

# ? Nov 10, 2011 16:59

taqueso: Mar 8, 2004

No Safe Word posted:

You can but I don't know of a good reason ever to do so.

Ya, I really should have used the word "should" in place of "can".

# ? Nov 10, 2011 17:26

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

Is there a reason that this:

code:

'a' > 1

is OK - with respect to language ideology, I mean. I'm so used to the interpreter yelling at me when I mix types accidentally that I was really surprised by a bug in my code where I mixed up a couple of similarly named variables and ended up with the above situation. Surprised it ran without throwing exceptions, that is. Also, it feels wrong that that would not be an error in a language where I have to explicitly convert ints to strings when I .join() them (for example).

Can someone more in tune with Python culture tell me if that's a wart or an intentional feature?

# ? Nov 10, 2011 17:38

tef: May 30, 2004; -> some l-system crap ->

Munkeymon posted:

Can someone more in tune with Python culture tell me if that's a wart or an intentional feature?

wart, no longer a feature in python 3

quote:

Python 3.0 has simplified the rules for ordering comparisons:

The ordering comparison operators (<, <=, >=, >) raise a TypeError exception when the operands don�t have a meaningful natural ordering. Thus, expressions like 1 < '', 0 > None or len <= len are no longer valid, and e.g. None < None raises TypeError instead of returning False. A corollary is that sorting a heterogeneous list no longer makes sense � all the elements must be comparable to each other. Note that this does not apply to the == and != operators: objects of different incomparable types always compare unequal to each other.

# ? Nov 10, 2011 17:49

tef: May 30, 2004; -> some l-system crap ->

fnordcircle posted:

What's the best way to remove items with a certain character in them from a list?

Internally, python keeps a counter as it progresses through the list. When you remove the current item, the counter isn't changed, so it effectively skips one item in the list. This is why that one item keeps slipping through. Do not rely on this behaviour.

If you mutate the list, you have to keep an explicit count yourself, and only increment it if you don't remove the current item.

code:

pos = 0
while pos < len(some_list):
    if '-' in some_list[pos]:
        del some_list[pos]
    else:
        pos+=1

edit: you can avoid this by iterating through the list backwards :q:

# ? Nov 10, 2011 17:53

fnordcircle: Jul 7, 2004; PTUI

tef posted:

Internally, python keeps a counter as it progresses through the list. When you remove the current item, the counter isn't changed, so it effectively skips one item in the list. This is why that one item keeps slipping through. Do not rely on this behaviour.

If you mutate the list, you have to keep an explicit count yourself, and only increment it if you don't remove the current item.
code:
pos = 0
while pos < len(some_list):
    if '-' in some_list[pos]:
        del some_list[pos]
    else:
        pos+=1
edit: you can avoid this by iterating through the list backwards

Awesome, that's what I was thinking was happening. That 'for each in list' means 'each' is a pointer, in essence, so when I was using the remove method I was screwing up my list index. I dunno how close I was but that at least makes sense to me.

On the flip side I do modify another list by appending to it as I go along so I can sort it for real values when some of the values are nested. I haven't seen any problems like this with appending should I expect to?

Example:

code:

source = ["group1","host5","group2","host6"]
groupdict = {"group1":"host1,host2","group2":"host3,host4"
hostdict = {"host1":"192.168.67.1","host2":"192.168.67.2","host3":"192.168.67.3",
"host4":"192.168.67.4","host5":"192.168.67.5"}

for each in source:
      if each in groupdict.keys():
           temp_source = groupdict[each]
           temp_source = temp_source.split(",")
           for temp_addresses in temp_source:
                 source.append(tmep_addresses)
      
      if each in hostdict.keys():
           final_source_list.append(each)

The idea here is to get a final list of all actual IP values. I haven't run into any problems that I can see doing it this way but I just want to be sure it's copacetic that I'm adding to the source list in the first for each until I finally have parsed every value.

# ? Nov 10, 2011 18:47

tef: May 30, 2004; -> some l-system crap ->

fnordcircle posted:

Example:

code:

source = ["group1","host5","group2","host6"]
groupdict = {"group1":"host1,host2","group2":"host3,host4"
hostdict = {"host1":"192.168.67.1","host2":"192.168.67.2","host3":"192.168.67.3",
"host4":"192.168.67.4","host5":"192.168.67.5"}

for each in source:
      if each in groupdict.keys():
           temp_source = groupdict[each]
           temp_source = temp_source.split(",")
           for temp_addresses in temp_source:
                 source.append(tmep_addresses)
      
      if each in hostdict.keys():
           final_source_list.append(each)

The idea here is to get a final list of all actual IP values.

Some quick style tips - as a guideline, not a doctrine.

Good names should reflect intent, not implementation. Avoid suffixes like dict - i.e prefer groups/hosts over group_dict/hosts_dict.

Use the right data structures. Don't store lists of data within strings, but of lists of strings.

i.e groups = {'group1': [ "host1", "host2"] }

(There are a lot of arguments about naming - with python, just use PEP8 or /lots/ of people will whinge at you)

Do one thing at a time - Instead of mutating the list and implicitly not breaking the iterator, collect the names first, before doing the lookup. (and if you must mutate a list in a for-loop, use *explicit* iteration, rather than relying on internal behaviour)

Programming is not a game of outwitting the compiler, but an exercise of explaining something to other programmers, as well as the computer. It isn't worth confusing either of them - write dumb code that is very straight forward and you will be surprised how little you come back to it. Write clever things and you waste immense amount of time coming back to fix edge and boundary conditions.

Relying on append not to break is a bad idea. It makes your code harder to follow.

code:

my_hosts = set()

for group_or_user in source:
    if group_or_user in groups:
        my_hosts.update(groups[group_or_user])
    if group_or_user in hosts:
        my_hosts.add(entry)

addresses = set()
for hosts in source_hosts:
        addresses.add(hosts[host])

sorted_addresses = list(sorted(addresses))

We use a set to represent the hostnames we're interested in, because we don't want to have duplicate entries.

# ? Nov 10, 2011 19:11

fnordcircle: Jul 7, 2004; PTUI

Thanks for all the feedback, Tef, that's some awesome stuff there. Not all of it fully sinks in on the first read but I'm sure I'll understand it eventually.

# ? Nov 10, 2011 20:38

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

tef posted:

wart, no longer a feature in python 3

Excellent - thanks

# ? Nov 10, 2011 22:27

Modern Pragmatist: Aug 20, 2008

I have 10,000 objects or so and I basically want to group them into categories based on the value of a specific property. The property that I care about is a list of 6 floats. My initial thought on how to approach this was to create a dict() where the keys are the 6 element list, but this isn't possible. What is the best approach for this?

code:

a.prop = [0 1.1 2.2 3.3 4.4 5.5]
b.prop = [5.5 4.4 3.3 2.2 1.1 0]
c.prop = [0 1.1 2.2 3.3 4.4 5.5]

# ideal result:
[[a,c],[b,]]

# ? Nov 11, 2011 17:47

TasteMyHouse: Dec 21, 2006

Lists are mutable and thus cannot be used as keys. Tuples, however, can.

code:

a_key=(0.0,1.1,2.2,3.3,4.4,5.5)
b_key=(5.5,4.4,3.3,2.2,1.1,0.0)
dict={a_key:"a!",b_key:"b!"}
print dict[a_key]

> a!

Or whatever.

# ? Nov 11, 2011 18:03

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 09:54

Modern Pragmatist: Aug 20, 2008

TasteMyHouse posted:

Lists are mutable and thus cannot be used as keys. Tuples, however, can.
code:
a_key=(0.0,1.1,2.2,3.3,4.4,5.5)
b_key=(5.5,4.4,3.3,2.2,1.1,0.0)
dict={a_key:"a!",b_key:"b!"}
print dict[a_key]

> a!
Or whatever.

Oh awesome. That makes perfect sense now that I think about it. Thank you.

# ? Nov 11, 2011 18:05

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »