Scaevolus posted:

Python 2.6b1 was released, changelog here

Things of note:

- Issue #2831: enumerate() now has a ``start`` argument.
- Issue #2138: Add factorial() the math module.
- Added the multiprocessing module, PEP 371.

Bonus posted:

Yeah. Also watch that you don't fall into an infinite loop within the __getattr__ or __setattr__ methods. Manipulate the __dict__ directly.

Bonus posted:

Yeah. Also watch that you don't fall into an infinite loop within the __getattr__ or __setattr__ methods. Manipulate the __dict__ directly.

HatfulOfHollow posted:

Is there any way to call a function that expects a number arguments with a list of those arguments without resorting to this:

code:

import cx_Oracle as oracle
date = [int(x) for x in time.strftime('%Y,%m,%d,%H,%M,%S').split(',')]
[b]timestamp = oracle.Timestamp(date[0],date[1],date[2],date[3],date[4],date[5])[/b]

The above works fine, it just seems really inelegant.

And yes, I am aware of the many alternate methods to get a timestamp from the DBAPI, this just came up in something I was working on and I wanted to know if there was a pretty way to call the oracle.Timestamp() function without doing "date[0],date[1],..."

tripwire posted:

m0nk3yz, I know you worked on pyprocessing so perhpas you can answer a stupid question. I'm trying to parallelize a serial python program; in the program there is a list of "chromosome" objects which are supposed to get a unique id when they are instanced. The relevant code for it looks like this:

code:

class Chromosome(object):
    _id = 0

    def __init__(self, some arguments..):
        self._id = self.__get_new_id()
        .
        .
        .
    id = property(lambda self: self._id)

    @classmethod
    def __get_new_id(cls):
        cls._id += 1
        return cls._id

My question is, what is the best/most elegant way to make sure that concurrent processes don't produce id's which collide with any other process?

Since new chromosomes are going to be generated constantly in a loop, should I give up just incrementing a counter to generate unique id's? Would it be better to rework the __get_new_id function such that it generates a pseudo-random value with a very low chance of collision? Would I be better off trying to make sure each concurrent process only access a central id factory in synchronization? There is probably a really simple solution but I'm too dumb to figure it out and I don't want to rush into trying to code without knowing what I'm trying to accomplish.

m0nk3yz posted:

You have a couple of ways of doing this - you have already touched on one which is to change __get_new_id to generate pseudo-random integers for each of the objects and basically making it a shared objects with locks/etc (see processing.RLock/etc). In my code, I generate empty objects with unique IDs ahead of time (think "BlankObject.id") and pump a good amount of them into a shared queue which I then pass into the processes (processing.Queue).

The bad thing about my approach is that you could run out of blank objects, so I have to keep a producer in the background pumping in new objects so the workers generating the objects always have new blanks. In your case it does make sense to make a new shared object which essentially generates the unique IDs. In my case, I could also just fill a queue with unique objects - or subclass processing.Queue and overwrite the get() method to generate batches of IDs if the queue is empty.

A simpler approach is to pick a seed and pass it to the child processes so they can in turn pass it to a random call - you have a pretty low chance of collisions with random especially if in your ID you include some other attribute of the object. In another implementation, each object I spawned used a random number (generated from a seed) + 2-3 other attributes of the object being created.

A few things to think about : When using processing.Queue, you pay a serialization and deserialization cost for things coming in and out of the Queue. The same goes for the cost of lock acquisition and releasing, it really depends on where you want to take the hit. If you go with the "shared object generating the ids" approach - that object is going to have to keep an ever-growing list of IDs it's handed out so that it really does ensure that there isn't a conflict.

Random thought: use a seed passed to random and the machine-time (time.time) to generate the IDs

tripwire posted:

My question is, what is the best/most elegant way to make sure that concurrent processes don't produce id's which collide with any other process?

Bonus posted:

Milde: Ah, yeah, right, I always consfuse get/setattr with get/setattribute. Those are really awkward names.

HatfulOfHollow: try timestamp = oracle.Timestamp(*date). You can also call keyword arguments in that way by using ** with any dict-like object.

e: beaten

ATLbeer posted:

Can someone show me the Python docs on the *array, **array information? I know what it does but, I've never actually seen the docs for it. It's also hard to google "python *"

Bozart posted:

Why not just use the incremented IDs, and have each process remember its process ID. If you want to combine the results, you can create unique IDs then from the PID and innovation number. Also, I am not sure if NEAT would work if the innovation numbers were not strictly increasing, but I could be wrong. I would mainly just be way too lazy to go through the code and see what I would have to change to use random IDs instead of incremented ones, but to each their own.

Sock on a Fish posted:

Is there a way to capture what gets printed to the screen after having Python execute a console command using os.system()? I'm trying to write a script that will start a Windows service, check for success, and then start the service again if needed. NET START returns the same result code for both when the service is already started and when it just plain fails to start. However, the text that it prints to the console is different depending on whether the service fails or is already started.

JoeNotCharles posted:

Use the subprocess module instead of system.

HatfulOfHollow posted:

Is there any way to call a function that expects a number arguments with a list of those arguments without resorting to this:

code:

import cx_Oracle as oracle
date = [int(x) for x in time.strftime('%Y,%m,%d,%H,%M,%S').split(',')]
[b]timestamp = oracle.Timestamp(date[0],date[1],date[2],date[3],date[4],date[5])[/b]

The above works fine, it just seems really inelegant.

And yes, I am aware of the many alternate methods to get a timestamp from the DBAPI, this just came up in something I was working on and I wanted to know if there was a pretty way to call the oracle.Timestamp() function without doing "date[0],date[1],..."

Sock on a Fish posted:

I tried that, but I can't find any way to get those error messages. stderr and stdout are both null.

tef posted:

I've not been having with Popen recently, maybe it is how you are calling it:

code:

import subprocess
p = subprocess.Popen(command,
                     stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE,
                     bufsize=1)

print p.stdout.readlines()
print p.stderr.readlines()

Sock on a Fish posted:

Score! I thought those extra arguments were the defaults and was just specifying my command. Thanks!

m0nk3yz posted:

Good idea - also, each process and thread inside of python does support get name / get id calls - you can even name them anything you want (say, a unique seed for each one which allows you to make unique names for each process namespace)

tripwire posted:

tripwire posted:

Am I better off ditching the current crossover function and trying to figure out a more parallel friendly version?

Bozart posted:

Just from an evolutionary algorithm perspective (which, once again, I am painfully new at) maybe you could just run the program through (for example) 100 generations in 10 different threads, and then take random species from each thread to generate 10 new threads, and repeat. Kind of like when one species invades a new ecology?

Then again this topic lends itself to very nice analogies which don't translate to useful improvements in the algorithms.

Zombywuf posted:

There is something like this called islanding. It can be useful to prevent your population getting stuck in local minema. Normally you would swap a few individuals between islands every x generations.