Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
floppo
Aug 24, 2005
I am looking to build a simple website. I am a new PhD student and I would like a place to share my publications, working papers, and perhaps eventually some d3.js or bokeh outputs. A newsfeed for updates would also be nice.

I use Python in my research but don't know much about flask, django and other frameworks. There is also definitely a ceiling to how much I want to learn about web dev: I am just looking to make a simple but nice site to better connect with colleagues in my field.

What's a good tutorial to get this kind of thing up and running quickly?

Adbot
ADBOT LOVES YOU

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

floppo posted:

I am looking to build a simple website. I am a new PhD student and I would like a place to share my publications, working papers, and perhaps eventually some d3.js or bokeh outputs. A newsfeed for updates would also be nice.

I use Python in my research but don't know much about flask, django and other frameworks. There is also definitely a ceiling to how much I want to learn about web dev: I am just looking to make a simple but nice site to better connect with colleagues in my field.

What's a good tutorial to get this kind of thing up and running quickly?

It sounds like you don't need Python at all, just write HTML+CSS or use something like Squarespace (squarespace is really good).

If you really want to use Python, there's a bunch of static site generators, though I can't really recommend one over the other as I haven't used them much. Google "python static site generator".

Dominoes
Sep 20, 2007

Seconding a service like Squarespace. Python web frameworks take lots of time and effort to learn.

dreesemonkey
May 14, 2008
Pillbug
I'm completely new to python as of this afternoon. I downloaded python for windows and am attempting this example (http://docs.python-guide.org/en/latest/scenarios/scrape/) to scrape a webpage.

I attempt to install the lxml library and get an error and I'm not sure if it's due to a newer python version, privileges, whatever. Can anyone point me in the right direction?

code:
C:\WINDOWS\system32>pip install lxml
Collecting lxml
  Using cached lxml-3.4.4.tar.gz
Installing collected packages: lxml
  Running setup.py install for lxml
    Complete output from command c:\users\ddreese\appdata\local\programs\python\python35-32\python.exe -c "import setuptools, tokenize;__file__='C:\\Users\\ddreese\\AppData\\Local\\Temp\\pip-build-w36__epd\\lxml\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\ddreese\AppData\Local\Temp\pip-mcas0zqx-record\install-record.txt --single-version-externally-managed --compile:
    Building lxml version 3.4.4.
    Building without Cython.
    ERROR: b"'xslt-config' is not recognized as an internal or external command,\r\noperable program or batch file.\r\n"
    ** make sure the development packages of libxml2 and libxslt are installed **

    Using build configuration of libxslt
    running install
    running build
    running build_py
    creating build
    creating build\lib.win32-3.5
    creating build\lib.win32-3.5\lxml
    copying src\lxml\builder.py -> build\lib.win32-3.5\lxml
    copying src\lxml\cssselect.py -> build\lib.win32-3.5\lxml
    copying src\lxml\doctestcompare.py -> build\lib.win32-3.5\lxml
    copying src\lxml\ElementInclude.py -> build\lib.win32-3.5\lxml
    copying src\lxml\pyclasslookup.py -> build\lib.win32-3.5\lxml
    copying src\lxml\sax.py -> build\lib.win32-3.5\lxml
    copying src\lxml\usedoctest.py -> build\lib.win32-3.5\lxml
    copying src\lxml\_elementpath.py -> build\lib.win32-3.5\lxml
    copying src\lxml\__init__.py -> build\lib.win32-3.5\lxml
    creating build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\__init__.py -> build\lib.win32-3.5\lxml\includes
    creating build\lib.win32-3.5\lxml\html
    copying src\lxml\html\builder.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\clean.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\defs.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\diff.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\ElementSoup.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\formfill.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\html5parser.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\soupparser.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\usedoctest.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\_diffcommand.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\_html5builder.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\_setmixin.py -> build\lib.win32-3.5\lxml\html
    copying src\lxml\html\__init__.py -> build\lib.win32-3.5\lxml\html
    creating build\lib.win32-3.5\lxml\isoschematron
    copying src\lxml\isoschematron\__init__.py -> build\lib.win32-3.5\lxml\isoschematron
    copying src\lxml\lxml.etree.h -> build\lib.win32-3.5\lxml
    copying src\lxml\lxml.etree_api.h -> build\lib.win32-3.5\lxml
    copying src\lxml\includes\c14n.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\config.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\dtdvalid.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\etreepublic.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\htmlparser.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\relaxng.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\schematron.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\tree.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\uri.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\xinclude.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\xmlerror.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\xmlparser.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\xmlschema.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\xpath.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\xslt.pxd -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\etree_defs.h -> build\lib.win32-3.5\lxml\includes
    copying src\lxml\includes\lxml-version.h -> build\lib.win32-3.5\lxml\includes
    creating build\lib.win32-3.5\lxml\isoschematron\resources
    creating build\lib.win32-3.5\lxml\isoschematron\resources\rng
    copying src\lxml\isoschematron\resources\rng\iso-schematron.rng -> build\lib.win32-3.5\lxml\isoschematron\resources\rng
    creating build\lib.win32-3.5\lxml\isoschematron\resources\xsl
    copying src\lxml\isoschematron\resources\xsl\RNG2Schtrn.xsl -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl
    copying src\lxml\isoschematron\resources\xsl\XSD2Schtrn.xsl -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl
    creating build\lib.win32-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1
    copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_abstract_expand.xsl -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1
    copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_dsdl_include.xsl -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1
    copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_schematron_message.xsl -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1
    copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_schematron_skeleton_for_xslt1.xsl -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1
    copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\iso_svrl_for_xslt1.xsl -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1
    copying src\lxml\isoschematron\resources\xsl\iso-schematron-xslt1\readme.txt -> build\lib.win32-3.5\lxml\isoschematron\resources\xsl\iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    c:\users\ddreese\appdata\local\programs\python\python35-32\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    error: [WinError 2] The system cannot find the file specified

    ----------------------------------------
Command "c:\users\ddreese\appdata\local\programs\python\python35-32\python.exe -c "import setuptools, tokenize;__file__='C:\\Users\\ddreese\\AppData\\Local\\Temp\\pip-build-w36__epd\\lxml\\setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:\Users\ddreese\AppData\Local\Temp\pip-mcas0zqx-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\ddreese\AppData\Local\Temp\pip-build-w36__epd\lxml

SurgicalOntologist
Jun 17, 2004

You're going to have trouble compiling packages on Windows, just use Anaconda. It will solve this and many, many other problems you will encounter running Python on Windows.

QuarkJets
Sep 8, 2008

Yeah, Anaconda is the gold standard in self-contained Python package environments and it makes the entire "Python in Windows" experience waaaaaaay better.

dreesemonkey
May 14, 2008
Pillbug
Ok thanks guys, I downloaded it and I'm messing about as we speak.

dreesemonkey
May 14, 2008
Pillbug
Success! It ain't pretty, but I have it scraping amazon's website and logging the best seller ranking for my sister's book she just released (as well as an older one) and logging it to a file.



Tomorrow I'm going to try and get it to write the results to a database via ODBC maybe and see if I can setup the multiple products in an array to run it looped in one script.

Cingulate
Oct 23, 2012

by Fluffdaddy
I'm teaching a Python class this semester.

:downs:

Nippashish
Nov 2, 2005

Let me see you dance!

Hammerite posted:

This is an example of what I was talking about where whatever these object types are have had the behaviour of & customised to do something the library creator thought would be useful. Now, I don't know anything about this library, but it strikes me as bad. I would prefer to see code that relies less on the reader being intimately familiar with how the & and >= operators are implemented for this library's data types, at the cost of most likely being more verbose.

Matrices and vectors are pretty much the canonical example of the place where operator overloading makes sense.

German Joey
Dec 18, 2004

Nippashish posted:

Matrices and vectors are pretty much the canonical example of the place where operator overloading makes sense.

yeah, that's a good one. I also really like them in ORMs, e.g. for doing Q operations in Django. (https://docs.djangoproject.com/en/1.8/topics/db/queries/#complex-lookups-with-q-objects)

Other than for very obvious thinigs like this though, I'm also in the "loathes overloaded operators" camp.

edit: Now that I think about, both examples are cases where overloaded operators are used in a closed system. That is, *all* possible operators on these objects are overloaded to have a similar mode of behavior.

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



dreesemonkey posted:

Tomorrow I'm going to try and get it to write the results to a database via ODBC maybe and see if I can setup the multiple products in an array to run it looped in one script.

Consider using https://docs.python.org/3.5/library/sqlite3.html It's a great starter database for small to medium sized applications and it comes with Python.

dreesemonkey
May 14, 2008
Pillbug

Munkeymon posted:

Consider using https://docs.python.org/3.5/library/sqlite3.html It's a great starter database for small to medium sized applications and it comes with Python.

I have an existing database platform that is my day job and it would make it super easy for me to manipulate the data/create reports/charts/etc. But I'll keep it in mind.

SurgicalOntologist
Jun 17, 2004

Is it possible to remove the leading zero when string formatting a float guaranteed to be between -1 and 1? I always look into this when I'm annotating correlations, and I always give up and do it in two steps. Now, though, I'm using a library that has you pass in the formatting string... so am I out of luck?

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

SurgicalOntologist posted:

Is it possible to remove the leading zero when string formatting a float guaranteed to be between -1 and 1? I always look into this when I'm annotating correlations, and I always give up and do it in two steps. Now, though, I'm using a library that has you pass in the formatting string... so am I out of luck?

http://stackoverflow.com/questions/10303797/print-floating-point-values-without-leading-zero

Looking over the solutions posted there, it looks like you might be out of luck. Ask your library author/vendor.

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



dreesemonkey posted:

I have an existing database platform that is my day job and it would make it super easy for me to manipulate the data/create reports/charts/etc. But I'll keep it in mind.

You're probably good, then. I thought maybe you were talking about using ODBC with Access and figured I'd point out a good alternative.

KernelSlanders
May 27, 2013

Rogue operating systems on occasion spread lies and rumors about me.

SurgicalOntologist posted:

Is it possible to remove the leading zero when string formatting a float guaranteed to be between -1 and 1? I always look into this when I'm annotating correlations, and I always give up and do it in two steps. Now, though, I'm using a library that has you pass in the formatting string... so am I out of luck?

When all else fails, regex.

Python code:
x = 0.31415925
re.sub(r'0\.(\d+)', r'.\1', "%.3f" % x)  # returns '.314'

Melian Dialogue
Jan 9, 2015

NOT A RACIST
Stupid question, Im new to Python and programming so please bear with me. I'm using Anaconda with the Spyder IDE that came with it.

When I type something that has a paranthesis, frequently it autofills in the closing paranthesis. However, when Im typing once I get to that paranthesis, I can't just hit space or enter to move past it, I have to reach across my keyboard and use the arrow buttons. Is there some sort of shortcut (like alt or tab or something) where it lets me skip the word?

In addition, is there another keyboard shortcut that lets me highlight a word or a variable? like say I have something like data['USFREQMOC'], and I want to just be able to select all of 'USFREQMOC' and delete it, so I can type something else in, instead of highlighting it or backspace or delete buttoning it.

I'm sorry this is really stupid but I feel like these little things are slowing me down a lot.

lord of the files
Sep 4, 2012

currently none of those features exist in spyder ide to the best of my knowledge. the ability to highlight and delete a segment should be in the 2.4 milestone release though.

https://github.com/spyder-ide/spyder/issues/729

lord of the files fucked around with this message at 18:45 on Oct 30, 2015

Cingulate
Oct 23, 2012

by Fluffdaddy

Melian Dialogue posted:

I have to reach across my keyboard and use the arrow buttons
Do you have really tiny hands/only one hand?

Munkeymon
Aug 14, 2003

Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.



Melian Dialogue posted:

When I type something that has a paranthesis, frequently it autofills in the closing paranthesis. However, when Im typing once I get to that paranthesis, I can't just hit space or enter to move past it, I have to reach across my keyboard and use the arrow buttons. Is there some sort of shortcut (like alt or tab or something) where it lets me skip the word?

Sometimes editors that do that will skip the auto-generated right-paren with... right-paren. What's the point? Dunno! That's why I usually turn that useless bullshit off.

Melian Dialogue
Jan 9, 2015

NOT A RACIST

Cingulate posted:

Do you have really tiny hands/only one hand?

No? Just that, on a regular QWERTY keyboard my hands are positioned perfectly not needing to move on the left-most part of the keyboard. Instead of being able to press something like TAB with my pinky, I have to move my entire hand out of position to press the Right Arrow button. My fingers rest on ASDF and JKL; naturally as a home position.

I'm a relatively okay typer (80-90WPM), so I hate having to slow down completely because of a dumb little thing. Can I maybe map it to Alt or some other button I don't use ever on its own in Spyder?

Or should I just turn off that auto-generate close parenthesis thing? I mean its useful for typing things in sometimes, other times its not useful.

QuarkJets
Sep 8, 2008

Melian Dialogue posted:

No? Just that, on a regular QWERTY keyboard my hands are positioned perfectly not needing to move on the left-most part of the keyboard. Instead of being able to press something like TAB with my pinky, I have to move my entire hand out of position to press the Right Arrow button. My fingers rest on ASDF and JKL; naturally as a home position.

I'm a relatively okay typer (80-90WPM), so I hate having to slow down completely because of a dumb little thing. Can I maybe map it to Alt or some other button I don't use ever on its own in Spyder?

Or should I just turn off that auto-generate close parenthesis thing? I mean its useful for typing things in sometimes, other times its not useful.

You don't have to use the arrow key to go around it, you can just type ")" when you reach the closing parenthesis. That'll move your cursor to the other side of the auto-generated closing parenthesis.

QuarkJets
Sep 8, 2008

Melian Dialogue posted:

In addition, is there another keyboard shortcut that lets me highlight a word or a variable? like say I have something like data['USFREQMOC'], and I want to just be able to select all of 'USFREQMOC' and delete it, so I can type something else in, instead of highlighting it or backspace or delete buttoning it.

Try ctrl-shift-arrow, or double-clicking the word if you'd rather use a mouse

Melian Dialogue
Jan 9, 2015

NOT A RACIST

QuarkJets posted:

You don't have to use the arrow key to go around it, you can just type ")" when you reach the closing parenthesis. That'll move your cursor to the other side of the auto-generated closing parenthesis.

:bang: Duh, Thats pretty obvious. Thanks, Im still pretty new to this.

QuarkJets posted:

Try ctrl-shift-arrow, or double-clicking the word if you'd rather use a mouse

Thanks! This sounds like what I need

Cingulate
Oct 23, 2012

by Fluffdaddy
I know iterating over dict.values(), keys() or items() is not guaranteed to result in any specific order, and I assume the order may change if I change the keys of the dict.

But is the order guaranteed to stay the same when I iterate over any of these multiple times? Potentially 1. iterating over different things (e.g. once over keys, once over items), 2. change values in between?

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Cingulate posted:

I know iterating over dict.values(), keys() or items() is not guaranteed to result in any specific order, and I assume the order may change if I change the keys of the dict.

But is the order guaranteed to stay the same when I iterate over any of these multiple times? Potentially 1. iterating over different things (e.g. once over keys, once over items), 2. change values in between?

Why do you feel that you might need it to stay in the same order? What are you trying to do?

I don't know whether the order will be the same every time (assuming no changes to the keys of the dictionary). I suspect that the answer to that question will turn out to be "it happens that in CPython it will be the same every time, but nothing guarantees that that won't change in the future".

If it is the same every time, then I would have thought changing the values should make no difference to that. The order is probably related somehow to the hash values of the keys.

You're presumably aware that if you create the same dictionary in two different Python sessions, the order in which items are iterated will be different, because hashes are influenced by a random element that is per-session.

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Cingulate posted:

I know iterating over dict.values(), keys() or items() is not guaranteed to result in any specific order, and I assume the order may change if I change the keys of the dict.

But is the order guaranteed to stay the same when I iterate over any of these multiple times? Potentially 1. iterating over different things (e.g. once over keys, once over items), 2. change values in between?

Generally worrying about order in a dict is a code smell for something that needs to be refactored. That said, if you really need it

Python code:

from collections import OrderedDict

Feral Integral
Jun 6, 2006

YOSPOS

Hey I'm messing around with some twisted perspective broker examples and I've come to a point where I have 3 basic pbs - one "client", one "server", and one "resource server". The "server" makes async remote calls on the resource server for itself and on behalf of the client. In this kind of situation, should I be doing a reactor.connectTCP("localhost", 8801, self.factory) type thing only once during construction of the pb object or am i supposed to be just throwing that poo poo around nilly willy (like anytime a remote method is called)?

Hughmoris
Apr 21, 2007
Let's go to the abyss!
I'm starting to use BeautifulSoup a little bit, and am stumped on a certain query. I'm trying to count the number of dropdowns in a select tag:
code:
<div>
    <label for="bySeason">Season:</label>
    <select id="bySeason" tconst="tt1826940" class="current">
      <option  value="1">
        1
      </option>
      <option  value="2">
        2
      </option>
      <option selected="selected" value="3">
        3
      </option>
      <option  value="4">
        4
      </option>
      <option  value="5">
        5
      </option>
    </select>
  </div>
Any ideas on how to go about getting the count? Essentially, its a dropdown box to select the season of a TV show. I'd like to count how many seasons there are.

*Here is the page I'm scraping: http://www.imdb.com/title/tt1826940/episodes?season=3

Hughmoris fucked around with this message at 04:12 on Nov 3, 2015

Hargrimm
Sep 22, 2011

W A R R E N

Hughmoris posted:

I'm starting to use BeautifulSoup a little bit, and am stumped on a certain query. I'm trying to count the number of dropdowns in a select tag:
code:
<div>
    <label for="bySeason">Season:</label>
    <select id="bySeason" tconst="tt1826940" class="current">
      <option  value="1">
        1
      </option>
      <option  value="2">
        2
      </option>
      <option selected="selected" value="3">
        3
      </option>
      <option  value="4">
        4
      </option>
      <option  value="5">
        5
      </option>
    </select>
  </div>
Any ideas on how to go about getting the count? Essentially, its a dropdown box to select the season of a TV show. I'd like to count how many seasons there are.

*Here is the page I'm scraping: http://www.imdb.com/title/tt1826940/episodes?season=3

You can just use a method that returns a collection of the select's child elements, and then count that.

code:
len(bs.find("select", {"id": "bySeason"}).findAll("option"))

or 

len(bs.find("select", {"id": "bySeason"}).findChildren())

Hughmoris
Apr 21, 2007
Let's go to the abyss!

Hargrimm posted:

You can just use a method that returns a collection of the select's child elements, and then count that.

code:
len(bs.find("select", {"id": "bySeason"}).findAll("option"))

or 

len(bs.find("select", {"id": "bySeason"}).findChildren())

I'll give this a try, thanks!

Death Chicken
Aug 15, 2015

Feral Integral posted:

Hey I'm messing around with some twisted perspective broker examples and I've come to a point where I have 3 basic pbs - one "client", one "server", and one "resource server". The "server" makes async remote calls on the resource server for itself and on behalf of the client. In this kind of situation, should I be doing a reactor.connectTCP("localhost", 8801, self.factory) type thing only once during construction of the pb object or am i supposed to be just throwing that poo poo around nilly willy (like anytime a remote method is called)?

you should be able to open the connection once and it will be running in the background, like so:

code:
from twisted.spread import pb
from twisted.internet import reactor

class Client(object):
    factory= pb.PBClientFactory()
    def connect(self):
        reactor.connectTCP("localhost", 8801,  self.factory)
        self.factory.getRootObject().addCallback(self.send)
        
    def send(self, message):
        message.callRemote("echo", "goats").addCallback(self.receive)

    def receive(self, message):
        print "received message: ", message

if __name__ == '__main__':
    Client().connect()
    reactor.run()
once the connection is open and callbacks are registered, it will run until you call "reactor.stop".

Jose Cuervo
Aug 25, 2004
I have a simulation where I run multiple replications of a given policy (the input parameters that define the policy are fixed, but the replications have different results due to the randomness in the simulation). Currently I run the simulation replications in serial, one after the other. However, I would like to take advantage of the fact that the replications are independent and that I have multiple CPUs on my computer. To do this, I am trying to use the "Embarrassingly parallel for loops" part of the joblib lirbrary, which itself uses the multiprocessing library.

The overall structure of my code is the following:
Python code:
def main():
	Set parameters
	simulate_current_policy(seed_range)

def simulate_current_policy(seed_range):
	for the_seed in seed_range:
		single_simulation_replication()
The "Set parameters" line refers to initializing the various parameters that define the policy, and the declarations for all these parameters are contained in constants.py, where they are all initialized to None. Some example lines from "Set parameters" is
Python code:
const.SCENARIO_SETTING = 2
const.INIT_CONDITION_SEED = 28723
const.REP_START_INDEX = 101
const.STOP_TIME = 30
The function single_simulation_replication is written so that it returns a list with all the relevant simulation information in it. Based on the documentation on the joblib page, I have rewritten the for loop as follows:
Python code:
import joblib

parallel_results = joblib.Parallel(n_jobs=2)(joblib.delayed(single_simulation_replication)(the_seed) for the_seed in seed_range)
This code does run, and produces the same results as the non-parallel code when n_jobs is set to 1. However, when n_jobs is set to 2 or more I get the following error:
code:
     65     elif const.SCENARIO_SETTING == 1:
     66         sim_ITLs, sim_FTLs, sim_WTLs, sim_movers = trial_data.load_scenario_1()
     67     elif const.SCENARIO_SETTING == 2:
     68         sim_ITLs, sim_FTLs, sim_WTLs, sim_movers = trial_data.load_scenario_2()
     69     else:
---> 70         assert False, 'Unknown SCENARIO_SETTING: %s' % const.SCENARIO_SETTING
AssertionError: Unknown SCENARIO_SETTING: None
inside the single_simulation_replication function. My interpretation of this error is that the parallel process does not KNOW that const.SCENARIO_SETTING (although initialized as None) was already set to 2 in main(). Is this what is happening, and if so, why? Second, what can I do to overcome this issue?

Nippashish
Nov 2, 2005

Let me see you dance!

Jose Cuervo posted:

inside the single_simulation_replication function. My interpretation of this error is that the parallel process does not KNOW that const.SCENARIO_SETTING (although initialized as None) was already set to 2 in main(). Is this what is happening, and if so, why? Second, what can I do to overcome this issue?

It doesn't know about the changes to const because its running in a different process and the parameters were never set in that process. The solution is to pass the constants as a function arguments instead of having it pull them in from the ambient environment. I think you can pass the const module, but really you should put the parameters in a dict and pass that because global variables are bad mojo.

e: it works with 1 job because joblib doesn't use multiple processes unless you request more than one.

Jose Cuervo
Aug 25, 2004

Nippashish posted:

It doesn't know about the changes to const because its running in a different process and the parameters were never set in that process. The solution is to pass the constants as a function arguments instead of having it pull them in from the ambient environment. I think you can pass the const module, but really you should put the parameters in a dict and pass that because global variables are bad mojo.

e: it works with 1 job because joblib doesn't use multiple processes unless you request more than one.

Yeah I knew using global variables was not good practice and it has come to bite me in the butt now. Thanks for the help and solution to overcome the issue.

Proteus Jones
Feb 28, 2013



https://www.continuum.io/blog/company-blog/anaconda-24-release

PyCharm 5 was released this week as well.

It's like Christmas :)

Dominoes
Sep 20, 2007

flosofl posted:

https://www.continuum.io/blog/company-blog/anaconda-24-release

PyCharm 5 was released this week as well.

It's like Christmas :)
Python 3.5 support's nice. It's weird they upgraded their Qt, but are still on v4.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

What exactly does it mean by 3.5 support? I've been making 3.5 environments for awhile now...

Adbot
ADBOT LOVES YOU

Dominoes
Sep 20, 2007

Native; non virtual.

  • Locked thread