Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
evensevenone
May 12, 2001
Glass is a solid.
Think about what you want in that case. If you're planning to store a configuration file or data file in the same place as the script, it might not make sense on some platforms. That folder may not even be writable for security reasons. Chances are the home directory might make more sense.

Adbot
ADBOT LOVES YOU

Dominoes
Sep 20, 2007

evensevenone posted:

Think about what you want in that case. If you're planning to store a configuration file or data file in the same place as the script, it might not make sense on some platforms. That folder may not even be writable for security reasons. Chances are the home directory might make more sense.
This gets into whether I'm doing the overall structure/packaging correctly. Both of the programs I'm working on save and load information from multiple files. I plan to release this one soon, initially for Windows. (The other one's more complicated, with multiple modules, but it's mostly for personal use) How do you recommend packaging it and organizing the files?

What I'm working with:
-Single python script
-Two configuration files (YAML)
-One databasish file (YAML)
-One icon
-A temporary directory used for downloading and merging PDF files, prior to loading them to a user-specified save directory (QT dialog).

I plan on distributing the program as a standalone folder, where the user can create a shortcut to the program and place it wherever, and move the whole folder wherever. It's a simple program, so I feel an installer's overkill.

I'm open to restructuring the way I'm doing this, since I don't know wtf I'm doing.

Where would you recommend placing the config files and temporary directory? I feel like keeping it together's good for portability.

Dominoes fucked around with this message at 00:46 on May 20, 2013

QuarkJets
Sep 8, 2008

You want os.path.abspath(os.path.dirname(__file__)), that will always give you the full directory path for that file's location no matter where you're calling it from

But it might make more sense to create and use a directory in the person's Documents folder, and it might also make sense to allow the user to set the temp directory path

If this is causing a crash with cx_freeze, then I'm sure that others have encountered the same problem and you might try googling for the issue

QuarkJets fucked around with this message at 03:33 on May 20, 2013

Dominoes
Sep 20, 2007

Found a solution on Stack Overflow

Uses a workaround where you locate a dummy module's directory, which is the same as the one it's in. Seems to find the correct directory, and not crash the frozen program. Quark, the solution you posted is likely to give the same glitch in cx_freeze; I think it's the use of __file__ it doesn't like.

Am I correct assuming using the documents folder for config files would require using an installer? I don't want to leave pieces of the program lying around a user's drive without a way to cleanly uninstall it. The program provides a quick way to select, download and merge multiple documents from the FAA's website that are published each month. Since the program's goal is to save time and effort, I'm trying to keep things as simple as possible for the user. I do let the user select the directory to download the final product, and it defaults to their Home/Downloads folder.

Dominoes fucked around with this message at 03:43 on May 20, 2013

QuarkJets
Sep 8, 2008

If the user selects the destination of the final product, then why do you need to know where this module is located?

evensevenone
May 12, 2001
Glass is a solid.
Use the tempfile module for temporary files. It'll handle creating a temp file/folder in a safe, user-writable place, not overwriting anything, and cleaning it up when done.

duck monster
Dec 15, 2004

Help me i'm trapped in a php hole and i cant get out.

:(

Dominoes
Sep 20, 2007

QuarkJets posted:

If the user selects the destination of the final product, then why do you need to know where this module is located?
I currently have the configuration files and temporary directory in the program's folder, so I need to be able to determine its path. The user selects where to put the generated PDF(s). (Don't need the temp directly if they go for the separate files option, since it can download the files directly to the specified folder.)

I'd like to avoid having the user set the temporary directory; I'd like for this to be as simple and user-friendly as possible. "I just want to download approach plates, what's a temporary directly, and why do I have to set one? I don't know where to put it."

The module locator workaround I got from Stack works by checking if the program's frozen. If so, uses dirname(sys.executable), if not, uses dirname(__file__).


evensevenone posted:

Use the tempfile module for temporary files. It'll handle creating a temp file/folder in a safe, user-writable place, not overwriting anything, and cleaning it up when done.
Cool, I'll look into it.

Dominoes fucked around with this message at 11:46 on May 20, 2013

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe
I wrote some code that involves searching through directories and doing stuff to any files found whose filenames match a regular expression. I'm interested to know whether I've done it "the right way" or what I should have done differently.

I wanted to provide the option to either (1) not search subdirectories, (2) search all subdirectories up to a specified depth, or (3) search all subdirectories regardless of depth. It seemed like the way you're supposed to navigate a file system is using os.walk(), so that's what I did.

http://pastebin.com/Dzy8mNPs

The relevant code is lines 202 through 258.

In particular, I'm not sure what to do if an IOError gets thrown as a result of something the function tries to do. I mean, at the moment they aren't caught, but with the errors that I'm raising myself I attach information about which file was being processed and how many files were previously processed. It would be nice to have escaping IOErrors have that information attached to them, too, but I don't know what's the "correct" way to do such a thing.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe
No-one answered my question but I think maybe I was overly complicating it. Anyway here's my question distilled into a more basic form.

Suppose I have a function that makes a change or a series of changes to the filesystem. Some of the changes it attempts to make might result in an exception being raised (for example, it might try to write a file somewhere, but Python might not have the necessary privileges to do so). If that happens, I want to just allow the caller to deal with the exception, so I allow it to propagate out of the function. But, by the time an exception is raised, the function might already have made some changes to the filesystem, and it would be nice to have a way of communicating back to the caller what those changes were. What is the best way to do this?

Haystack
Jan 23, 2005





Well, there's always the option of attaching something to the exception. You catch the exception, and then either raise your own custom exception or monkeypatch and re-raise the original exception.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Haystack posted:

Well, there's always the option of attaching something to the exception. You catch the exception, and then either raise your own custom exception or monkeypatch and re-raise the original exception.

See, both those things occurred to me, but I have no idea if there's a Correct Way of doing things that I don't know about because I'm a know-nothing newbie :kiddo:

It occurred to me that I could raise my own exception and have it say something like 'While writing file "%s" after writing %d files already, this happened: ' followed by the __str__ of the existing exception, or the __str__ preceded by the class name, as when the exception is printed. But for all I knew that might discard some useful information that is carried by the original exception.

It also occurred to me to interfere with the original exception, but then that might also have been frowned upon for reasons entirely unknown to me.

Dominoes
Sep 20, 2007

I noticed that Python modules seem portable - I was able to install a module on Ubuntu that I couldn't get working by using build/install, by copying and pasting the folder from my Windows install's site-libs folder. Why aren't they distributed as stand-alone folders? Was it an exception that this one (PyPdf2) worked by copy/paste? I'm asking because the python setup build/install method gives me errors most of the time.

Dominoes fucked around with this message at 03:07 on May 22, 2013

Dren
Jan 5, 2001

Pillbug
Hammerite I think the thing to do in your case is to store the state of the directory walk outside of the scope of the function doing the work. Something like this:

Python code:
import os

class WalkState(object):
    def __init__(self):
        self.processed = []

def do_thing_to_file(path):
    pass

def walk_and_do_thing(dirpath, func, context=None):
    for root, dirs, files in os.walk(dirpath):
        for filename in files:
            func(os.path.join(root, filename))
            if context is not None:
                context.processed.append(os.path.join(root, filename))

def main():
    walk_state = WalkState()
    try:
        walk_and_do_thing('/', do_thing_to_file, walk_state)
    except IOError as e:
        print str(e)
        print 'Processed files before error: %s' % ', '.join(walk_state.processed)

if __name__ == '__main__':
    main()
If you need to you could even do something like pickle the current state and load it when the app comes back up to allow the user to resume from where they left off.

I actually prefer writing processing pipelines like this as coroutines but that's opening a whole can of worms.

AntennaGeek
May 30, 2011

BigRedDot posted:

No, sorry I shouldn't have been so coy. I gave the Numpy tutorial.

.... I'm fairly convinced Numpy is going to change my life.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Dren posted:

Hammerite I think the thing to do in your case is to store the state of the directory walk outside of the scope of the function doing the work. Something like this:

...

If you need to you could even do something like pickle the current state and load it when the app comes back up to allow the user to resume from where they left off.

I actually prefer writing processing pipelines like this as coroutines but that's opening a whole can of worms.

Thanks, that's really interesting. I hadn't heard of coroutines before and they seem complicated to wrap one's head around, but interesting.

QuarkJets
Sep 8, 2008

Dominoes posted:

I noticed that Python modules seem portable - I was able to install a module on Ubuntu that I couldn't get working by using build/install, by copying and pasting the folder from my Windows install's site-libs folder. Why aren't they distributed as stand-alone folders? Was it an exception that this one (PyPdf2) worked by copy/paste? I'm asking because the python setup build/install method gives me errors most of the time.

A) Sometimes there is compiled code that needs to be installed in order for the Python modules to work. For instance, parts of NumPy use compiled C and FORTRAN code.

B) It's a lot easier to send out an RPM and have people install that than it is to give explicit instructions regarding where to place a stand-alone folder, or a set of stand-alone folders when it comes to larger projects. This also helps to ensure that most users will have a worry-free install process. If you use a Makefile, then that Makefile will probably come with some standard defaults and will at least help to minimize the pain.

C) Using a standard install procedure creates uniformity between systems, which makes development more straightforward. If I log onto server X, then ideally I shouldn't have to set my PYTHONPATH to some special area because an admin decided to install it in a different place than on my home server.

There are other reasons, too. On Ubuntu there's almost no reason to build/install Python modules yourself, just use apt-get unless you want to do something special

Dominoes
Sep 20, 2007

Thank you.

Foiltha
Jun 12, 2008

QuarkJets posted:

There are other reasons, too. On Ubuntu there's almost no reason to build/install Python modules yourself, just use apt-get unless you want to do something special

There are two very good reasons and they're called pip and virtualenv.

Sab669
Sep 24, 2009

So a buddy of mine is going back to college to get a BS in Comp Sci in a few months, and I have a really dumb question. He decided to start with the MIT lecture last night and I guess it starts off having him use Python. I don't know the first thing about Python or the Python Shell application it was having him use (Or any scripting language for that matter, I'm a newbie C# guy). He was having a problem trying to "understand" how he would actually use this in the real world. By that I mean, if he saved a file and ran it, a command window would pop up and immediately disappear, even though the script requests user input.

I also think part of it for him is he expects to be creating full blown desktop applications, but you need to learn to walk before you can run. Most day-to-day users would poop themselves if they had to deal with the command line in any way.

He was just texting me about it this morning as I was getting ready for work, so I didn't have a chance to sit down and try to learn anything, though I figured while he's taking this course I probably should too :)

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Sab669 posted:

So a buddy of mine is going back to college to get a BS in Comp Sci in a few months, and I have a really dumb question. He decided to start with the MIT lecture last night and I guess it starts off having him use Python. I don't know the first thing about Python or the Python Shell application it was having him use (Or any scripting language for that matter, I'm a newbie C# guy). He was having a problem trying to "understand" how he would actually use this in the real world. By that I mean, if he saved a file and ran it, a command window would pop up and immediately disappear, even though the script requests user input.

I also think part of it for him is he expects to be creating full blown desktop applications, but you need to learn to walk before you can run. Most day-to-day users would poop themselves if they had to deal with the command line in any way.

He was just texting me about it this morning as I was getting ready for work, so I didn't have a chance to sit down and try to learn anything, though I figured while he's taking this course I probably should too :)

Sounds like he is working on Windows and is running into a limitation of the Windows command line. Basically, when you run a command line application on Windows, it will close the window as soon as the program exits, which is inconvenient if you wanted to see what the program did.

If your program is waiting for user input then it will not close until it gets it (and goes on to exit), but probably your friend's newbie code contains a syntax error or hits some other error condition which causes an exception to be thrown, which isn't caught and causes the program to exit with a printed error message, which counts as exiting and causes Windows to close the window.

There are ways around this. One way is to open a windows command line from the Start menu and run python from there, by typing "python <name of script>.py". If you are changing the script as you go, you can do this repeatedly without having to reopen the window. Note that for maximum convenience, the python install directory should be added to the Windows PATH environment variable, and the directory where he is saving his scripts should be added to the PYTHONPATH environment variable. This prevents errors where Windows can't find Python, or Python can't find the scripts.

Your friend should also acquaint himself with the Python command line and how it differs from the Windows command line.

Bunny Cuddlin
Dec 12, 2004

Sab669 posted:

So a buddy of mine is going back to college to get a BS in Comp Sci in a few months, and I have a really dumb question. He decided to start with the MIT lecture last night and I guess it starts off having him use Python. I don't know the first thing about Python or the Python Shell application it was having him use (Or any scripting language for that matter, I'm a newbie C# guy). He was having a problem trying to "understand" how he would actually use this in the real world. By that I mean, if he saved a file and ran it, a command window would pop up and immediately disappear, even though the script requests user input.

I also think part of it for him is he expects to be creating full blown desktop applications, but you need to learn to walk before you can run. Most day-to-day users would poop themselves if they had to deal with the command line in any way.

He was just texting me about it this morning as I was getting ready for work, so I didn't have a chance to sit down and try to learn anything, though I figured while he's taking this course I probably should too :)

There are windowing libraries available for python just like any other mature language. You're right about walking before you can run, though. The windowing stuff is, by and large, handled by external libraries. You have to learn to write code before you can start using them. Your first didactic project in engineering isn't going to be building a bridge just like your first one in programming isn't going to be writing a fully-featured web browser.

Also (depending on where he's getting this CS degree, I suppose) CS is only tangentially about programming. Maybe he's looking for Software Engineering?

Sab669
Sep 24, 2009

Hammerite posted:

Sounds like he is working on Windows and is running into a limitation of the Windows command line. Basically, when you run a command line application on Windows, it will close the window as soon as the program exits, which is inconvenient if you wanted to see what the program did.

If your program is waiting for user input then it will not close until it gets it (and goes on to exit), but probably your friend's newbie code contains a syntax error or hits some other error condition which causes an exception to be thrown, which isn't caught and causes the program to exit with a printed error message, which counts as exiting and causes Windows to close the window.

There are ways around this. One way is to open a windows command line from the Start menu and run python from there, by typing "python <name of script>.py". If you are changing the script as you go, you can do this repeatedly without having to reopen the window. Note that for maximum convenience, the python install directory should be added to the Windows PATH environment variable, and the directory where he is saving his scripts should be added to the PYTHONPATH environment variable. This prevents errors where Windows can't find Python, or Python can't find the scripts.

Your friend should also acquaint himself with the Python command line and how it differs from the Windows command line.


He was just doing some super primitive thing, forgive the syntax mistakes because I'm typing this from memory...

x=raw_input("Whats your DOB?")
y=raw_input("Whats your name?")
print "Hi " + y + ", you were born on " + x


So it was requiring input, which is why I thought it was weird it would close instantly. Thanks though, I believe the Windows command line of "python fileName.py" is more-or-less what he was looking for.


He'll be attending University of Buffalo, Course List. I think he has to take all the 100 and 200 courses, then a 3 of the 300's and 3 of the 400's. Seems pretty open ended. I myself have a degree in Software Engineering, not CompSci. Maybe I just went to a lovely school but I'm quite unhappy with my education. I can build programs, sure, but I feel I lack a real technical vocabulary / understanding on some levels and definitely wish I had an actual CS degree. Like I am just awful with data structures and algorithm analysis.

Sab669 fucked around with this message at 16:10 on May 22, 2013

Dren
Jan 5, 2001

Pillbug
What your friend should do is launch a command shell and run his programs from there. Tell him to do:

code:
windows key + r
cmd
Once he's got a shell he should cd to the directory containing his python program and run "python yourprogram.py". Unlike when he double-clicks his program to launch it, the shell will not terminate once his program does.

I'm surprised he wants to write a GUI app, most teenage/college people I've talked to only care about writing web apps or phone apps.

BeefofAges
Jun 5, 2004

Cry 'Havoc!', and let slip the cows of war.

Python is often used for serverside applications and scripts that don't have a GUI or any user interaction at all, and therefore it doesn't matter that they're launched from the command line or as a service.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Foiltha posted:

There are two very good reasons and they're called pip and virtualenv.

Yeah. I can't remember the last time I apt-getted a python module. I'm not sure about this, but I kind of think newbies should be using pip + virtualenv (really virtualenvwrapper) (or whatever you're supposed to do with Python3) from the very beginning.

Sab669
Sep 24, 2009

Dren posted:

I'm surprised he wants to write a GUI app, most teenage/college people I've talked to only care about writing web apps or phone apps.

I'm not really sure what his "end goal" is. Right now he has a Bachelor's in Business Management and after spending 2 years in an office he's miserable, decided to go back for a BS in CS. I know I personally want to get into mobile development, but I think he just wants something completely different :v:

accipter
Sep 12, 2003

Sab669 posted:

"Friend" wants to start Python.

A few thoughts...

People have been suggesting running python from a cmd.exe window, which is a good idea. A nice shortcut to starting this to browse to the directory with Windows Explorer, and then type "cmd" in the location bar and hit enter. That way you don't have to navigate to the required directory.

I would also encourage you to suggest a good text editor (SublimeText2, Notepad++, or vim). Vim isn't probably the best suggestion at this point, but it is my preferred editor.

Python provides an interactive interpreter, which is great for interacting with objects to understand their behavior or trying out code. The interpreter that ships with Python is okay, but IPython has a number of improvements.

Installing Python Extensions on Windows can be a bit of hassle. I highly recommend this site: http://www.lfd.uci.edu/~gohlke/pythonlibs/ as it has pre-build binaries with Windows installers.

Going from application to GUI is an order of magnitude increase in complexity.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Dren posted:

Hammerite I think the thing to do in your case is to store the state of the directory walk outside of the scope of the function doing the work. Something like this:

Python code:
import os

class WalkState(object):
    def __init__(self):
        self.processed = []

def do_thing_to_file(path):
    pass

def walk_and_do_thing(dirpath, func, context=None):
    for root, dirs, files in os.walk(dirpath):
        for filename in files:
            func(os.path.join(root, filename))
            if context is not None:
                context.processed.append(os.path.join(root, filename))

def main():
    walk_state = WalkState()
    try:
        walk_and_do_thing('/', do_thing_to_file, walk_state)
    except IOError as e:
        print str(e)
        print 'Processed files before error: %s' % ', '.join(walk_state.processed)

if __name__ == '__main__':
    main()
If you need to you could even do something like pickle the current state and load it when the app comes back up to allow the user to resume from where they left off.

I actually prefer writing processing pipelines like this as coroutines but that's opening a whole can of worms.

So now suppose I want to do it like this:

Python code:
def depthCheckingDirectoryWalk (
   myGenerator, directory, subdirectories, outdata = None, **kwargs
):
   if outdata is None: outdata = {}
   if subdirectories >= 0:
      depthDict = {os.path.abspath(directory): 0}
      myGenerator = myGenerator(outdata, **kwargs)
      next(myGenerator)
      for (currentDir, subdirs, files) in os.walk(directory):
         myGenerator.send(currentDir, subdirs, files)
         if subdirectories is not True and subdirectories <= depthDict[currentDir]:
            subdirs.clear()
         else:
            subdirDepth = depthDict[searchdir[0]] + 1
            for d in subdirs:
               depthDict[os.path.join(searchdir[0], d)] = subdirDepth
   return outdata
And for the functionality that handles a given directory, I need to write a generator function that first sets up its environment, then repeatedly receives (currentDir, subdirs, files) tuples using the yield keyword. But I also want to write a convenience function that just calls depthCheckingDirectoryWalk() with the appropriate arguments. My question is, I want the generator function to accept specific keyword arguments. Where should I declare those? I guess I need to declare them for the generator function so that they are available in the function. But it feels like they should be declared for the convenience function too, so that they appear in help() output and suchlike; and yet then I'm putting the same thing in two places, which seems off.

Dominoes
Sep 20, 2007

Sab669 posted:

So a buddy of mine is going back to college to get a BS in Comp Sci in a few months, and I have a really dumb question. He decided to start with the MIT lecture last night and I guess it starts off having him use Python. I don't know the first thing about Python or the Python Shell application it was having him use (Or any scripting language for that matter, I'm a newbie C# guy). He was having a problem trying to "understand" how he would actually use this in the real world. By that I mean, if he saved a file and ran it, a command window would pop up and immediately disappear, even though the script requests user input.

I also think part of it for him is he expects to be creating full blown desktop applications, but you need to learn to walk before you can run. Most day-to-day users would poop themselves if they had to deal with the command line in any way.

He was just texting me about it this morning as I was getting ready for work, so I didn't have a chance to sit down and try to learn anything, though I figured while he's taking this course I probably should too :)
I had your friend's mindset two months ago. Advice:

1: Go through all of the Python tutorials on Codeacademy. They're very good.

2: Install a text editor that will run code automatically. I use Notepad++ with the NPP plugin. I can press alt+shift+f5, and the code will immediately run in a Windows command prompt that doesn't close after running.

3: Install Pip - this will make installing some packages painless. I don't remember how to do it offhand; I thought there was an installer, but I can't find it.

4: To make a program others can use, you need Cx_freeze and a GUI toolkit. Toolkits include TKinter, WxWidgets, QT and GTK. I've only used QT, but learning it was more difficult than learning Python itself, due to a lack of good or interactive tutorials like CodeAcademy. The official QT docs describe the properties of each GUI element without explaining what they do. You will need to package msvcp100.dll and msvcr100.dll, found somewhere in your windows directly with the program when distributing it, or ensure the user has Microsoft Visual C++ 2010 redistributable installed.

Dominoes fucked around with this message at 22:51 on May 22, 2013

Sab669
Sep 24, 2009

Thanks for the links, I'll check them out and send them to him when I get out of work in 3 more painful hours :suicide:

accipter
Sep 12, 2003

Dominoes posted:

The official QT docs describe the properties of each GUI element without explaining what they do.

What docs are you using? The PySide documentation is pretty nice. Here is an example.

Dominoes
Sep 20, 2007

accipter posted:

What docs are you using? The PySide documentation is pretty nice. Here is an example.
The official QT ones. For example, here's the QFileDialog page. I spent a while trying to figure out how to set the "Save as type" dropdown, and only found the answer after asking on the QT forums. It's a property called filter. It's mentioned in the document, but not explicitly, or with other common names associated with it, like extension, type or suffix.

The docs are thorough, but there's no available tutorials on par with Codeacademy, and figuring out how to structure your documents, and how the QT code fits with python takes work.

Dominoes fucked around with this message at 22:38 on May 22, 2013

Dren
Jan 5, 2001

Pillbug

Hammerite posted:

So now suppose I want to do it like this:

Python code:
def depthCheckingDirectoryWalk (
   myGenerator, directory, subdirectories, outdata = None, **kwargs
):
   if outdata is None: outdata = {}
   if subdirectories >= 0:
      depthDict = {os.path.abspath(directory): 0}
      myGenerator = myGenerator(outdata, **kwargs)
      next(myGenerator)
      for (currentDir, subdirs, files) in os.walk(directory):
         myGenerator.send(currentDir, subdirs, files)
         if subdirectories is not True and subdirectories <= depthDict[currentDir]:
            subdirs.clear()
         else:
            subdirDepth = depthDict[searchdir[0]] + 1
            for d in subdirs:
               depthDict[os.path.join(searchdir[0], d)] = subdirDepth
   return outdata
And for the functionality that handles a given directory, I need to write a generator function that first sets up its environment, then repeatedly receives (currentDir, subdirs, files) tuples using the yield keyword. But I also want to write a convenience function that just calls depthCheckingDirectoryWalk() with the appropriate arguments. My question is, I want the generator function to accept specific keyword arguments. Where should I declare those? I guess I need to declare them for the generator function so that they are available in the function. But it feels like they should be declared for the convenience function too, so that they appear in help() output and suchlike; and yet then I'm putting the same thing in two places, which seems off.

Welp, got myself into this.

First of all, check out the slides from david beazley's talk on coroutines. http://dabeaz.com/coroutines/Coroutines.pdf Read at least through slide 75. If you don't want to read that far then don't use coroutines. The code used in his presentation is available at http://dabeaz.com/coroutines/ for you to follow along. He's also got a handy decorator that keeps you from having to do setup on your coroutines.

For your question about where to put the keyword arguments the answer is probably both places. It sounds like you're creating a coroutine to handle the data from your private API and a convenience function for your public API.

Here is an example of how to setup a proper processing pipeline w/ coroutines that does most of what you need. You can wrap what is in my main up into a convenience function.
Python code:
import os
from coroutine import coroutine

@coroutine
def do_thing_to_file(arg1, arg2):
    while True:
        try:
            filepath = (yield)
            # do stuff here
        except GeneratorExit:
            return

@coroutine
def dirwalk(target):
    """Walk a directory structure and send file paths to target
    
    Use .send() to provide a directory name to walk.

    arguments:
    target - coroutine to send file paths to
    """
    while True:
        try:
            dirpath = (yield)
            for root, dirs, files in os.walk(dirpath):
                for filename in files:
                    filepath = os.path.sep.join([root, filename])
                    target.send(filepath)
        except GeneratorExit:
            target.close()
            return

@coroutine
def track_processed(processed, target):
    """Forward what it is sent to target then record that processing completed

    Use .send() to provide data to forward.

    arguments:
    target    - coroutine to forward to
    processed - list where processed items are stored
    """
    while True:
        try:
            item = (yield)
            target.send(item)
            processed.append(item)
        except GeneratorExit:
            target.close()
            return

def main():
    processed = []
    try:
        pipeline = dirwalk(
                       track_processed(processed,
                           do_thing_to_file(arg1="arg1value", arg2="arg2value")
                       )
                   )
        pipeline.send('/')
    except IOError as e:
        print str(e)
        print 'Processed files before error: %s' % ', '.join(processed)

if __name__ == '__main__':
    main()
Something worth mentioning here is that this problem doesn't actually require the use of coroutines. The reason .send() was included in the language was because of the interaction it has with regular generators. .send() can be used to reset a regular generator to a new state. e.g.

Python code:
def counter(start):
    value = start
    while True:
        new_value = (yield value)
        if new_value is not None:
            value = new_value
        else:
            value += 1

c = counter(1)

print c.next()
print c.next()
print c.next()
print c.send(16)
print c.next()
print c.next()
outputs

code:
$ python coroutine_justification.py 
1
2
3
16
17
18
Processing pipelines are cute and all, but you don't actually need them. You can do stuff with plain ol' functions. The reason I say to not use coroutines if you don't want to read through slide 75 of the presentation is that if you use them and someone else uses your code, that person will get confused and you will have to explain yourself. If you didn't read enough I'm afraid you won't be able to.

Dren fucked around with this message at 22:37 on May 22, 2013

Plorkyeran
Mar 22, 2007

To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed
Moving everything but the body of the try blocks to a decorator would make the coroutine-based code look pretty similar to regular functions.

Dominoes
Sep 20, 2007

evensevenone posted:

Use the tempfile module for temporary files. It'll handle creating a temp file/folder in a safe, user-writable place, not overwriting anything, and cleaning it up when done.
Is there a way to use this with strings?

For example,
Python code:
with open(tempfile.TemporaryDirectory() + '\\filename', 'wb') as f:
Doesn't work - TypeError, unsupported operand types for tempdirectory and str. Is there another way to specify/save/open a file in a temporary directory?

accipter
Sep 12, 2003

Dominoes posted:

Is there a way to use this with strings?

For example,
Python code:
with open(tempfile.TemporaryDirectory() + '\\filename', 'wb') as f:
Doesn't work - TypeError, unsupported operand types for tempdirectory and str. Is there another way to specify/save/open a file in a temporary directory?

Just work with tempfile.NamedTemporaryFile() and look at the name attribute for its name. This might not be a meaningful name, but that shouldn't make a difference because it is a temporary file.

Dominoes
Sep 20, 2007

accipter posted:

Just work with tempfile.NamedTemporaryFile() and look at the name attribute for its name. This might not be a meaningful name, but that shouldn't make a difference because it is a temporary file.
Like this?

Python code:
file = tempfile.NamedTemporaryFile().name + filename
    with open(file, 'wb') as f:
Leaving out the filename variable works, but then I don't know how to access the files. Using NamedTemporaryFile instead of directory, can I still look at all the files in the directory and then perform operations on them? I have working code using my own temporary folder.


Example code using a temp directory I made. Dir is a global variable that represents the program's path. empty_temp is a function that deletes all .pdf files in my temporary directory. Omitted some unnecessary-for-example code. Is it possible/worthwhile to use the tempfile module for this case?
Python code:
path = dir + '\\temporary\\'
empty_temp()
for plate in plates:
    filename = ...
    url = ...
    web_file = get(url).content
    save_path = path + filename
    with open(save_path, 'wb') as f:
            f.write(web_file)
    combine()
    with open(path + 'Plates.pdf', 'rb') as f: 
        return f.read()

def combine():
    path = dir + '\\temporary\\'
    filenames = [f for f in listdir(path) if f.endswith('.pdf')]     
    if len(filenames) > 0:
        merger = PdfFileMerger()
        for f in filenames:
            merger.append(fileobj = path + f)
        with open(path + 'Plates.pdf', 'wb') as f: 
            merger.write(f)
        merger.close()

Dominoes fucked around with this message at 00:47 on May 23, 2013

accipter
Sep 12, 2003

Dominoes posted:

Like this?

Python code:
file = tempfile.NamedTemporaryFile().name + filename
    with open(file, 'wb') as f:

Like this:

Python code:
>>> import tempfile
>>> with tempfile.NamedTemporaryFile() as fp:
>>>     print(fp.name)
c:\temp\tmpx3n3vl

Adbot
ADBOT LOVES YOU

Dominoes
Sep 20, 2007

accipter posted:

Like this:

Python code:
>>> import tempfile
>>> with tempfile.NamedTemporaryFile() as fp:
>>>     print(fp.name)
c:\temp\tmpx3n3vl
Thanks. Got it working!

Python code:
def download_combined(window, plates):
    global d2
    d = tempfile.TemporaryDirectory()
    d2 = d.name
    for plate in plates:
        filename = ...
        url = ...
        web_file = get(url).content
        with open(d2 + '\\' + filename, 'wb')  as f:
            f.write(web_file)
    combine()
    window.ui.statusbar.showMessage("Download complete") 
    with open(d2 + '\\Plates.pdf', 'rb') as f:
        return f.read()

        
def combine():
    filenames = [f for f in listdir(d2) if f.endswith('.pdf')]     
    if len(filenames) > 0:
        merger = PdfFileMerger()
        for f in filenames:
            merger.append(fileobj = d2 + '\\' + f)
        with open(d2 + '\\Plates.pdf', 'wb') as f: 
            merger.write(f)
        merger.close()
If I read the docs correctly, the files/folder will delete themselves when done. How does it know it's done? Should I use cleanup()?

Dominoes fucked around with this message at 02:55 on May 23, 2013

  • Locked thread