|
In order to get your last code block to work, you would want to have push() raise an exception in the cases where it is not successful. This could be one of the predefined ones that you laid out in the second code block (IOError, OSError) or you can define your own if you're feeling saucy. Take a look at 8.4 and 8.5 in the documentation here for instructions on how to do both of those things.
|
#
?
Feb 18, 2020 17:04
|
|
|
|
| # ? May 13, 2024 21:46 |
|
|
Also worth looking at "finally" in python if you aren't familiar with it. Probably not useful for this situation, but useful for many.
|
|
#
?
Feb 18, 2020 17:41
|
|
|
If you don’t need any fancy error handling, running cleanup() directly after push() in the same block is an option, and you can structure push to not catch any critical errors that mean it failed. If push() fails by raising an exception, cleanup() won’t be executed. Does that make sense? I feel like lots of people do too much exception management and swallow errors that should interrupt execution. There’s no functional reason to manage exceptions unless you have a recovery strategy. If an error isn’t recoverable (or not worth making deep changes to your program to recover) it’s reasonable to crash. It’s possible there’s an understanding gap here about how exception handling works. If a statement inside of push() raises an exception, you don’t have to catch that with a try/except inside of push(). Exceptions bubble all the way up the stack until an enclosing try block handles it in some way or your entire program crashes. YanniRotten fucked around with this message at 17:50 on Feb 18, 2020 |
|
#
?
Feb 18, 2020 17:43
|
|
|
YanniRotten posted:There’s no functional reason to manage exceptions unless you have a recovery strategy. If an error isn’t recoverable (or not worth making deep changes to your program to recover) it’s reasonable to crash. This makes sense but I think in this case I need to actually catch the error because the script is going to run every hour or so and we'd want an email alert if it wasn't working... presumably that would be harder to do if the program just crashes. YanniRotten posted:It’s possible there’s an understanding gap here about how exception handling works. If a statement inside of push() raises an exception, you don’t have to catch that with a try/except inside of push(). Exceptions bubble all the way up the stack until an enclosing try block handles it in some way or your entire program crashes. Yeah this makes me think I'm going about it the wrong way. I should maybe just put the whole script in one big try block that catches any exception? Then send an email alert with the exception. For my own peace of mind I think I still need some kind of check that push() has done its job properly (maybe by counting the remote files before and after it's called and checking the right number have been uploaded), because if the "cleanup" process moves local files before they've been uploaded that would be pretty disastrous... Thanks for the replies! Will keep working on it tomorrow. This is my first opportunity in this new job to do some actual coding so it's a nice change. 
|
|
#
?
Feb 18, 2020 18:22
|
|
|
fuf posted:I should maybe just put the whole script in one big try block that catches any exception? Then send an email alert with the exception.
|
|
#
?
Feb 18, 2020 18:24
|
|
|
If the script is being run with cron, cron will just email you the stderr when the script crashes. (assuming your host system is configured correctly)
|
|
#
?
Feb 18, 2020 18:27
|
|
|
yeah, you're gonna have a much better time keeping the script simple and letting an external runner send you an email when the script fails. if anything, you may want to do some logging to help diagnose any issues that come up. if you use systemd timers instead of cron then logging is simply a matter of printing log events to stdout
|
|
#
?
Feb 18, 2020 22:21
|
|
|
fuf posted:this is some real programming 101 poo poo and I'm embarrassed to ask but... what's the best way to check if a function has finished successfully before running a second function? Several things: One, you'd generally assume that everything is fine and continue the program with that assumption. Returning a bool for success/failure is something you do in languages that don't have exceptions, or if you want to do tests (e.g. if this_file_exists(): ). On failure you can let the exception thrown by functions bubble up or throw your own. Second, if you need to do setup/cleanup you could consider writing a context handler instead, so you could do something like this (completely made up) example: code:Third, you could use the python built-in logging module. I use this a lot in scripts that run in cron. Catching an exception and exiting with a non-zero exit code after writing the exception message with logging.error() or logging.critical() is the most unix-y way of terminating after an unrecoverable error. Letting the exception bubble to the top prints a stack trace to stderr and that's often scary to people who get to see it.
|
|
#
?
Feb 19, 2020 12:07
|
|
|
ant73YBIrcjUaGP9z82f tPaoT8BKnTOFItQt6iWW sHjfi9IWi8r8W094AXUT C9VLwa1fvajX1VJxLNpj fUUWQJhZdyvyaetarV0U VXgNjSCoh1Lp9puLTaKF XYIflrfSmPTnBoVlSlNQ OGN0KYE84Kl9axBZtJEL 8WmwGxwywN0mv56Fsjz0 BpkyVd8ivulI39aMew3W Plasmafountain fucked around with this message at 23:58 on Feb 27, 2023 |
|
#
?
Feb 20, 2020 20:09
|
|
|
You can use NetworkX to draw graphs with matplotlib, but the docs for this will tell you to use a dedicated tool like graphviz. graphviz is a separate executable but the graphviz python module will call it for you to render graphs, so you don't need to feed it anything manually.
|
|
#
?
Feb 20, 2020 20:32
|
|
|
skull mask mcgee posted:You can use NetworkX to draw graphs with matplotlib, but the docs for this will tell you to use a dedicated tool like graphviz. Same suggestion adding maybe this might help: (uses graphviz) https://github.com/fastai/fastdot
|
|
#
?
Feb 21, 2020 00:48
|
|
|
Zero Gravitas posted:From what I can find I seem to be pushed towards Graphviz but from what I can see graphviz is installed as a seperate exe that then has prepared files fed into it. Graphviz has this kind of thing figured out pretty well so there aren't many competing solutions. You can feed it from stdin though so there are a number of wrappers available. The file format is pretty simple and human-writable. Also: Gravitas  Gang
|
|
#
?
Feb 21, 2020 10:06
|
|
|
The impression I got from Zero Gravitas' post is they wanted something like an expanded treeview GUI widget rendered to an image. I haven't seen that done before myself using a plotting library.
|
|
#
?
Feb 21, 2020 20:05
|
|
|
Rocko Bonaparte posted:The impression I got from Zero Gravitas' post is they wanted something like an expanded treeview GUI widget rendered to an image. I haven't seen that done before myself using a plotting library. Sounds like plotting a decision tree except with >2 branches per root node. fastai uses grapviz for this, e.g.: 
|
|
#
?
Feb 21, 2020 20:17
|
|
|
Regex wizards, I need your help. I have a text document that contains answers to practice questions. I want to turn said questions/answers in to flashcards. I've already been able to regex and capture the question bank. I'm having trouble creating a regex that will capture each answer. In the link below, you can see the format of the document. Basically, for every answer I want to capture all the characters up til the next answer. The below regex101 hopefully illustrates the point. I'm missing the ability to deal with some sort of newline character (this was exported from a PDF to Word, then pasted to .txt). https://regex101.com/r/1eDkVN/1 Any help you can provide is greatly appreciated! Hughmoris fucked around with this message at 22:47 on Feb 21, 2020 |
|
#
?
Feb 21, 2020 22:27
|
|
|
Hughmoris posted:Regex wizards, I need your help. e: forgot multiple digits on the lookahead, fixed: https://regex101.com/r/1BhLlg/2 ee: and removed an unnecessary pair of parentheses: https://regex101.com/r/1BhLlg/3 eee: Looks like you also forgot the possibility of multiple digits on the question numbers, and you'll probably want to capture those to match questions to answers, so: https://regex101.com/r/1BhLlg/4 OK I can't stop: with named groups to separately capture the question number, correct choice and full text of the answer: https://regex101.com/r/1BhLlg/5 KICK BAMA KICK fucked around with this message at 23:21 on Feb 21, 2020 |
|
#
?
Feb 21, 2020 22:49
|
|
|
KICK BAMA KICK posted:No wizard but I like figuring these out -- at a glance this looks like it works on the input provided? I think what I did was a) make your .* that grabs text indefinitely lazy, b) turn on the s flag for single-line mode, which causes the . to also match newlines and c) add a positive lookahead matching either the number-dot-tab pattern or the end of string as a place to stop. You are a scholar and a saint! Your magic worked like a charm. I had forgotten about the look-ahead feature. Yesterday I was weeping at the thought of turning 450 practice questions in to Anki flashcards by copy/pasting from a PDF. I dusted off Python and hacked together a workable solution in about 30 minutes (a lot of it spent remembering how to use the regex module and CSV writer). I love how easy Python can make things.
|
|
#
?
Feb 21, 2020 23:40
|
|
|
CarForumPoster posted:Sounds like plotting a decision tree except with >2 branches per root node. fastai uses grapviz for this, e.g.: Apparently not, though: Zero Gravitas posted:Everything I google turns up decision trees which is not what im after; not least cos the flow is top down rather than bottom up. My impression is some kind of tree renderer takes care of most of it but there's a peculiarity to the layout. I wonder if a similar view can be done if somehow one can enforce: 1. Parents are up and left of children 2. Lines connect parents to children by going down and right
|
|
#
?
Feb 22, 2020 00:44
|
|
|
Rocko Bonaparte posted:Apparently not, though: Comedy option, make it a DataFrame, plot it in matplotlib and save it as an image. Seems like maybe all he needed is to add text to an image using PIL, nesting as needed. Something like this example that is pulled straight from my rear end without testing code:CarForumPoster fucked around with this message at 02:29 on Feb 22, 2020 |
|
#
?
Feb 22, 2020 02:18
|
|
|
Yeah I guess there's always just drawing it. It looks like they went and hid though hahaha. As for my own stuff: I am trying to switch to a log configuration instead of a hard-coded one. There are a few things that are dynamically-calculated that I can't readily put in my configuration. The current tool allows for a different logging root directory and the log file is created using a certain signature. I saw something about using a filter to do this for the file handler, but it's also kind of clumsy. That stuff came from Stack Exchange back in 2012 or so; I hoped something newer came along that made this kind of thing easier. What I'm thinking of doing is loading in the dictionary with some reserved words that I'll recursively replace with the dynamically-created settings if I find them. That's then fed into logging.config.dictConfig. So for a standard fileHander, the file path will be something like "%{logPath}%\stuff_%{timestamp}%\%{pidstamp}%.log".
|
|
#
?
Feb 26, 2020 20:24
|
|
|
Responding to myself since some time's elapsed:Rocko Bonaparte posted:I am trying to switch to a log configuration instead of a hard-coded one. There are a few things that are dynamically-calculated that I can't readily put in my configuration... I've pretty much finished this all up and determined that there isn't any good moon magic for the dynamically-calculated part. I had to declare a standard in my strings to allow injection of the values coming from the outside, and I just replace them recursively in the dictionary before I feed them into logging.dictConfig. You might see me in the general thread asking for tools and standards for that kind of preprocessing.
|
|
#
?
Feb 28, 2020 22:46
|
|
|
Does PyCharm (or a plugin) have any kind of shortcut where I can put the cursor on a function or class get the full dotted-path name to that from the project root? i.e., for some weird reason I want to run a specific test from an external command line and that would be more convenient than typing out the entire hierarchy of my test packages.
|
|
#
?
Feb 29, 2020 20:18
|
|
|
KICK BAMA KICK posted:Does PyCharm (or a plugin) have any kind of shortcut where I can put the cursor on a function or class get the full dotted-path name to that from the project root? i.e., for some weird reason I want to run a specific test from an external command line and that would be more convenient than typing out the entire hierarchy of my test packages. I don't have PyCharm in front of me to look up what it's called, but I have Ctrl-Alt-Shift-C mapped to the action that does this. edit: I think it's called "Copy Reference".
|
|
#
?
Feb 29, 2020 21:38
|
|
|
Thermopyle posted:I don't have PyCharm in front of me to look up what it's called, but I have Ctrl-Alt-Shift-C mapped to the action that does this.
|
|
#
?
Feb 29, 2020 21:59
|
|
|
fuf posted:Thanks for the replies! Will keep working on it tomorrow. This is my first opportunity in this new job to do some actual coding so it's a nice change. Here is my first pass at this little automation script. Posting just in case anyone can spot anything embarrassingly bad. Some actual developers are probably gonna look at it before they let us put it live so don't wanna look like too much of an idiot (no more than is strictly necessary I mean). Still not sure about the logging / exception stuff really. code:
|
|
#
?
Mar 2, 2020 15:29
|
|
|
fuf posted:Here is my first pass at this little automation script. Posting just in case anyone can spot anything embarrassingly bad. Some actual developers are probably gonna look at it before they let us put it live so don't wanna look like too much of an idiot (no more than is strictly necessary I mean). Still not sure about the logging / exception stuff really. The only non-nitpicky thing I see here is that sftp should really be a parameter to each of the functions that use it. Right now they all reference a global "sftp" variable, which is generally not good practice (the exceptions to this are module imports, and constants). I usually like to write the main part of my scripts inside a function and then call that function to run the script, in part because it makes it hard to use globals like this by accident. For example, like this: code:
|
|
#
?
Mar 2, 2020 16:47
|
|
|
Nippashish posted:The only non-nitpicky thing I see here is that sftp should really be a parameter to each of the functions that use it. Right now they all reference a global "sftp" variable, which is generally not good practice (the exceptions to this are module imports, and constants). I usually like to write the main part of my scripts inside a function and then call that function to run the script, in part because it makes it hard to use globals like this by accident. ahhhh, clever, thanks. I've rewritten it following your structure.
|
|
#
?
Mar 2, 2020 17:28
|
|
|
Also, depending on your python version (anything >=3.6 I think?), I would replace the string format stuff for logging with f-strings, which are both faster and more readable. https://www.python.org/dev/peps/pep-0498/
|
|
#
?
Mar 2, 2020 19:31
|
|
|
There's nothing wrong with using the os.path stuff, but I'd be using pathlib on 3.4 and up just because I like it's interface better.
|
|
#
?
Mar 2, 2020 20:03
|
|
|
Hollow Talk posted:Also, depending on your python version (anything >=3.6 I think?), I would replace the string format stuff for logging with f-strings, which are both faster and more readable. The logging module’s printf-style formatting can be even faster, since it won’t evaluate the message if the logging level means it’s not going to be logged. Of course this makes your logging calls a lot uglier.
|
|
#
?
Mar 2, 2020 20:27
|
|
|
Hollow Talk posted:Also, depending on your python version (anything >=3.6 I think?), I would replace the string format stuff for logging with f-strings, which are both faster and more readable. Ah that's cool. Speed isn't really an issue but more readable is definitely good. Annoyingly the server has python 2.6.6 currently, and it might be hard to request an upgrade unless there's a good reason. The python version might be an issue because I've been using argparse to pass command line arguments (to specify "push" or "pull" mode) and I don't think that's included in 2.6. I'll have to come up with another way or request an upgrade. Thermopyle posted:There's nothing wrong with using the os.path stuff, but I'd be using pathlib on 3.4 and up just because I like it's interface better. This looks good but same problem with the python version here unfortunately.
|
|
#
?
Mar 3, 2020 10:31
|
|
|
Python 2.6 is even more EOL than what's EOL but still supported by distros (2.7). Even Debian oldstable has Python 3.5 and 2.7 in its repos. And Debian old-oldstable has 2.7.9. Even CentOS 7 has 2.7!!
|
|
#
?
Mar 3, 2020 11:23
|
|
|
Yeah I think 2.6 is CentOS 6 era. Yikes, but I can't honestly say that my orgs don't have any older servers kicking around for stupid/bad reasons If you're allowed to install software, then you could at least set up a virtual environment with a newer version of python. I'd use Anaconda for this but everyone has their own preferences
|
|
#
?
Mar 3, 2020 11:39
|
|
|
QuarkJets posted:Yeah I think 2.6 is CentOS 6 era. Yikes, but I can't honestly say that my orgs don't have any older servers kicking around for stupid/bad reasons One note re: Anaconda. If it’s in a corporate environment I’m pretty sure you need to pay for a license, so if budget is an issue it may not be an option.
|
|
#
?
Mar 3, 2020 12:57
|
|
|
Pyenv if the tool chain can still build it as another option.
|
|
#
?
Mar 3, 2020 13:04
|
|
|
Proteus Jones posted:One note re: Anaconda. If it’s in a corporate environment I’m pretty sure you need to pay for a license, so if budget is an issue it may not be an option. I don't think that's the case? They sell Anaconda Enterprise but it's for the plus-up in features related to deployment/management of installations.
|
|
#
?
Mar 3, 2020 13:43
|
|
|
fuf posted:The python version might be an issue because I've been using argparse to pass command line arguments (to specify "push" or "pull" mode) and I don't think that's included in 2.6. I'll have to come up with another way or request an upgrade. Python 2.6 is very old, but sometimes its not worth it to fight the status quo. Either way it is worth learning how to use virtual environments so that you can develop locally using whatever version of python is on the server you need to deploy to (even if you can swing an upgrade, who knows if they'll upgrade to the same version of python you happen to have locally). If you make a virtual environment with python 2.6 and develop using that it will prevent you from using features that don't exist where the code needs to run.
|
|
#
?
Mar 3, 2020 13:48
|
|
|
pmchem posted:I don't think that's the case? They sell Anaconda Enterprise but it's for the plus-up in features related to deployment/management of installations. You’re right, I was mistaking it with a different free-for-individual-paid-for-commercial product. Sorry for the confusion.
|
|
#
?
Mar 3, 2020 17:30
|
|
|
Thanks for the replies. The servers are definitely CentOS, not sure what version, but it doesn't surprise me at all if they are on a really old version and an old version of python. There's a lot of institutional sluggishness here. It's not really something I want to stick my neck out about any time soon so I'll probably try and limit myself to stuff that works on 2.6. Setting up a new virtual environment with 2.6 is a very good idea.
|
|
#
?
Mar 4, 2020 10:38
|
|
|
|
| # ? May 13, 2024 21:46 |
|
|
iirc, CentOS5 has Python 2.4, CentOS6 had Python 2.6, and CentOS7 had Python 2.7 as the default versions for system python. But there are python3 RPMs available in the EPEL, plus a whole bunch of other python packages that you may want. You should ask your IT people about access to the EPEL, or even just try searching for "python3" with yum and see if the EPEL (or some other repository with python3 in it) is enabled. You'd probably be on an older version of python3 but that's still better than being on 2.6 (imo). I guess this would be more than doing nothing, but since the RPMs are coming from official repositories IT shouldn't take any issue with it.
|
|
#
?
Mar 4, 2020 10:47
|
|