|
Mrenda posted:Is there a particular setup I should have for developing with Python on windows? I would recommend a miniconda installation. https://conda.io/miniconda.html
|
# ? Jul 6, 2018 14:55 |
|
|
# ? May 14, 2024 10:04 |
|
Boris Galerkin posted:I'm trying to play around with __getattr__ and __setattr__ so that I can do something like the following: You need to avoid calling __setattr__ again in __setattr__... which is what will happen if you have code:
code:
code:
|
# ? Jul 6, 2018 14:58 |
|
Mrenda posted:Is there a particular setup I should have for developing with Python on windows? Is developing on windows a requirement or a preference? Also: Web development or desktop app or machine learning etc? NtotheTC fucked around with this message at 15:21 on Jul 6, 2018 |
# ? Jul 6, 2018 15:08 |
|
Quote is not edit NtotheTC fucked around with this message at 15:20 on Jul 6, 2018 |
# ? Jul 6, 2018 15:11 |
|
TBH, just a vanilla Python Windows install works pretty well nowadays. 5 years ago it was bad. 2 years ago it was kinda frustrating. Now it's OK.
|
# ? Jul 6, 2018 15:13 |
I second Anaconda, I just normally spring for the full sized distro since it just works (tm).
|
|
# ? Jul 6, 2018 15:18 |
|
NtotheTC posted:Is developing on windows a requirement or a preference? I have python setup in a Linux VM where I'm playing around with Flask, but I want to be able to use it for stuff on Windows too rather than switching about in languages. I've done a few MOOC courses but never really made/finished anything, so I'm just looking to make a lil' program for Win10 that pulls weather info from an API I've yet to find, stores it and displays it.
|
# ? Jul 6, 2018 15:25 |
|
Mrenda posted:I have python setup in a Linux VM where I'm playing around with Flask, but I want to be able to use it for stuff on Windows too rather than switching about in languages. I've done a few MOOC courses but never really made/finished anything, so I'm just looking to make a lil' program for Win10 that pulls weather info from an API I've yet to find, stores it and displays it. Fair enough - I'm a big proponent of developing on the target OS so personally I'd install the latest python3 binary from python.org and then use pipenv to manage your python requirements. I don't recommend trying to go through git bash or bash for windows, if you want a posix environment then I'd just develop on native linux
|
# ? Jul 6, 2018 15:30 |
|
cinci zoo sniper posted:I second Anaconda, I just normally spring for the full sized distro since it just works (tm). Unless you're using PowerShell aka the standard terminal in Windows 10 (it's free software) Finally getting around to maybe thinking about how to implement activate and deactivate after 4 years
|
# ? Jul 6, 2018 17:53 |
|
Oh man I remember being super irritated about that a long time ago. Lol that they're just now doing something about it.
|
# ? Jul 6, 2018 18:35 |
|
Boris Galerkin posted:I'm trying to play around with __getattr__ and __setattr__ so that I can do something like the following: It was already mentioned but inside __getattr__ you need to call super().__getattr__ (pre-edit I said __getattribute__ here but I'm pretty sure that's incorrect - it's been a while though) to perform attribute access on self to avoid recursion. Similarly with super().__setattr__ if you want to set something on self in __setattr__. (I think super() is ok but it might be necessary to call them on object directly? I don't remember) Don't dig in the __dict__ directly unless you're absolutely sure you want to bypass basically all normal attribute access behavior. Another thing to keep in mind is that __getattr__ is a fallback that only gets called when __getattribute__ fails (there is a little more to it than that, reference the Python Data Model document). Absent any shenanigans that means that the name wasn't found in the instance or class. So, checking for the name on self in __getattr__ does nothing. If you're proxying a known thing and it's really just an issue of trying to avoid writing out 6 lines of getter and setter over and over, consider writing a factory function for your properties or your own property-like descriptor. Using a more narrowly targeted hook will save you headaches. breaks fucked around with this message at 00:19 on Jul 7, 2018 |
# ? Jul 6, 2018 20:43 |
|
I've actually managed to do something and it was easier than I thought despite there being a few hurdles. I'm now getting weather data from an API, saving it to an SQLite db and showing the temperature when it gets the data. The next step is putting this all in a gui/graphical format. What's best for that? (and has decent documentation/SO resources.) I'm trying to use as much standard Python as I can so TkInter sounds best? The other thing I have to do is stop using a loop with a time.sleep() to get the current weather info every hour, maybe a delta time since so I don't have the whole program waiting for just that.
|
# ? Jul 8, 2018 10:09 |
|
Mrenda posted:I've actually managed to do something and it was easier than I thought despite there being a few hurdles. I'm now getting weather data from an API, saving it to an SQLite db and showing the temperature when it gets the data. The next step is putting this all in a gui/graphical format. What's best for that? (and has decent documentation/SO resources.) I'm trying to use as much standard Python as I can so TkInter sounds best? You're probably better off using the timer object for updating - you're going to need to use it if you want to use a gui because you won't want to block the main thread. I don't have a strong opinion on python UI frameworks because I've never used python for UI commercially, but qt worked fine for me in a hobby project before.
|
# ? Jul 9, 2018 14:52 |
|
Bruegels Fuckbooks posted:You're probably better off using the timer object for updating - you're going to need to use it if you want to use a gui because you won't want to block the main thread. I don't have a strong opinion on python UI frameworks because I've never used python for UI commercially, but qt worked fine for me in a hobby project before. I went with TkInter as the default and it had enough things built in to update/call functions after a certain time so it's now all good. This is the first time I've made anything, even an extremely simple anything, that I have use for myself.
|
# ? Jul 9, 2018 15:27 |
|
If anyone ever wants a really simple GUI, particularly to make a form for taking arguments and running a script, big shout out to Gooey. Tutorial here: http://pbpython.com/pandas-gui.html I used this and a python to exe maker and made some little tools that helped my coworkers who would be scared of the command line.
|
# ? Jul 10, 2018 00:52 |
|
I have a Jupyter notebook where I have a set of 3 subplots which share an x-axis. The x-axis is time in weeks, and I would like to make the plot scrollable (ideally) with two buttons at the bottom - a forward button which increases the upper and lower bound on the x-axis by 4 weeks, and backward button which decreases the upper and lower bound on the x-axis by 4 weeks. This is the code i use right now to generate a static image in the notebook cell: Python code:
|
# ? Jul 10, 2018 19:07 |
|
I don't know quite how to ask for what I'm looking for because I know very little about actual software development even if I've done some programming. So I wrote some scripts that manipulate text in certain ways and output certain strings into a new text file. Custom, very weird use case NLP tasks. These more or less work for my purposes, but my bigger goals are to (1) collect all of these into a command line program that would just be easier for me (and ideally others) to use (2) get the whole shebang online for people to use. Can anyone direct me towards some resources to help me think through how to do (1)? (2) is a huge stretch goal, but ideally I would be able to do (1) in a way that would set me up for success on (2). I'm looking for some explanations of how to go from a folder of .py files that do little tasks to an organized piece of software that I can work on, add features to, etc. I want to have a sense of best practices as my lil scripts get more complicated over time. There must be a name for what I'm trying to get at here, but I don't even know.
|
# ? Jul 11, 2018 14:15 |
|
Business posted:I don't know quite how to ask for what I'm looking for because I know very little about actual software development even if I've done some programming. I have not tried it at all so I can't vouch for it, but Python Fire might be useful for your purposes?
|
# ? Jul 12, 2018 02:07 |
|
Seconding Click- using it to create neat command line apps is one of life's purest pleasures and I wish I could make it my full time job somehow
|
# ? Jul 12, 2018 10:47 |
|
Ghost of Reagan Past posted:Try Click. It's very nice to work with, kind of guides you into writing good user interfaces, and has really extensive documentation that hits on some design questions. You should refactor your individual scripts into functions and classes you can import around into other files, and then write a CLI app around your various tools using click. So you might have a script that right now that tags parts of speech and prints the tree (or whatever). You might then refactor that into a function that parses the sentence and a function that prints the tree. Then write a CLI app with click that you use like nlp tagger "this is a sentence" --tree, but then you can write an option for the app nlp tagger "this is a sentence" --table that prints a table. Started learning the click stuff today, the documentation is really good. Much appreciated!
|
# ? Jul 12, 2018 20:44 |
|
In the python docs for glob, it's defined like so:code:
|
# ? Jul 13, 2018 05:24 |
porksmash posted:In the python docs for glob, it's defined like so: The * means that's the end of positional arguments. recursive is a keyword-only argument. You can call it with glob.glob(somepath, recursive=True) if you'd like.
|
|
# ? Jul 13, 2018 06:24 |
|
SurgicalOntologist posted:You need to avoid calling __setattr__ again in __setattr__... which is what will happen if you have breaks posted:It was already mentioned but inside __getattr__ you need to call super().__getattr__ (pre-edit I said __getattribute__ here but I'm pretty sure that's incorrect - it's been a while though) to perform attribute access on self to avoid recursion. Similarly with super().__setattr__ if you want to set something on self in __setattr__. (I think super() is ok but it might be necessary to call them on object directly? I don't remember) Don't dig in the __dict__ directly unless you're absolutely sure you want to bypass basically all normal attribute access behavior. Thanks, I’ve got it working now. Random unrelated question, when I write a function with *args and keyword arguments, is it suppose to go like this code:
code:
|
# ? Jul 13, 2018 13:46 |
|
Boris Galerkin posted:Random unrelated question, when I write a function with *args and keyword arguments, is it suppose to go like this It depends on if you want c to be a keyword-only argument or not. Should a call of func("Hey", "you", "guys") be interpreted as c="guys" or args=("guys",)?
|
# ? Jul 13, 2018 14:24 |
|
I would like to make a pandas dataframe out of a JSON file containing scraped court data. I'd like all the data to appear in columns but there's lots of nesting and I can't figure out how to get it in a nice big table like I'd like. I thought the way to go would be something like: code:
csvkit can give me a csv output which is easier but it requires a ton of reoganizing and clean up. I figured a custom solution might be in order. Data sample: (Very long so I pastebinned) https://pastebin.com/PhDkrCME
|
# ? Jul 14, 2018 16:15 |
You won't be able to trivially map a decently hierarchical JSON document to a table or set of tables. I had to solve a similar problem at work and my final (naturally a temporary quick and dirt one to be rewritten better at the time) solution was to write a MongoDB script that does some server-side filtering and removes unnecessary fields and everything, a long-ish Python script with functions to read what i need and write it into pandas. On Python side I did iterate over individual JSON documents. The process was good enough to comb through a couple gigabytes of JSON poo poo reasonably quickly. Edit: index orient is probably not what you may want in any case. cinci zoo sniper fucked around with this message at 16:30 on Jul 14, 2018 |
|
# ? Jul 14, 2018 16:27 |
|
Someone who knows what they're doing will be along soon, but looking at it your issue might be the fact (I think, I'm on a phone here) that you're passing it an array wrapped in an object, so it looks like one item containing one value (your array structure) Try making your array the top level instead (remove the outer { } and get rid of the array's name key). Or maybe parse the JSON and pluck out the array object to pass to pandas for your nefarious needs
|
# ? Jul 14, 2018 16:36 |
*coughs* This is literally my job, getting meaningful things out of JSON and XML piles.
|
|
# ? Jul 14, 2018 16:43 |
|
cinci zoo sniper posted:You won't be able to trivially map a decently hierarchical JSON document to a table or set of tables. I had to solve a similar problem at work and my final (naturally a temporary quick and dirt one to be rewritten better at the time) solution was to write a MongoDB script that does some server-side filtering and removes unnecessary fields and everything, a long-ish Python script with functions to read what i need and write it into pandas. On Python side I did iterate over individual JSON documents. The process was good enough to comb through a couple gigabytes of JSON poo poo reasonably quickly. I dont mind some time investment in solving this issue. I will need to do it weekly or daily and the data will be formatted like this always. Whats the best way to solve it and got a link to learn more about it? I have no experience with MongoDB and dont control the server sending the data but am willing to learn. CarForumPoster fucked around with this message at 16:58 on Jul 14, 2018 |
# ? Jul 14, 2018 16:55 |
CarForumPoster posted:I dont mind some time investment in solving this issue. I will need to do it roughly weekly and the data will be formatted like this always. What's your desired output, to roughly sketch on that sample you linked? I've not seen that many good links on this to be honest, just random blog posts ways back. It's a technologically simple problem that can be labour-"intensive" if your source data and your desired output don't cooperate (like it often is for me since if there's one thing that people's credit histories are not then it's concise). I'll take a look at your file a bit later, on the go right now. Also yeah, good point you raise - how exactly do you receive this stuff? cinci zoo sniper fucked around with this message at 16:59 on Jul 14, 2018 |
|
# ? Jul 14, 2018 16:56 |
|
cinci zoo sniper posted:What's your desired output, to roughly sketch on that sample you linked? I've not seen that many good links on this to be honest, just random blog posts ways back. It's a technologically simple problem that can be labour-"intensive" if your source data and your desired output don't cooperate (like it often is for me since if there's one thing that people's credit histories are not then it's concise). I'll take a look at your file a bit later, on the go right now. My goal is to use the data in a variety of machine learning, NLP and statistics tasks. CSVkit (the 70% option) may be a better option than I once conceived. The goal would be to map the fields to a set of names that I can play with. E.g. right now I take judge_name, one hot encode them, and look for correlations to outcomes. code:
CarForumPoster fucked around with this message at 17:20 on Jul 14, 2018 |
# ? Jul 14, 2018 17:06 |
Okay that data example is really straightforward.code:
Good luck! E: To be noted, I don't remember what's behaviour on pd.DataFrame(None) so this may die if crudely extended to an empty file (e.g. if missing data doesn't preserve same hierarchy early on). cinci zoo sniper fucked around with this message at 17:34 on Jul 14, 2018 |
|
# ? Jul 14, 2018 17:32 |
|
cinci zoo sniper posted:*coughs* This is literally my job, getting meaningful things out of JSON and XML piles. I typed that before anyone replied
|
# ? Jul 14, 2018 17:37 |
baka kaba posted:I typed that before anyone replied It’s all good, I’m cranky anyways since I’m out of Diet Coke unexpectedly.
|
|
# ? Jul 14, 2018 17:39 |
|
I've been painfully reminded that I need to learn Python beyond the barest of bare basics. As part of an interview process, I've been asked to parse an nginx log file and find all IPs if something performs a PUT, POST, PATCH, or DELETE 3 or more times in a 10 second interval Question: Given a nginx access logfile, find all the ip’s which perform a PUT, POST, PATCH or DELETE requests 3 or more times within a 10 second interval. I thought it would be easy enough, but the inclusion of this code has left me not sure what to do. Filtering the log file and extracting the IP address and the actions logged is simple enough, but the logic for filtering for 10 second intervals within this code has left me feeling like a drooling nitwit. On a completely related note, can people here still vouch for CodeAcademy being a good way to start truly learning Python? I'm at the point in my career (Systems Admin / Engineer / whatever) now where I really should learn proper Python coding. code:
|
# ? Jul 16, 2018 16:57 |
|
CHEF!!! posted:On a completely related note, can people here still vouch for CodeAcademy being a good way to start truly learning Python? I'm at the point in my career (Systems Admin / Engineer / whatever) now where I really should learn proper Python coding. To be honest, if you understand basic python and programming in general, you'd probably be better served by reading Fluent Python.
|
# ? Jul 16, 2018 17:04 |
|
If you want to see a fully featured python program for some examples on how to do nginx log analysis check out https://github.com/lebinh/ngxtop
|
# ? Jul 16, 2018 18:06 |
On whiteboard exercise level with using basic Python, I'd just pile a dictionary of nested lists where each line read checks delta of last and 3rd from end request datetimes. This would likely be slow as gently caress, but fast and straightforward to write. Another quick option would be to filter by IP into separate lists, then just filter lists down to right request type, and compute running delta on last 3 request datetimes. Most likely much better on performance although I haven't done text filtering like that.
|
|
# ? Jul 16, 2018 18:26 |
|
Another thing you could try is having a list containing the last 10 seconds' of IPs (filtered to only include the request types you're looking for). So when you get a new IP, add it to the list, then pop off the entries from the front of the list that have a timestamp > 10s earlier than your new entry. Then you can scan your list for matches on that new IP and see if you get 3+ Like a moving window, basically
|
# ? Jul 16, 2018 19:41 |
|
|
# ? May 14, 2024 10:04 |
|
baka kaba posted:Another thing you could try is having a list containing the last 10 seconds' of IPs (filtered to only include the request types you're looking for). So when you get a new IP, add it to the list, then pop off the entries from the front of the list that have a timestamp > 10s earlier than your new entry. Then you can scan your list for matches on that new IP and see if you get 3+ This is basically the ideal use case for a deque and more efficient than the general purpose list.
|
# ? Jul 16, 2018 19:52 |