|
Hughmoris posted:Speaking of Pandas, I run in to trouble when I need to create additional columns that are filled based on other column criteria. For example, if I have a CSV of: code:
|
# ? Aug 15, 2017 03:00 |
|
|
# ? Jun 3, 2024 22:40 |
|
vikingstrike posted:
Or even simpler: code:
|
# ? Aug 15, 2017 03:34 |
|
For people getting to grips with pandas, the book "Python for Data Analysis" written by pandas creator Wes McKinney is a good primer.
|
# ? Aug 15, 2017 03:47 |
|
Ok i'll check out pandas. Thanks everyone. I was using QtConsole launched through anaconda yesterday but would spyder be more suitable?
|
# ? Aug 15, 2017 08:44 |
|
Jose posted:Ok i'll check out pandas. Thanks everyone. I was using QtConsole launched through anaconda yesterday but would spyder be more suitable?
|
# ? Aug 15, 2017 09:23 |
|
vikingstrike posted:
Jose Cuervo posted:Or even simpler: Thanks for these.
|
# ? Aug 16, 2017 00:58 |
|
Can anyone link a good guide for combining pandas and matplotlib? Basically how matplotlib usage differs if I'm using pandas data frames
|
# ? Aug 16, 2017 10:38 |
|
Jose posted:Can anyone link a good guide for combining pandas and matplotlib? Basically how matplotlib usage differs if I'm using pandas data frames
|
# ? Aug 16, 2017 10:49 |
|
python is fukin lame btw
|
# ? Aug 16, 2017 12:39 |
|
Jose posted:Can anyone link a good guide for combining pandas and matplotlib? Basically how matplotlib usage differs if I'm using pandas data frames Pandas has some plotting functions that will output matplotlib axes that you can tweak and save from there. Plot() is the main interface, but some others like hist() and boxplot() have one off functions. Like Cingulate said, seaborn is also a nice library that helps bridge these worlds and it is dataframe aware. Although in either case you might have to use a bit of matplotlib to make things exactly the way you want.
|
# ? Aug 16, 2017 12:51 |
|
You might also be interested in other plotting libraries. I'm personally fond of ggplot, a port of the R library. There's also Bokeh, which has some interactivity functionality and is useful for web data visualizations. But seaborn + matplotlib will serve you well.
|
# ? Aug 16, 2017 13:48 |
|
Here is a great article that explains routine usage of matplotlib + pandas. http://pbpython.com/effective-matplotlib.html
|
# ? Aug 16, 2017 15:29 |
|
Ok, so I have a bit of a puzzle. I've been loving about with Unreal Engine and I found this very fun bit of library: https://github.com/20tab/UnrealEnginePython I've been coding in C++ on this thing but honestly I'm a bit poo poo at C++, something about that loving language just gives me the hebeejebees. If it was C, no probs, but something about C++ just pisses me off. Anyway , this library is awesome. it works, you get the GPU crushing sexiness of UE4, but you can script the *poo poo* out of it with Python. Theres bit of a focus on editor automation, which is fine (Makes pulling 3d-coat textures together into proper materials much saner). But it also seems to work pretty well with in game scripting because it has some magic introspection auto api generating poo poo. I strongly recomend playing with this, if you fancy making games, because it turns UE4 into the first AAA quality python engine. Anyway, heres the problem. UE4 uses an object/component model. Each item in the game (and light, and camera and sound and whatever) is an Actor object, and has a list of components attached, with the basic idea that you build up your objects by composition of components. Its kind of the gold-standard pattern in game dev. So the UE4 python plugin lets you use a python class as a component. The life cycle of the thing is that when the Actor is instantiated., the PythonComponent instantiates the class ,then at some point , very soon after, all the components (C++, Python, etc) have a begin_play (or BeginPlay in the C++ classes) method which acts as the setup function when the scene starts winding into action. Of course like any python class the __init__ is called at true instantiation, but this is not really when one should construct as the UE4 python plugin appears to wait till AFTER __init__ to add the bits and pieces to the class to let it behave like a real UE4 object. Anyway, I want to have a component called "Transmitter" that acts as a relay to a python server via rabbitMQ. The other objects in the game need to register with it using a register method. All good and easy, and I need to grab a reference to that transmitter so the registration can take place. Normally I'd just use a Singleton but my usual method of making a singleton in python involves the rest of the program agreeing not to call __init__() and instead using a static method that grabs an instance and everything else agrees to behave itself. The problem is __init__ is called by the UE4 framework, and I have no control over the order of things being instantiated and I am not entirely sure I can be sure the BeginPlay on the objects wanting to register before the Transmitter has done its __init__. So the question is, is it possible to code a Singleton that can if necessary transmogrify its own instance into an existing instance if snarfed via __init__()
|
# ? Aug 16, 2017 17:43 |
|
duck monster posted:Ok, so I have a bit of a puzzle. Have you seen this: https://forums.unrealengine.com/showthread.php?54343-Communication-Between-UE4-and-a-Python-UDP-Server ?
|
# ? Aug 16, 2017 18:16 |
|
I've not wrapped my head around how this UE4 python thing works (in fact, I don't anything at all about UE4), but what about using modules as singletons?
|
# ? Aug 16, 2017 18:21 |
|
You could just create the singleton and store it on the class in init and then have the instances proxy everything to it, or just access it through that attribute, or whatever you want. Maybe not ideal for performance in a game context but then neither is Python so... If it's really needed you can customize instance creation with __new__ but I don't have much practical experience with it. And as always, mucking about with the machinery should really be a last resort... breaks fucked around with this message at 20:51 on Aug 16, 2017 |
# ? Aug 16, 2017 20:40 |
|
Thermopyle posted:I've not wrapped my head around how this UE4 python thing works (in fact, I don't anything at all about UE4), but what about using modules as singletons? I've tried that, but it seems to be doing something a bit weird, where if I use module level variables in the class the class cant seem to find them, indicating to me that the UE4python thing seems to be das oing something a bit magical with the scope of the classes. I think I've found a solution though, as apparently I can run some scripts prior to scene start up, and thus can bootstrap up the dependencies for the scene there, including the transmitter object
|
# ? Aug 17, 2017 07:21 |
|
breaks posted:You could just create the singleton and store it on the class in init and then have the instances proxy everything to it, or just access it through that attribute, or whatever you want. Maybe not ideal for performance in a game context but then neither is Python so... I'm not too worried about performance. The vast majority of where a game is spending its time does is the game engine itself, and thats all speedy C++ under the hood stuff. Regardless, this is mostly just for pipeline and prototyping. If this idea I'm toying with works, I can always rewrite anything intensive in C++, and then run the rest through Cython.
|
# ? Aug 17, 2017 11:46 |
|
So speaking of pandas, I think it's really neat and I like how it can do things likecode:
Which got me wondering how this was possible. So rather than wondering I played around with and hacked together a rudimentary DataFrame like class. I just used standard [var]list[/var] objects to hold each array, and then learned about [var]__getitem__[/var] to access each array by name. I think I switched to numpy arrays at some point to get better functionality with them and it was super duper easy cause I just had to change the "backend" and all of my code/tests passed without changes to them. Now I wanted to be able to do something like this: code:
So I learned about slicing and other magic methods to do this. I don't actually have a point here but I think I learned a lot Coming from Fortran I think I'm starting to "get" some semblance of OOP and it's pretty rad.
|
# ? Aug 19, 2017 09:29 |
|
I'm looking for a python tutor and not really sure how to go about it. I'm taking a data science course and my lack of proficiency with python is my biggest weakness. So obviously I want to focus on data science/data analysis concepts, but also not-quite-so-directly related things like web scraping and working with APIs. I've read "Python Crash Course" and "Automate the Boring Stuff with Python" so I'm familiar with the basics but I tend to struggle with coming up with code on my own, or analyzing existing scripts. I live in Toronto, but this seems like the kind of thing that could be done via skype or discord or whatever. I guess(?) The catch is that I'm unemployed and receiving employment insurance, so I don't have a lot of cash to throw around. I'll try to work out something reasonable, regardless.
|
# ? Aug 22, 2017 03:12 |
|
Seventh Arrow posted:I'm looking for a python tutor and not really sure how to go about it. I'm taking a data science course and my lack of proficiency with python is my biggest weakness. So obviously I want to focus on data science/data analysis concepts, but also not-quite-so-directly related things like web scraping and working with APIs. Have you tried the IRC #python channel on Freenode?
|
# ? Aug 22, 2017 04:09 |
|
I know the convention is 3 spaces for rST and 4 spaces for Python, but if I'm documenting things in docstings for Sphinx do I stick with 3 or 4 spaces?
|
# ? Aug 22, 2017 07:47 |
|
accipter posted:Have you tried the IRC #python channel on Freenode? I haven't but I'll take a look into it, thanks!
|
# ? Aug 22, 2017 15:59 |
|
Any recommendations for intermediate/advanced books for Python automation and ops-type things? My team manages a data platform across several clusters hosted internally and in cloud and I'd like some ideas on building some self-healing into our systems and making our jobs easier.
|
# ? Aug 23, 2017 13:51 |
|
I figured I'd spin up a small web service that will give me the average of the last 10 days of my local aquifer's level, so I know when we'll go off of water restrictions (but mostly for my own education). I have two questions. 1) I got the parsing done relatively easily, but I feel like my code could be a little cleaner, particularly how I have to take multiple steps to get down to 10 floats from the list of tags. Any pointers here? Python code:
|
# ? Aug 26, 2017 19:24 |
|
(You should use requests, not urllib3...but probably not worth switching since you've already done this.) 1. Parsing HTML tables is usually weird and gross, so I also usually don't really like the code that does it either. I'm too lazy to actually look at the page and figure out if you're doing it the best. 2. What do you want on the web exactly? Each time you visit a url it should run your code to fetch the data? Do you want to run it once a day and then store the results for display any time during the day? "Web service" implies you want to serve it in a machine readable way like JSON...is that what you want or are you wanting to display a HTML site with your data?
|
# ? Aug 26, 2017 21:15 |
|
Yeah a lot of it's about knowing what BeautifulSoup can do and if it can make massaging your data easier, like maybe you can use getText(), that kind of thing I'd say your 'take 10' bit should come at the end though. Get a pipeline going converting your data, and only pluck off what you need when you're going to do your average or whatever. Use generator comprehensions if you don't want to process more than you need
|
# ? Aug 26, 2017 21:24 |
|
Thermopyle posted:2. What do you want on the web exactly? Each time you visit a url it should run your code to fetch the data? Do you want to run it once a day and then store the results for display any time during the day? "Web service" implies you want to serve it in a machine readable way like JSON...is that what you want or are you wanting to display a HTML site with your data? Shortly after I posted I thought it'd be sufficient to just send me an email once a day. That's a lot simpler, and I've already done a little with cron anyway. baka kaba posted:Yeah a lot of it's about knowing what BeautifulSoup can do and if it can make massaging your data easier, like maybe you can use getText(), that kind of thing The problem with doing that is there are elements that have the specified class which aren't numbers, so I can't convert all the elements first. I didn't know about generator comprehensions, though; I'll look into those. hooah fucked around with this message at 21:52 on Aug 26, 2017 |
# ? Aug 26, 2017 21:48 |
|
It's no big deal but you can provide a string as an argument to strip to tell it which characters you want to remove. So I'd probably do something like .strip(string.whitespace + "'") instead of .strip()[:-1] in a case like this. It's not really needed for this simple case but if you don't mind installing something, the tool I like for simplifying the extraction of poo poo from random strings is parse. It basically lets you write a format string style specification of what you want to pull out, so you'd replace your sanitization and conversion work with something like parse.search('{:f}', some_string). Anyway it's very handy if you do a lot of this kind of stuff. breaks fucked around with this message at 22:48 on Aug 26, 2017 |
# ? Aug 26, 2017 22:46 |
|
hooah posted:The problem with doing that is there are elements that have the specified class which aren't numbers, so I can't convert all the elements first. I didn't know about generator comprehensions, though; I'll look into those. You might want to do something like this Python code:
The other aspect is I used a few BeautifulSoup conventions to cut the code down - stuff like using property and method notation instead of typing 'find' and 'find_all' which cuts down on things and makes one-liners look nicer. I split out the actual 'convert cell contents into a number' bit into its own function, and the actual conversion step looks a bit nicer for it I think I threw in a generator expression for the hell of it (no need to create temporary lists everywhere) but it doesn't really matter here, and I did a list in the end anyway because the slice notation reads nice and python doesn't have a nice clean take(10) function yet, so whatever. Generator functions are nice for a functional pipeline, you basically set up how sequences are handled and you can do filtering and stuff at each step, so for big data you can avoid putting everything in memory. Doesn't really matter here but it's good to know about!
|
# ? Aug 27, 2017 00:12 |
|
I'm writing an implementation of a data structure, a graph used for storing words for extremely fast searching, along with an interface for actually using it. I've reached a point where it works correctly in the sense that it gives the outputs I expect when using the interface (verified with tests), but may not necessarily be optimal in terms of time/space, the internal implementation may change in future in the name of efficiency. Is it acceptable to spin off the current implementation as a sort of reference implementation to test any future iterations against? Or is that some sort of faux pas? EDIT: I'm a moron. When I started writing tests I was thinking "This would be so much quicker if I could just compare to a reference implementation, rather than having to work out the expected outputs myself". But in writing the tests to verify this current implementation works so that I could use it as a reference implementation, I've done exactly that ArcticZombie fucked around with this message at 10:32 on Aug 28, 2017 |
# ? Aug 28, 2017 10:06 |
I've now been working strictly in golang for about three months. Having stepped away from Python, here the three things I miss the most: 1. Set type as a primitive 2. Comprehension literals (especially generator comprehensions!) 3. Context managers ( with statement) Python is a good language.
|
|
# ? Aug 28, 2017 16:24 |
|
I enjoy my C#, but I really think python3 is the most well put together programming language today. If something is terribly complex to express in python, I'm definitely thinking too hard about it.
|
# ? Aug 28, 2017 18:06 |
|
The Coding Horrors thread killed any mild interest I had in golang
|
# ? Aug 28, 2017 20:28 |
|
same
|
# ? Aug 28, 2017 20:36 |
|
Eela6 posted:I've now been working strictly in golang for about three months. QuarkJets posted:The Coding Horrors thread killed any mild interest I had in golang Thermopyle posted:same I'm in a similar boat to Eela6. Most of my career has been Python stuff but the past few months has been very Go-dominated, because I've been working on various things related to Kubernetes. I dislike Go less now than when I started, but there's nothing specific to Go that I wish I could take back to Python. The best feature of Go isn't a feature of the language, it's that your program compiles to a static binary. If Python had enforcement on type hints and a static binary generator that wrapped up cx_freeze/py2exe/py2app into pythonc, would Go have ever left the launchpad?
|
# ? Aug 28, 2017 21:44 |
|
Our secondary product at work is written in go (it's particularly well suited for concurrency workflows), it's a real bear to work with, we have a team of six writing customizations for it etc etc. It's probably 40,000 loc at this point. One of our sales engineers rewrote the whole thing in python in a couple of hours as a proof of concept, maybe 270 lines, and now we're running in to tech debt hell with the go product, so fleshed out the python version a bit more and comes in at 1200 lines, plus it has five years of hindsight to draw from so it's a lot more maintainable at this point. We're still testing the Python version internally but there's a good chance it may replace the five year old Go product by the end of the year.
|
# ? Aug 28, 2017 22:01 |
|
chutwig posted:If Python had enforcement on type hints and a static binary generator that wrapped up cx_freeze/py2exe/py2app into pythonc, would Go have ever left the launchpad? This isn't what you're talking about, but you reminded me that I read about it earlier today. quote:Nuitka is a Python compiler.
|
# ? Aug 28, 2017 22:43 |
|
Memory usage in a data structure I've been working on (pyGADDAG) is far higher than I'd like it to be, so I've been trying some different memory profiling modules and I came across objgraph, which has some pretty handy tools for visualising my data structure. Here's a graph it produced for a GADDAG storing the word "careen": It looks pretty much exactly how I expected it to look, except for the references leading to the bottom right node. How does the dictionary of the node before it have many references for the key "n"? Python dicts can't have duplicate keys can they? Adding the word "care", a prefix of "careen", to the GADDAG: Not likely to be the cause of my memory usage, but is this some quirk of objgraph or is there something else causing that?
|
# ? Aug 31, 2017 20:27 |
|
|
# ? Jun 3, 2024 22:40 |
|
Got a new laptop, any favorite stickers to pursue?
|
# ? Sep 1, 2017 18:34 |