Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›18 »

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Whybird posted:

but having the data held in this way will presumably mean that I don't have to search through the entirety of the visibility data each time I need to iterate through one player's visible entities or one entity's viewing players.

I'm not speaking directly to you as it sound like you're in a state where you're not even sure what makes the most conceptual sense, but...as a general rule you should use the data structure that makes the most sense conceptually and then if you hit a performance issue consider changing it.

# ¿ Jan 19, 2017 15:54

Adbot: ADBOT LOVES YOU

# ¿ May 9, 2024 16:59

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Where does one even get CSV-formatted data nowadays? Excel exports?

# ¿ Jan 23, 2017 22:53

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Cingulate posted:

I write all my behavioral experimental result files in what amounts to CSV format, what'd be wrong with that?

I don't know that there's anything wrong with it other than the pain that comes from dealing with CSV...but every data interchange format has some sort of pain. I was just asking because I'm curious...it's been a decade since I've had to consume data in CSV.

# ¿ Jan 23, 2017 23:17

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

KernelSlanders posted:

I'm curious what format Thermopyle is consuming data in.

Almost always pulled from a database.

# ¿ Jan 24, 2017 07:46

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I wonder how well passing around a sqlite database file would work. Because, man, SQL is awesome. I don't really know how well the performance would work out if you need to slurp it all into memory or something...

# ¿ Jan 24, 2017 20:54

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

LochNessMonster posted:

I'm still playing with requests and bs4 and am running into an issue I'm not sure how to solve.

The html page I contains something like this:
HTML code:
div class="out-of-stock" qtlid="12345">
                                        Currently not in stock
                                        <br><span style="display: none;" qtlid="67890">
                                            Last in stock: {value}.
                                        </span>
</div>
When I visit the page in a browser I see a date / time where the html code says {value}. I assume this is something that should be filled by a JavaScript. Looking at the source I probably know (the path to) the script whch does this.

Is there an easy way to let my python app run the specific JavaScript so I can scrape the date/time?

The best way to do this is look at that javascript and see what it doing to get the date/time and then do that with python. So, maybe it's doing an AJAX request to some url, and you can do this with requests in python. This often fails because you've got to have specific cookies or something else the page sets up in its environment and it's a bitch to reverse engineer.

If it's too complicated, you can try PyExecJS, but that often fails because it's not running in a browser environment. Usually what I do when I get to this point is use PhantomJS with Selenium to do what I need to do...that usually means throwing away all your BS4 and requests code and doing it all in PhantomJS.

# ¿ Jan 25, 2017 20:16

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Munkeymon posted:

You can also have Phantom/Slimer save the rendered DOM as HTML to disk pretty easily if there's no complicated login process to get to it.

Oh yeah, I forgot about this...which is funny because just last week I did it.

In all honesty, if a page has any javascript doing fetching or anything I just always go to using Phantom because it's just easier than trying to figure out wtf the page is actually doing. Of course, that comes with the downside of it being more resource-intensive, but that usually doesn't matter too much for me.

# ¿ Jan 26, 2017 19:13

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

LochNessMonster posted:

Thanks for the additional info. Phantom and Slimer appear to be JS. I'm still getting to know my way around Python so I'll just stick with that for now.

As baka kaba says and as I mentioned in my first post replying to you, you use Selenium to control Phantom. No need to use JS at all unless you need to do some more advanced stuff.

I actually kind of enjoy scraping with Phantom+Selenium in some ways. I've got a project now where I'm scraping a very JS-heavy site and I've created a classes where I can do stuff like:

Python code:

>>> page = PhantomAndSeleniumBackedPageAbstraction()
>>> page.fill_form(foo=1, bar=datetime.datetime.now())
>>> page.submit_form()
>>> print(page.form_results)

Where the class instance takes care of abstracting away a lot of messy poo poo you've got to deal with on this particular page...including running JS in Phantom because this page is impossible and stupid and I can't do exactly what I want from Selenium itself.

Anyway, sounds like you don't need to go down this path yet, but I'm just letting you know.

# ¿ Jan 26, 2017 21:37

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

socialsecurity posted:

We are having someone send their address via a http wordpress form the info gets posted via j query Ajax to the python script, the python then takes the address info and runs it through a series of pre-made apis that it's supposed to send the info from the apis back. We are using python because that's what the api sending code was written in originally. So far the former sends the data to the python script which does process the data right if we pipe it to the console everything looks as it should just can't get it to respond properly to the Ajax post request.

I'm pretty sure your life would be a lot easier with Flask or Django here.

# ¿ Jan 28, 2017 23:27

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

socialsecurity posted:

I've started looking into this but it seems it would require quite a bit of rewrite I didn't think sending a json back to a html post would be this much of an undertaking but thank you I think in the end going down this path would be easier to work on down the road.

http is a bitch with a bunch of weird edge cases and whatnot.

It's just way easier to use a library that someone else dealt with all this crap.

# ¿ Jan 28, 2017 23:47

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Also, both Flask and Django are perfectly capable of doing something super simple like you mentioned, but their getting started docs both assume you're wanting to do something more complicated so...that sucks for you. Just keep powering through and you're going to know stuff that is useful to know...

# ¿ Jan 28, 2017 23:48

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

socialsecurity posted:

This makes sense more then other options I've seen up until a point, why does it look like I'm running another server to serve this out this is already hosted/exposed via my apache does it have to be hosted on its own to return properly?

https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface

# ¿ Jan 29, 2017 06:25

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I like to engineer the most impractical and complicated solution possible.

In this case I'd set up celery workers and celery beat as daemons and then configure a periodic task to run.

^{(no i wouldnt i'm just bitter that I worked on fixing up a project that did this once for just one little periodic thing instead of just using cron)}

# ¿ Jan 30, 2017 22:05

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

accipter posted:

What are peoples thoughts on unicode variable names (e.g., Φ)? I do a lot of programming of equations that might have variables without physical meaning (no proper name) and are just a greek symbol. I will usually use an ascii representation of that variable because it is simpler to enter, but it is a little removed from the original reference.

I dunno...if I had to figure out how to type such a thing when working with your code, I'd hate you forever.

However, I rarely do anything requiring me to type anything other than whats on my US keyboard. Maybe if your code is only ever going to be used by math people who routinely use unicode math symbols it'd be ok?

# ¿ Feb 1, 2017 19:01

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I'd passive-aggresively go and change every variable called "unused" to "_" because goddamn it's a super common convention and whether or not you realize it or not conventions matter...a lot!

# ¿ Feb 1, 2017 21:51

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Like...what do you gain by using the word "unused"?

As far as I can tell the only thing you gain is readability by super-new programmers at the expense of making it harder to read for everyone else.

# ¿ Feb 2, 2017 00:20

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

QuarkJets posted:

Note that _ isn't mentioned in PEP8 anywhere, I think it's the kind of convention that just kind of spread by word of mouth and is generally safe to ignore for those who want to ignore it

That's because it's a programming convention, not a Python convention. It's used in probably dozens of languages.

Hammerite posted:

I've dragged this argument (in which I am clearly in the minority) out for too many posts already, but... I don't see that it does make it harder to read for anybody, irrespective of their level of experience.

Because there's more cognitive overhead in reading a word vs recognizing an underscore. But mainly because everyone knows what a _ means, and then you come across some guy wanting to use "unused" and then you don't know wtf is going on. Did this guy really not know that everyone uses "_"? Maybe he did, so now maybe he means something different by "unused"?

# ¿ Feb 2, 2017 17:25

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Hammerite posted:

I disagree. There is no more cognitive overhead. However, I will recognise that whether one thing presents more, less, or same amount of cognitive overhead compared to another is down to individual perception, so what presents overhead to you may not present it to me, and vice versa.

You seem to be unaware that there is psychological literature on the subject pointing out that there is an objective reality on symbol vs word recognition and it's not just a personal preference thing, but as you're not interested in continuing the discussion (and TBH, it's not really worth me digging up the cites) I won't go into it.

# ¿ Feb 3, 2017 00:38

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

QuarkJets posted:

Okay you're exaggerating though, right? If you saw someone assign something to a variable named "unused" surely you wouldn't clutch your head in confusion, instead you'd probably think "oh this variable must not be used by anything" regardless of your experience level

No, I'd think "this variable is probably unused, but I can't be sure of that because everyone knows you use '_' for unused variables". Then, I'd either have to trace code to see if it's really unused or if it's a boolean for something in the code that indicates whether the business logic needs to know if something is unused, or I'd just hope it's just some rear end in a top hat bucking convention on principle or some newbie who doesn't know the convention...and even in the latter case I'd still never be sure until I actually traced through the code.

People can make all sorts of arguments about whether '_' is appropriate or not based upon other usages of '_' in Python...but it doesn't matter, '_' won and everyone knows what it means when you see something assigned to it. It's 100% clear unless you're a newbie. 'unused' or whatever is not.

# ¿ Feb 4, 2017 18:41

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Nippashish posted:

Its almost like clarity of different conventions depends on which conventions you're used to seeing and using yourself.

Nah, there's definitely a universal objective answer.

I don't thins there's a universal truth, but then again that doesn't have anything to do with my point.

Are there code bases where unused is used instead of_? Sure, probably.

Is that the norm, or you might say...the convention? No. And that means the code base has extra on-boarding costs for additional developers.

# ¿ Feb 4, 2017 19:44

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

QuarkJets posted:

This. Even just in this thread we've seen several posters who had either not seen the "_ as unused" convention, had seen a similar but different convention, or who knew of the convention but don't use it because they think that a different convention is better. These all seem like okay states to be in and I don't see anything wrong with someone using an alternative if they think that's better, especially when that alternative is already recognized in pylint

We are talking about something that is usually inconsequential (variable name) that contains something that is definitely inconsequential (unused return value) so maybe it's just more important that a codebase is internally consistent than that it aligns with an unstated convention ported over from other languages?

I think this is the first time this argument has been brought up against the "_" in this discussion which seems odd if it's the best argument or right-est argument.

I also don't think it's an accurate characterization of the ubiquity of '_', nor accuracy of pylint. But pinning down both of those things is the only way to go forward with this conversation, and I certainly don't think that's an easy thing to quantify, I don't guess there's anywhere else to go because I certainly am not interested in spending the time quantifying the ubiquity of '_', nor whether pylint's rules say anything about that ubiquity.

Regardless, I don't think anyone would argue against staying consistent with an already existing codebase...using a style different from the existing codebase is even worse than not using '_'.

Thermopyle fucked around with this message at 02:45 on Feb 5, 2017

# ¿ Feb 5, 2017 02:42

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

In other news I've been spending some time recently converting a bunch of bash scripts to python because gently caress bash.

I came across the plumbum library that's pretty neat for easing this type of work.

code:

>>> from plumbum import local
>>> ls = local["ls"]
>>> ls
LocalCommand(<LocalPath /bin/ls>)
>>> ls()
u'build.py\ndist\ndocs\nLICENSE\nplumbum\nREADME.rst\nsetup.py\ntests\ntodo.txt\n'
>>> notepad = local["c:\\windows\\notepad.exe"]
>>> notepad()                                   # Notepad window pops up
u''                                             # Notepad window is closed by user, command returns

code:

>>> chain = ls["-a"] | grep["-v", "\\.py"] | wc["-l"]
>>> print chain
/bin/ls -a | /bin/grep -v '\.py' | /usr/bin/wc -l
>>> chain()
u'13\n'

...and a ton of other features.

# ¿ Feb 5, 2017 02:59

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

QuarkJets posted:

Pylint recognizes other conventions as being okay, why won't you?

I'm not sure you understand what you highlighted in my quote means?

I mean...using your own style in a code base is bad and worse than using a less popular convention for an unused variable. I'm not sure what's controversial about that.

HardDiskD posted:

I mean, gently caress bash and that's a really sexy API, but I feel like this is just hiding the bash under a Python rug and calling it a day.

That's kind of what we do as programmers, no?

Nippashish posted:

This is neat. I've used sh for this in the past, but plumbum looks a bit more featureful. Does it handle piping properly (i.e. hooking stdin/stdout of the processes together at the OS level) or does it shuffle everything through python? I've used a few shell replacement libraries that do the later and it can be a real show stopper if you need to move a large amount of data around.

A bit of fuckery is worth it though, just to avoid bash's insane string quoting rules.

A quick perusal of the source makes it look like it uses OS piping via popen, but I'm not 100% confident in that. https://github.com/tomerfiliba/plumbum/blob/1441c5ae7dedc2f40aaf37207ff2585557c5746b/plumbum/commands/base.py#L266

Tigren posted:

That does seem fun, but why not learn the Standard Library os module and rely on subprocess when needed? They're already included for you.

I kind of feel like the answer to this is similar to the answer I gave HardDiskD. The only reason to use any library is if it makes your job easier...after all you can implement anything you want yourself. You could use C instead of Python. You could use assembly instead of C. It just might be easier to implement the functionality you need by using subprocess, or it might not. I've used subprocess extensively, but it's certainly not as concise as this particular library. This library seems to make it easier to express what you intend.

# ¿ Feb 6, 2017 00:37

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Anyone used uvloop as a replacment for the async event loop?

I've got an async-using application that is going to be approaching some limits in requests per second and I'm researching some of my alternatives and uvloop looks promising...

# ¿ Feb 10, 2017 20:30

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

config.py ftw.

Also, like Nippashish, I'll use something to make it where I can use dot-notation rather than accessing keys because I just don't like accessing keys. Feels and looks ugly.

# ¿ Feb 25, 2017 22:21

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I've got a new thread for Python pretty much ready to go from a couple years ago, I just never got around to posting it.

I'll clean it up and post it by the end of the week.

edit: Well, I just read it over and its kind of dated and actually from the end of 2013. If someone wants to start with it as a base here it is. There's good suggestions in the comments on that page for improvements.

Thermopyle fucked around with this message at 22:16 on Feb 27, 2017

# ¿ Feb 27, 2017 22:11

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Boris Galerkin posted:

Do I use 3 or 4 spaces in my docstrings if I'm using sphinx?

e: logging chat:

If I use a third party package to build an app, and that third party package has a logger already set up, should I just hook into their logger or should I need to create my own and keep them separated? The info that their logger outputs is useful to me; my extra log messages are just additional things I've added for my app.

Create your own logger and logging config.

# ¿ Mar 4, 2017 17:25

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

I'm going to do it if y'all don't.

Don't forget that google doc i posted. If you want to use it, it should mostly be copy/paste with some updates for modern python.

# ¿ Mar 5, 2017 22:39

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Boris Galerkin posted:

Should also put in big bold letters at the top when the OP was last updated and keep it updated. IMO the problem with these huge threads/ops is that I click one one and see it posted in 20xx and think "hmm okay I'm just gonna assume everything is outdated."

Just look at the last edited date?

# ¿ Mar 6, 2017 14:31

Adbot: ADBOT LOVES YOU

# ¿ May 9, 2024 16:59

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Hammerite posted:

I know that and you know that, but it doesn't send out the message that the op is keeping the thread up to date (or, if it's a date far in the past, that they aren't doing so). Basically it adds confidence on the part of the reader who might not think to check the last edited date.

Yes, I was giving the poster advice on how to solve the problem with threads that don't have what he asked for.

# ¿ Mar 6, 2017 15:29

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›18 »