Nippashish posted:

You could also just commit to not import your scripts as modules and then you don't need any double underscore shenanigans.

Cock Democracy posted:

I'm looking for some advice on how to refactor a giant ETL/data pipeline script. Here's a summary of what it does currently:

- Download files from various partners (some use FTP, others are HTTP)
- Load the downloaded files into MySQL temp tables
- De-dupe records (some records appear in the feeds of multiple partners)
- Take the combined records, create a CSV and upload that to an API to get some additional data, then parse the results
- Prepare the final output as MySQL tables + Elasticsearch index used by the Django app
- Verify that the new record counts are reasonable, then put the final output on production
- Save logs to filesystem and a table

The requirements are nothing too complicated, but the code now is a bit of a mess. It's all procedural, has tons of exceptions and hacks for certain partners, and is generally just not very extendable. When it fails, our devs have a hell of a time debugging it. It's sort of fallen under my ownership and I'm not happy about its current state.

Are there any libraries or PEPs to help guide me in rewriting this beast? My thought is I'd like a setup like Django migrations, where each "step" is its own file, containing a class that follows a standard format. Then there could be a file that lists all the classes and what order they run in. Hell, some of the steps could run concurrently, so it would be awesome to support that.

Dominoes posted:

Does anyone else dislike the if __name__ == __main__ syntax? I still have to look it up every time.

Thermopyle posted:

I just type "main" and press TAB and Pycharm types it all for me.

keyframe posted:

Guys I was wondering what is a good way to see how long it takes a python script to run and how much memory/resources it is using?

I just discovered timeit but all the examples I have seen for that seems to be run on single line comments with some weird have everything enclosed in a string quote syntax.

cinci zoo sniper posted:

Official cProfile is a good start.

E: For memory you can try package called memory-profiler.

KICK BAMA KICK posted:

Just to make sure, cause this behavior could actually be useful to shorten some conditionals into one-liners if it wasn't bananas: everyone would consider it extremely weird and unPythonic to access an item of a two-item sequence using a Boolean value silently evaluated to 0 or 1 as the index, i.e. 'ab'[False] == 'a' or 'ab'[True] == 'b' right?

KICK BAMA KICK posted:

Yeah that was just an illustration, the case I'm thinking of is more like

code:

if b:
    x = foo
else:
    x = bar

being condensed into

code:

x = (bar, foo)[b]

That doesn't raise any eyebrows if b is clearly a bool?

breaks posted:

QuarkJets posted:

What breaks wrote is the pythonic way to do this. Creating a tuple just to immediately index into it... that's no good

NtotheTC posted:

I've seen that pattern used once in the wild, where it was something like

Python code:

string_from_bool = ("falsily", "verily",)

>> print(string_from_bool[False])
falsily

>> print(string_from_bool[True])
verily

cinci zoo sniper posted:

loving hell, why?!

NtotheTC posted:

Well that example is slightly simplified, the two strings were actually entire sentences, and were stored in a config file and often accessed, so it was neater than writing

Python code:

my_string_response = STORED_TRUTHY_STRING if True else STORED_FALSEY_STRING

or similar, i dunno it was several jobs ago but I never found it particularly egregious

Nippashish posted:

It should be a dictionary.

NtotheTC posted:

I'm not trying to nominate it for code of the year, just pointing out a real-world example of it

keyframe posted:

Hi guys,

I am trying to scrape craigslist for apartment prices. I am having a problem where when I scrape the prices in <span class="result-price">$xxxx</span> I get a lot of duplicates which poo poo on my plan on making a urllink/price dictionary with zip(). Any idea why the duplicates are happening? I am using requests/beautiful soup with html.parser.

craigslist link if anyone wants to try themselves:

https://vancouver.craigslist.org/d/apts-housing-for-rent/search/apa

KICK BAMA KICK posted:

At a glance, it looks like each listing's "result-price" is repeated under a <span class="result-meta"> tag. Grab your data only once per entry from either that or its parent.

keyframe posted:

Ahh I see what you mean. I have no idea how to do that though, any pointers?

KICK BAMA KICK posted:

Don't remember the BeautifulSoup syntax off the top of my head (and also just glanced at the page source the one time) so this is all approximate but instead of searching for all class="result-price" tags on the page you'll want to start by searching for only the the class="result-price" tags that are children of class="result-meta" tags. I put it that way cause I'm pretty sure you can find the tags matching both those conditions in one search and then iterate over those to process them the same way you were doing before; if you can't do that or if you don't follow search for all the class="result-meta" tags, iterate over those and find the one class="result-price" tag underneath each. Poke around in the BeautifulSoup docs a little bit, this should be pretty easy.

I guess the absolute simplest way if you want to immediately solve the problem that might get the correct result but is not a great way to do it would be to step through the list of results you've retrieved with whatever line searches for those "result-price" tags by twos with a [::2] slice.

breaks posted:

Just use the soup's select method with a ".result-meta>.result-price" selector and call it a day. IMO always pick elements using CSS selectors up until they don't support what you need, that's what they were made for.

(On that page, there 221 matches for .result-price but 120 for .result-meta>.result-price. There are 120 items per page and at first glance the lower than expected number of .result-price results seems to be due to posts lacking pics only having the price displayed once? or something like that.)

Malcolm XML posted:

Airflow is the worst out there except for everything else

Luigi doesn't do scheduling just sequencing Iirc

Godspeed goon.

Thermopyle posted:

Humble Bundle has a bunch of python books.

Bundy posted:

Also hi, thread newcomer

Bundy posted:

Packt currently have an everything for five bucks deal going (Packt usually provide the Humble Python books) if you want to cherry pick. Also hi, thread newcomer

Some resources I often find useful that I didn't notice in the OP:

Writing Idiomatic Python:
https://jeffknupp.com/writing-idiomatic-python-ebook/

This guy is great, if you're poor just write to him and he'll give you a copy of the book for free.

Nedbat's Big O presentation:
https://nedbatchelder.com/text/bigo.html

Nedbat's talk on Pragmatic Unicode:
https://nedbatchelder.com/text/unipain.html

The Clean Architecture in Python (also a nice watch for anyone interested in FP approaches in Python):
https://www.youtube.com/watch?v=DJtef410XaM

If you don't like configparser and/or YAML, consider TOML:
https://github.com/toml-lang/toml

I'll share more as I come across them in my bookmarks. I've worked with Python (3) for about five years now and I figured I'd see if there was anything goon related