Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

LochNessMonster posted:

Thanks, that sounds like exactly what I am looking for.

Is this the kind of stuff I�d learn from reading Fluent Python?

It's just math.

Fluent Python will teach you about the python structures you use to implement the math...

# ? Aug 2, 2018 19:38

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 10:41

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

baka kaba posted:

Or if you just want to check it's close to all the values
Python code:
all(abs(a-b) < tol for b in query)
uses a generator expression instead of building a list so all can quit early when it hits a false value, without having to calculate all the remaining values. Might be more efficient if your query results are long (or you're doing a lot of this)

(In case you don't know generator expressions are like list expressions but inside ( ) instead of [ ], but because it's already the only thing in all's parentheses you can skip them)

I really should look into generators at some point. I never use them.

e:

I have n lists of bools and I want an nth + 1 list to be built where it's True if the same elements for the other lists are all True and false otherwise.

code:

a = [True, True, True]
b = [True, False, True]
c = [False, True, True]
d = [False not in vals for vals in zip(a, b, c)]  # [False, False, True]

This works but is there an easier or built-in way to do this?

Boris Galerkin fucked around with this message at 08:21 on Aug 3, 2018

# ? Aug 3, 2018 06:23

dougdrums: Feb 25, 2005; CLIENT REQUESTED ELECTRONIC FUNDING RECEIPT (FUNDS NOW)

Python code:

import operator as op
map(op.and, zip(a, b, c))

I'm not exactly sure what is being specified though. I think you just want to and the two lists itemwise because bool([ False, False, False ]) is True. Ofc you can just any() the result if that is the goal.

E: actually am I wrong about this? I think op.and does bitwise and? Are there not boolean operations in operator?

When I do dir(op) in 3.7 it doesn't even list it ...

This works though! Just a different name... which is why I suppose operator has no boolean and. I feel kinda silly for mentioning any but missing this ...

Python code:


map(all, zip(a, b, c))

dougdrums fucked around with this message at 09:59 on Aug 3, 2018

# ? Aug 3, 2018 09:40

LochNessMonster: Feb 3, 2005; I need about three fitty

Boris Galerkin posted:

I don�t know that book but the abs(a-b) < tol part to find a distance in both answers is just math. You learned it when you learned about number lines

Apparently my math completely sucks. I�d expect this to check if a-b is smaller than 10. That�s not checking both ways right?

I�d have to come uo with something like: abs(a-b) < tol or abs(b-a) < tol right?

I didn�t know about generator expressions and they seem a lot more efficient than looping through lists all the time.

baka kaba posted:

Or if you just want to check it's close to all the values
Python code:
all(abs(a-b) < tol for b in query)

I am looking if just 1 value is close so this helped me find out about any(). Thanks.

LochNessMonster fucked around with this message at 10:22 on Aug 3, 2018

# ? Aug 3, 2018 10:19

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

dougdrums posted:

I think you just want to and the two lists itemwise because bool([ False, False, False ]) is True.

Nah I wanted the list in my example comment (d = [False, False, True]).

Python code:

map(all, zip(a, b, c))

That works but only if I convert into a list,

Python code:

list(map(all, zip(a, b, c)))

# ? Aug 3, 2018 10:21

Linear Zoetrope: Nov 28, 2011; A hero must cook

LochNessMonster posted:

Apparently my math completely sucks. I�d expect this to check if a-b is smaller than 10. That�s not checking both ways right?

I�d have to come uo with something like: abs(a-b) < tol or abs(b-a) < tol right?

I didn�t know about generator expressions and they seem a lot more efficient than looping through lists all the time.

No, without the absolute value you'd need a-b < tol or b-a < tol, but the absolute value removes any negative sign so you only need to do the one comparison.

# ? Aug 3, 2018 10:23

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

LochNessMonster posted:

Apparently my math completely sucks. I�d expect this to check if a-b is smaller than 10. That�s not checking both ways right?

I�d have to come uo with something like: abs(a-b) < tol or abs(b-a) < tol right?

I didn�t know about generator expressions and they seem a lot more efficient than looping through lists all the time.

It's checking both directions because you take the absolute value of the result.

If a = 10 and b = 15, then

a - b = -5
b - a = 5

but

abs(a - b) = 5 = abs(b - a)

and 5 is the distance between the two numbers.

# ? Aug 3, 2018 10:24

dougdrums: Feb 25, 2005; CLIENT REQUESTED ELECTRONIC FUNDING RECEIPT (FUNDS NOW)

It's generally called absolute difference, for the sake of searching. Just didn't see it mentioned.

quote:

Nah I wanted the list in my example comment (d = [False, False, True]).

Haha ya ofc, it's early here.

Python code:

list(map(all, zip(a, b, c)))

[/quote]
I've been wondering if it is generally considered "more proper" to do this or the comprehension syntax.

Comprehensions are cool and good but I'm really annoyed when I have to come up with a name every time.

dougdrums fucked around with this message at 11:50 on Aug 3, 2018

# ? Aug 3, 2018 11:32

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

I've finished writing some code and am struggling to get to import. I'm using PyCharm if it matters.
The project structure looks like
https://imgur.com/5M75kfR

I told it to create a setup.py, the contents of which is

Python code:

from setuptools import setup

setup(
    name='banking_scrape',
    version='0.13',
    packages=['banking_scrape', 'banking_scrape.banks'],
    url='',
    license='',
    author='obow',
    author_email='',
    description=''
)

If I tell PyCharm to run setup.py task, and ask it to make a wheel, it creates one.
I make a new project, use the command line to pip install it and in the venv it shows it in site-packages (a banking_scrape folder and banking_scrape-0.13.dist-info folder).
However, if I do import banking_scrape it succeeds, but nothing I import seems usable.

code:

print(dir(banking_scrape))
print((banking_scrape.__path__))

['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
['/Users/obow/venv/untitled1/lib/python3.7/site-packages/banking_scrape']

If instead I use from banking_scrape import *, it gives nothing either.

My 2 __init__.py files are empty, but that seems to be normal? I'm so confused.

# ? Aug 3, 2018 13:08

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

~~What happens when you import banking_scrape.banks?~~

I think that when you import banking_scrape it will execute the __init__.py file and since your __init__.py is blank nothing gets imported. You probably need to add in your __init__.py file e.g., from .core import * and so on.

You also don't need to add banking_scrape.banks as a separate package. In the __init__.py file for it you can do the same from .virgin import *. In the top level __init__.py you can then just do from .banks import *.

e: If there are name clashes in those bank files then you could just do

code:

# banks/__init__.py
import virgin
import tesco

# __init__.py
import banks

This would let you do

code:

import banking_scrape

banking_scrape.banks.virgin.do_something()

Boris Galerkin fucked around with this message at 13:37 on Aug 3, 2018

# ? Aug 3, 2018 13:24

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

Boris Galerkin posted:

~~What happens when you import banking_scrape.banks?~~

I think that when you import banking_scrape it will execute the __init__.py file and since your __init__.py is blank nothing gets imported. You probably need to add in your __init__.py file e.g., from .core import * and so on.

You also don't need to add banking_scrape.banks as a separate package. In the __init__.py file for it you can do the same from .virgin import *. In the top level __init__.py you can then just do from .banks import *.

e: If there are name clashes in those bank files then you could just do
code:
# banks/__init__.py
import virgin
import tesco

# __init__.py
import banks
This would let you do
code:
import banking_scrape

banking_scrape.banks.virgin.do_something()

If I put from .banks import * in __init__.py it throws an error.

code:

ModuleNotFoundError: No module named '__main__.banks'; '__main__' is not a package

Process finished with exit code 1

If I put from banks import * in, and import it says...

code:


  File "/Users/obow/venv/untitled1/lib/python3.7/site-packages/banking_scrape/__init__.py", line 1, in <module>
    from banks import *
ModuleNotFoundError: No module named 'banks'

On the off chance it helps to see the code itself, it's on Bitbucket - https://bitbucket.org/dluther23/bankscrape/src/master/

edit -

If I change __init__.py to

Python code:

from banking_scrape import *

__all__ = ['core', 'db_connect', 'mail', 'overview', 'security', 'todo_log']

and then have my main.py as

Python code:

import banking_scrape.overview
scraper = banking_scrape.overview.BankScraper()

It throws the error...

code:

  File "/Users/obow/PycharmProjects/untitled1/home.py", line 1, in <module>
    import banking_scrape.overview
  File "/Users/obow/venv/untitled1/lib/python3.7/site-packages/banking_scrape/overview.py", line 2, in <module>
    import security
ModuleNotFoundError: No module named 'security'

Which I can fix by going into overview.py and changing it to import banking_scrape.security as security

But that feels very much like not what I'm supposed to do.

Sad Panda fucked around with this message at 14:48 on Aug 3, 2018

# ? Aug 3, 2018 14:26

LochNessMonster: Feb 3, 2005; I need about three fitty

Boris Galerkin posted:

It's checking both directions because you take the absolute value of the result.

If a = 10 and b = 15, then

a - b = -5
b - a = 5

but

abs(a - b) = 5 = abs(b - a)

and 5 is the distance between the two numbers.

drat this is getting pretty embarasssing...

I guess I need to start spending some time on codewars/hackerrank.

# ? Aug 4, 2018 09:19

pmchem: Jan 22, 2010

LochNessMonster posted:

drat this is getting pretty embarasssing...

I guess I need to start spending some time on codewars/hackerrank.

Speaking of those type of sites, which do people like best for Python? Preferably with useful exercises for scientific/numerical programming?

# ? Aug 4, 2018 14:36

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

pmchem posted:

Speaking of those type of sites, which do people like best for Python? Preferably with useful exercises for scientific/numerical programming?

I probably hear project euler mentioned the most, and since its math-focused is probably the most appropriate for the discussion at hand. Not sure how similar you want to call it to hackerrank or the like...

LochNessMonster posted:

drat this is getting pretty embarasssing...

My math is very weak so I feel your pain! I've been out of school for close to two decades with very minimal usage of any maths since then.

# ? Aug 4, 2018 15:32

Dominoes: Sep 20, 2007

Thermopyle posted:

My math is very weak so I feel your pain! I've been out of school for close to two decades with very minimal usage of any maths since then.

It's not too late! This class starts at the end of Aug

# ? Aug 4, 2018 16:11

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Fine, I'll do it!

Though, I might find a precalc course to do first...

# ? Aug 4, 2018 16:27

LochNessMonster: Feb 3, 2005; I need about three fitty

Dominoes posted:

It's not too late! This class starts at the end of Aug

I�m not an english native. Math is difficult enough in my primary language. Doing it in english makes it even harder.

I should give it a shot though...

# ? Aug 4, 2018 16:53

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Sad Panda posted:

If I put from .banks import * in __init__.py it throws an error.
code:
ModuleNotFoundError: No module named '__main__.banks'; '__main__' is not a package

Process finished with exit code 1
If I put from banks import * in, and import it says...
code:
  File "/Users/obow/venv/untitled1/lib/python3.7/site-packages/banking_scrape/__init__.py", line 1, in <module>
    from banks import *
ModuleNotFoundError: No module named 'banks'
On the off chance it helps to see the code itself, it's on Bitbucket - https://bitbucket.org/dluther23/bankscrape/src/master/

edit -

If I change __init__.py to
Python code:
from banking_scrape import *

__all__ = ['core', 'db_connect', 'mail', 'overview', 'security', 'todo_log']
and then have my main.py as
Python code:
import banking_scrape.overview
scraper = banking_scrape.overview.BankScraper()
It throws the error...
code:
  File "/Users/obow/PycharmProjects/untitled1/home.py", line 1, in <module>
    import banking_scrape.overview
  File "/Users/obow/venv/untitled1/lib/python3.7/site-packages/banking_scrape/overview.py", line 2, in <module>
    import security
ModuleNotFoundError: No module named 'security'
Which I can fix by going into overview.py and changing it to import banking_scrape.security as security

But that feels very much like not what I'm supposed to do.

In your overview.py file, you need to change your local imports from other modules from

code:

from core import CoreTools
import security
from db_connect import DatabaseInterface
from todo_log import ToDoLog

code:

from .core import CoreTools
from . import security
from .db_connect import DatabaseInterface
from .todo_log import ToDoLog

I'm not really sure why to be honest (read: I'm too lazy to google why) but that's what works for me and those are the examples I see in the official documentation. If I make those changes and also changing the __init__.py file to what you wrote in the post then I am able to do

code:

$ python
Python 3.7.0 (default, Jun 28 2018, 13:15:42) 
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import banking.overview

just fine. I get errors about a server connection being rejected but that's probably cause I didn't install everything I needed. I'm not sure what that first error in your post is about. (You'll probably have to add the periods in front of your banks imports as well. I didn't get that far down the import chain because of the server error. You'll also probably need to change the other files with local imports too.)

quote:

If I change __init__.py to

Python code:

from banking_scrape import *

__all__ = ['core', 'db_connect', 'mail', 'overview', 'security', 'todo_log']

This does absolutely nothing by the way. I'm actually surprised this doesn't cause a circular dependency issue. Because it's saying "from the banking package, execute __init__.py, which then says "from the banking package, execute __init__.py, which then says, "from the ...""". You need to actually tell __init__.py what to import. Also your __all__ variable doesn't do what you think it does. __all__ is use for when you want to limit what gets imported when you do import *.

For example if I had a file

code:

# example.py

UserFacingClass:
    ...

DummyClassThatShouldNotBeImportedWithAWildCard:
    ...

If you do from .example import * then you will import both classes shown. If you add __all__ = 'UserFacingClass' to that file, then you will only import UserFacingClass when you do from .example import *.

# ? Aug 5, 2018 13:25

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Thermopyle posted:

Fine, I'll do it!

Though, I might find a precalc course to do first...

Whenever I tutor calculus (or any other higher level math) the thing that 99% of my students have trouble with isn't precalc or calculus or the harder stuff in differential equations. They all have issues that stem from ... algebra. People have no idea how to simplify/reduce equations or how fractions work. I would suggest brushing up with an algebra course :v:

# ? Aug 5, 2018 13:28

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

It's funny you said that because as I was looking over prerequisites for various courses I decided an algebra refresher was exactly what I should start out with!

Thanks for confirming I wasn't a big dummy for thinking that.

# ? Aug 5, 2018 15:34

TerraGoetia: Feb 21, 2011; A cup of spiders.

Hi thread.

I am new to Python and trying to construct a tree for homework. I did a short test and have a problem:

code:

class Node:
    children = []


n = Node()
m = Node()
x = Node()
y = Node()


n.children.append("N")
n.children.append("N's second child")
x.children.append("X's child")


for c in y.children:
    print(c)

The output is:

N
N's second child
X's child

I would expect that node y would have no children, so I'm confused. What am I doing wrong?

# ? Aug 6, 2018 01:53

QuarkJets: Sep 8, 2008

TerraGoetia posted:

Hi thread.

I am new to Python and trying to construct a tree for homework. I did a short test and have a problem:
code:
class Node:
    children = []


n = Node()
m = Node()
x = Node()
y = Node()


n.children.append("N")
n.children.append("N's second child")
x.children.append("X's child")


for c in y.children:
    print(c)
The output is:

N
N's second child
X's child

I would expect that node y would have no children, so I'm confused. What am I doing wrong?

Try defining your class like this:

Python code:

class Node:
    def __init__(self):
        self.children = []

The way that you defined Node sets up "children" as a class attribute, so all instances of the class are accessing the same list

Using the constructor like in the code snippet here instantiates Node.children as an instance attribute, so each instance of Node gets its own list independent from other instances

QuarkJets fucked around with this message at 02:26 on Aug 6, 2018

# ? Aug 6, 2018 02:23

TerraGoetia: Feb 21, 2011; A cup of spiders.

That makes sense! Thank you!

# ? Aug 6, 2018 02:27

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Thermopyle posted:

Thanks for confirming I wasn't a big dummy for thinking that.

Don't feel silly because if you think about it when was the last time you or most people (in the US) studied/learned algebra? I would wager it was in high school and from whatever teacher they could find that was vaguely "good at math." And after that one algebra class people move on and never revisit it again, and by the time you're taking calculus every professor just assumes that you know algebra because it's "basic stuff" despite the fact that you only learned what you needed to learn to pass whatever state mandated fill in the bubble with a #2 pencil exam there was at the time.

Boris Galerkin fucked around with this message at 07:01 on Aug 6, 2018

# ? Aug 6, 2018 06:59

Mark Larson: Dec 27, 2003; Interesting...

If I'm writing some scripts to do ETL, should I be concerned with building classes and stuff? I'm not really a programmer so some of the OO concepts are foreign to me.

# ? Aug 6, 2018 14:20

cinci zoo sniper: Mar 15, 2013

Mark Larson posted:

If I'm writing some scripts to do ETL, should I be concerned with building classes and stuff? I'm not really a programmer so some of the OO concepts are foreign to me.

If you just do transport, probably not. Otherwise - probably yes.

# ? Aug 6, 2018 14:31

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

Boris Galerkin posted:

lots of help

Thank you for that. I ended up getting it fixed (https://bitbucket.org/dluther23/bankscrape/src/master/). I tried forever with the .core thing, but couldn't get it working properly.

Python code:

from banking_scrape.core import CoreTools
from banking_scrape import security
from banking_scrape.db_connect import DatabaseInterface
from banking_scrape.todo_log import ToDoLog
import logging

That and an empty __init__.py seem to work just fine and import without error into my Django project.

# ? Aug 6, 2018 22:51

Mark Larson: Dec 27, 2003; Interesting...

What's the best way to take some JSON from an API response, parse it and load it to a Postgres/Redshift database? I'm able to get the API response, and use psycopg2 to load a table with JSON in it, so the only thing now is to unpack it and load it.

The load is actually a dictionary of dictionaries, so I would need to do some additional unpacking to load three tables from one API call/response.

Is pandas.io.json_normalize the best way to handle this? I thought I could get some answers from Stack overflow but it's kind of all over the place.

# ? Aug 8, 2018 10:57

cinci zoo sniper: Mar 15, 2013

Mark Larson posted:

What's the best way to take some JSON from an API response, parse it and load it to a Postgres/Redshift database? I'm able to get the API response, and use psycopg2 to load a table with JSON in it, so the only thing now is to unpack it and load it.

The load is actually a dictionary of dictionaries, so I would need to do some additional unpacking to load three tables from one API call/response.

Is pandas.io.json_normalize the best way to handle this? I thought I could get some answers from Stack overflow but it's kind of all over the place.

If you have very well formed, consistently serves, JSON, then yes, you could maybe try pandas normalisation. Otherwise I�d just write a custom thing for that specific JSON.

# ? Aug 8, 2018 11:34

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I'd like to plot the Recam�n sequence as they do here:

https://www.youtube.com/watch?v=FGC5TdIiT9U

Not only would I like to create the plot, I'd like to animate it so you see the line growing from zero to whatever.

I've never done anything with computer graphics of any type so I have no idea where to start. My outside view is that the stuff you normally hear about when doing graphs with python like matplotlib are more for static graphics. Maybe I should use PyGame? I don't really know what I need here.

# ? Aug 8, 2018 17:35

Dominoes: Sep 20, 2007

Here's an API designed to make animating in MPL easier.

Can you accomplish what you want using snapshots of MPL, or do you need something more custom?

# ? Aug 8, 2018 17:43

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Just to give you an idea where my experience level is when it comes to putting graphics on a computer screen via a programming language:

I don't even know wtf MPL is.

# ? Aug 8, 2018 18:28

cinci zoo sniper: Mar 15, 2013

Thermopyle posted:

Just to give you an idea where my experience level is when it comes to putting graphics on a computer screen via a programming language:

I don't even know wtf MPL is.

Matplotlib, hands down the most confusingly complex API I've ever worked with. You can get anything plotted with it in Python, sometimes just at a cost.

# ? Aug 8, 2018 18:31

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Oh, well fuckin duh, Thermopyle.

# ? Aug 8, 2018 18:31

Dominoes: Sep 20, 2007

cinci zoo sniper posted:

Matplotlib, hands down the most confusingly complex API I've ever worked with. You can get anything plotted with it in Python, sometimes just at a cost.

I made an Api smoother for plotting funcs and using the (slighter better) non-class-based API in Notebooks to address a part of this. The (brand new) lib I linked above addresses the animation part.

In the major release a few months back, they changed the default display settings to be more sensible, at least.

If you want something truly custom, you could try this SDL binding. SDL is a C-based lib commonly used to draw arbitrary things on the screen. Probably overkill/too low-level for your use here. Qt has a decent drawing lib too.

Dominoes fucked around with this message at 18:58 on Aug 8, 2018

# ? Aug 8, 2018 18:39

SurgicalOntologist: Jun 17, 2004

Matplotlib would get the job done, but I would personally recommend bokeh, possibly with holoviews. With any of these you can achieve the animation natively, use a slider, or export images to be stitched into a video with another library or external program.

In any case, start by just making a static plot and go from there.

# ? Aug 8, 2018 18:55

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Alternatively it seems like a very turtle thing:

# ? Aug 9, 2018 13:50

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

I'm trying to make a project of mine a little more modular by having a dependency injector manage selecting what version of something to use based on runtime constraints. The source is constrained by subdirectories. Let's see I am trying to implement Butt:

root_name/api/butt_api.py: Contains abstract base classes
root_name/default/butt.py: Most common fallback of Butt
root_name/windows/butt.py: A Windows-specific version of Butt

The method of choosing which one to use has been solved a long time ago; I have no problems there. My issue is that I'd like to be able to install the Butt feature independent of other modules in this tree structure, while having it coexist with everything else. So I'd like to package a butt.whl that installs into root_name under my Python site-packages. If I come along and install foo.whl then I want that to also go into root_name in the right places. Ditto for uninstalling. I'd prefer if root_name could be uninstalled if the last module that uses it is removed, but it's probably more practical to have the parent project that consumes this tree and does the injection to initially set up that whole structure, if possible. Is that a thing I can do from one Python wheel?

# ? Aug 11, 2018 23:54

salisbury shake: Dec 27, 2011

I've got a producers/consumer setup that adds and removes data between a set of queues.

One step of the pipeline is network IO bound, where it loads data from one queue, fetches related data from the internet and puts the result in another queue to be consumed by a pool of CPU-bound processes. Ran some benchmarks and using asyncio is much faster than using threads in this context.

Is there a way to share queues between an async context and a synchronous one without blocking on the async side?

# ? Aug 12, 2018 20:07

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 10:41

Hollow Talk: Feb 2, 2014

A bit late but :effort:

Mark Larson posted:

If I'm writing some scripts to do ETL, should I be concerned with building classes and stuff? I'm not really a programmer so some of the OO concepts are foreign to me.

This strongly depends on what kind of ETL we are talking about, how complex your processes are, and if there are dependencies between various steps. For slightly more complex requirements with dependencies, required job control, and cascades, have a look at Spotify's Luigi. It's particularly useful for interdependent steps and I use it at work for a number of relatively light-weight ETL processes. It uses a pretty neat set of Classes to encapsulate inputs, outputs and dependencies between tasks.

Mark Larson posted:

What's the best way to take some JSON from an API response, parse it and load it to a Postgres/Redshift database? I'm able to get the API response, and use psycopg2 to load a table with JSON in it, so the only thing now is to unpack it and load it.

The load is actually a dictionary of dictionaries, so I would need to do some additional unpacking to load three tables from one API call/response.

Is pandas.io.json_normalize the best way to handle this? I thought I could get some answers from Stack overflow but it's kind of all over the place.

Have a look if pandas is good enough. Depending on how deep the nesting is, I usually tend to use nested dict comprehensions to flatten dictionaries when I want more control and/or need to ensure data quality in a more granular way. I find this particularly useful for large data, since it gives me more explicit control over chunking, and I can use iterators for lazy evaluation (pandas is not great when trying to parse a 90GB JSON file, for example).

Hollow Talk fucked around with this message at 23:07 on Aug 12, 2018

# ? Aug 12, 2018 23:00

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »