Boris Galerkin posted:

I want to play around with machine learning for fun. To start with say I want to build and train a model to tell me if a picture is a (hot)dog or not a (hot)dog. How do I get started?

Hollow Talk posted:

Have a look if pandas is good enough. Depending on how deep the nesting is, I usually tend to use nested dict comprehensions to flatten dictionaries when I want more control and/or need to ensure data quality in a more granular way. I find this particularly useful for large data, since it gives me more explicit control over chunking, and I can use iterators for lazy evaluation (pandas is not great when trying to parse a 90GB JSON file, for example).

Mark Larson posted:

Thanks for your responses. I will take a look at Airflow and Luigi once it starts getting complex enough. I'm also using Stitch so I could also change over to the Singer Tap methodology once it gets complicated enough to benefit from the Stitch backend, and integrate with their other integrations.

For the second point, could you tell me what you mean by using nested dict comprehensions to flatten dictionaries? Like a code example from Stack Overflow would be good. My searches haven't led to much. There are 3 levels of nesting I'm potentially looking at, but they are optional since I can get those attributes by calling another API.

Mark Larson posted:

Thanks for your responses. I will take a look at Airflow and Luigi once it starts getting complex enough. I'm also using Stitch so I could also change over to the Singer Tap methodology once it gets complicated enough to benefit from the Stitch backend, and integrate with their other integrations.

For the second point, could you tell me what you mean by using nested dict comprehensions to flatten dictionaries? Like a code example from Stack Overflow would be good. My searches haven't led to much. There are 3 levels of nesting I'm potentially looking at, but they are optional since I can get those attributes by calling another API.

cinci zoo sniper posted:

https://www.datacamp.com/community/tutorials/python-dictionary-comprehension should be an OK start.

BannedNewbie posted:

If you want to have a clue what you're doing take Andrew Ng's machine learning class on coursera, if you don't install scikit learn, read some of the docs and go nuts.

Boris Galerkin posted:

This one? Just signed up to Coursera and this course says I can either pay or do it for free. Do I lose out on anything if I do it for free other than the certificate? (Never done one of this online courses before. Are the certificates even worth anything?)

Dominoes posted:

The import's lower-case for that module, despite the uppercase module name. You said you tried it, but your examples all use upper-case. I can't see what's wrong; try import quandl again.

a forbidden love posted:

Anyway, any help would be appreciated. Thanks!

Peeny Cheez posted:

Are you running python 2 and 3 side by side? You may have installed it in 2 and not 3.

a forbidden love posted:

I've searched Google and it's the same answer over and over, pretty much just making sure the programmer has pip-installed the package correctly or showing how to pip-install using command lines that don't work on my system. (which I would also love to get some info or reading material as to why, btw).

Symbolic Butt posted:

Alternatively it seems like a very turtle thing:

Boris Galerkin posted:

Can you do a help('modules') in whichever interpreter you want to use and see if Quandl is listed?

Boris Galerkin posted:

Which command line commands don't work for you?

Boris Galerkin posted:

I really want to like pipenv but I just can't because it feels like it's a buggy cryptic mess and nothing it tells me/outputs makes any sense at all.

For example I just created a new pipenv with the following Pipfile:

code:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
scipy = "*"
pandas = "*"

[dev-packages]

[requires]
python_version = "3.6"

Everything works great. Now I add jupyter and it gives me this mess of an output:

code:

Successfully installed [...] widgetsnbextension-3.4.0

Adding jupyter to Pipfile's [packages]...

Pipfile.lock (94e126) out of date, updating to (52d57c)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches widgetsnbextension==3.3.1,~=3.3.0,~=3.4.0
Tried: [...], 3.4.0, 3.4.0
Skipped pre-versions: [...]
There are incompatible versions in the resolved dependencies.

So pipenv tells me it installed widgetsnbextension-3.4.0, and then it tells me that it can't find a compatible version of widgetsnbextension that matches either 3.3.1, 3.3.0+, or 3.4.0+. Then it also tells me that it tried to install version 3.4.0 (twice), but for some reason it doesn't think 3.4.0 matches version number 3.4.0.

It was a fun experience but back to conda environments I go.

e: Meanwhile, jupyter was successfully installed in this pipenv and it works fine. What the hell does any of this even mean.

Thermopyle posted:

pipenv is kind of weird still, but note that it just gave you a warning, not an error.

pipenv could be more helpful in explaining what happened here, but I think what its telling you is that the sub-dependencies of your dependencies declare require versions of widgetsnbextension that are not resolvable. In other words:

1. one sub-dependency requires exactly version 3.3.1.
2. one sub-dependency requires 3.3.*
3. one sub-dependency requires 3.4.*

There is not a version you can install that meets all of the requirements at the same time. Pipenv defaults to installing the latest version and does so, and thats why things actually work. Libraries generally don't check the versions of their sub-dependencies at runtime, so the package that required 3.3.1 and the package and 3.3.* go ahead and just do their imports and use the code and you just happen to luck out that those packages don't use any APIs whose behavior changed between 3.3 and 3.4.

Or maybe the behavior has changed, but it's in a subtle way that will not cause a crash but will cause errors is your data!

Anyway, pipenv can't fix this poo poo, but it could give better warning messages.

Of note, your conda environments have the exact same problem, its just that you don't even get a warning.

dependencies and dependency resolution is poo poo and we develop on a house of cards

a forbidden love posted:

a forbidden love posted:

According to the python documentation website (https://docs.python.org/3/installing/index.html#installing-index) I should be able to install using

python -m pip install SomePackage

But to get it to work on my shells I use:

py -m pip install somepackage

Boris Galerkin posted:

I found on one of their closed GitHub issues that apparently the "*" version signifier doesn't mean wildcard as it does in every other usage in the world, but instead it means "latest version." I don't really have anything constructive to say about that so it's whatever.

Maybe I assumed too much, but I thought that specifying wildcards for every package would be translated to "I don't care which versions you use, just get me the latest compatible versions of every package I listed." Isn't that what requirements.txt does if you just list packages and not versions?

pmchem posted:

For those of you who use Python for scientific computing, do you use Spyder as your editor / development environment?

If "yes", what changes/features would you like to see in it?
If "no", what changes/features would convince you to use it?

If you don't know what I'm talking about, check out Spyder here: https://github.com/spyder-ide/spyder . It's the default IDE for the Anaconda Python distribution.

I may have the chance to work on enhancing Spyder with a hardcore python dev. I'm genuinely interested in serious replies, thanks!

BaronVonVaderham posted:

Coming from an astrophysics perspective, I prefer to use Jupyter notebooks for a lot of my work (which runs as a local server you load in a browser tab). I use PyCharm generally for my standalone scripts and such, but I like using Jupyter for compiling and presenting my work since I can inline LaTeX blocks, plots, etc.

I've never heard of Spyder and I don't know enough to have an opinion on it yet, but I'll add it to the long list of new things to look into.

unpacked robinhood posted:

I'm trying to apply a function f(x,y) to each possible combination of two strings in a list.

Python code:

    names = ['joel','barney','georges','amsterdam']
    # 4 is a random placeholder value
    d = pd.DataFrame(4,names,names)

code:

           joel  barney  georges  amsterdam
joel          4       4        4          4
barney        4       4        4          4
georges       4       4        4          4
amsterdam     4       4        4          4

I'd like to compute a value for each cell (in my case the Levenshtein distance) using the row index and column index as value.
Googling gets me vague and complicated answers so I'm probably not going in the right direction ?

unpacked robinhood posted:

I'm trying to apply a function f(x,y) to each possible combination of two strings in a list.

Python code:

    names = ['joel','barney','georges','amsterdam']
    # 4 is a random placeholder value
    d = pd.DataFrame(4,names,names)

code:

           joel  barney  georges  amsterdam
joel          4       4        4          4
barney        4       4        4          4
georges       4       4        4          4
amsterdam     4       4        4          4

I'd like to compute a value for each cell (in my case the Levenshtein distance) using the row index and column index as value.
Googling gets me vague and complicated answers so I'm probably not going in the right direction ?

Boris Galerkin posted:

I found on one of their closed GitHub issues that apparently the "*" version signifier doesn't mean wildcard as it does in every other usage in the world, but instead it means "latest version." I don't really have anything constructive to say about that so it's whatever.

Maybe I assumed too much, but I thought that specifying wildcards for every package would be translated to "I don't care which versions you use, just get me the latest compatible versions of every package I listed." Isn't that what requirements.txt does if you just list packages and not versions?

cinci zoo sniper posted:

If you want it to just work, and not optimally fast, then create a column base dict (e.g. {'A': None, 'B': None}), and the do loop over keys and values, updating copy of that dict that you then throw into your insert.

Mark Larson posted:

What would be the fast way then?