Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Dominoes
Sep 20, 2007

Thermopyle posted:

Does it handle imports done with importlib? That's pretty common, so if it doesn't you should probably make a note in the readme or something.
Nope. Need to do that.

Adbot
ADBOT LOVES YOU

Banjo Bones
Mar 28, 2003

Thank you guys for the clarification on the functions and classes.

Apologies in advance for asking all these super low-level questions. I really spend a lot of time googling these before coming here to ask.
I'm trying to work with files on a basic level, like getting python to read a .txt file in the terminal, but I'm super stuck on this:

code:
 with open(r'C:\Users\bromp\OneDrive\Desktop\Python\pi_digits.txt') as file_object:
	contents = file_object.read()
	print(contents)

FileNotFoundError: [Errno 2] No such file or directory.
I've tried renaming the file, same error. I've tried just putting the .txt file in the default directory and opening it without a file path, but it gives me the same error.
Does it have something to do with it not being able to read how Windows creates a .txt file?
I'm able to open the .txt directly through my IDE. I'm not sure what I'm missing here.

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

bromplicated posted:

Thank you guys for the clarification on the functions and classes.

Apologies in advance for asking all these super low-level questions. I really spend a lot of time googling these before coming here to ask.
I'm trying to work with files on a basic level, like getting python to read a .txt file in the terminal, but I'm super stuck on this:

code:
 with open(r'C:\Users\bromp\OneDrive\Desktop\Python\pi_digits.txt') as file_object:
	contents = file_object.read()
	print(contents)

FileNotFoundError: [Errno 2] No such file or directory.
I've tried renaming the file, same error. I've tried just putting the .txt file in the default directory and opening it without a file path, but it gives me the same error.
Does it have something to do with it not being able to read how Windows creates a .txt file?
I'm able to open the .txt directly through my IDE. I'm not sure what I'm missing here.

I'm sure you checked it already, but are you absolutely 100% sure that there are no typos or wrong capitalizations?

What happens if you put the file somewhere else, not in your OneDrive? Say in C:\Users\bromp\Desktop or something like that? It could be a permission or filesystem issue depending on what magic Microsoft does under the hood to integrate OneDrive with Windows.

Might be worth taking a look at pathlib for this type of stuff too, assuming you are on 3.4+ which you should be unless you have a very specific reason to stick with 2.7. Someone else talked about it here ages and ages ago and I thought it was a gimmick but I am 100% on board the pathlib train.

code:
file_path = pathlib.Path(r'C:\Users\bromp\OneDrive\Desktop\Python\pi_digits.txt')
contents = file_path.read_text()
print(contents)
(The library automatically opens and closes the file, so no need to do the 'with' statement here.)

You can also say "gently caress these backwards slashes on Windows" and do this:

code:
file_path = pathlib.Path.home() / 'OneDrive' / 'Desktop' / 'Python' / 'pi_digits.txt'
# or:  file_path = pathlib.Path.home() / 'OneDrive/Desktop/Python/pi_digits.txt'

Data Graham
Dec 28, 2009

📈📊🍪😋



^^^ Hahaha. I can't decide if that's the most pythonic or the least pythonic thing ever

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Data Graham posted:

^^^ Hahaha. I can't decide if that's the most pythonic or the least pythonic thing ever

The first one is a bit excessive for me personally, but it’s useful being able to do stuff like

code:
prefix = pathlib.Path(“/home/user/whatever”)

fp1 = prefix / “foo.txt”
fp2 = prefix / “bar.txt”

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Boris Galerkin posted:

The first one is a bit excessive for me personally, but it’s useful being able to do stuff like

code:
prefix = pathlib.Path(“/home/user/whatever”)

fp1 = prefix / “foo.txt”
fp2 = prefix / “bar.txt”

Yeah, it's most useful for adding stuff onto an existing path instead of using path.join.

pathlib is cool and good. There's a couple of things that are kind of WTF about it, but I can't think of any of them off the top of my head.

Data Graham
Dec 28, 2009

📈📊🍪😋



But what if u wanted to divide the string by something

like u'zero'

Solumin
Jan 11, 2013
It's not a string, it's a path-like object! :eng101:

KICK BAMA KICK
Mar 2, 2009

Noticed an issue in PyCharm I wasn't having before: when I use the "amend commit" option, which I often do cause I'm a dummy who forgets poo poo I meant to do before committing, and then push to origin it's getting rejected and forcing me to go through a merge that I barely understand but seems unnecessary -- obviously, to me at least, I just want the newer code. It used to work! Couldn't tell you exactly when it started happening but I'm not sure if it was when I either upgraded to PyCharm Professional or I switched from Bitbucket to Github when they started offering private repos for free. Is there a setting somewhere -- PyCharm, Github, wherever, I can change to make it work again?

necrotic
Aug 2, 2005
I owe my brother big time for this!
If you've already pushed you have to either force push the newly amended commit, or do what PyCharm is making you do. Maybe a setting changed and it's pushing automatically, if you didn't do that manually.

Or it just force pushed before and they realized that was dumb, so now it doesn't.

Dominoes
Sep 20, 2007

How OS-versatile are Python compiles? Ie, if I compile on Ubuntu 19.04, can I expect it to work on other Ubuntu versions? How about Debian? Other Linux Distros? For Windows, can I expect the binary to work on any modern version of Win as long as the bit-architecture matches?

I'm building a Python install/version manager into the dep manager. It appears I'll have to build and host the binaries.

Dominoes fucked around with this message at 23:22 on Sep 6, 2019

Solumin
Jan 11, 2013
They basically aren't, as far as I know. Pyenv, for example, downloads and builds the interpreters you want.

Banjo Bones
Mar 28, 2003

Boris Galerkin posted:

I'm sure you checked it already, but are you absolutely 100% sure that there are no typos or wrong capitalizations?

What happens if you put the file somewhere else, not in your OneDrive? Say in C:\Users\bromp\Desktop or something like that? It could be a permission or filesystem issue depending on what magic Microsoft does under the hood to integrate OneDrive with Windows.



So I've tried to disable OneDrive, rename the .txt file, and place the .py file and the .txt file I want to open directly in my local Python directory, but I'm still getting the same [File Not Found] [Errno2] error.
code:
with open(r'C:\Users\bromp\AppData\Local\Programs\Python\Python37-32\pywork.txt') as file_object:
	contents = file_object.read()
	print(contents)
I have also tried the pathlib method and it spits out the following but ends in the same error:

code:
  File "file_reader.py", line 3, in <module>
    contents = file_path.read_text()
  File "C:\Users\bromp\AppData\Local\Programs\Python\Python37-32\lib\pathlib.py", line 1189, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "C:\Users\bromp\AppData\Local\Programs\Python\Python37-32\lib\pathlib.py", line 1176, in open
    opener=self._opener)
  File "C:\Users\bromp\AppData\Local\Programs\Python\Python37-32\lib\pathlib.py", line 1030, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\bromp\\AppData\\Local\\Programs\\Python\\Python37-32\\pywork.txt'
I'm wondering if it is a way that Windows 10 creates a .txt file is unrecognizable ?

edit:

It must be. Because I copied someone else's pi_digits.txt, copied to my default directory, and now it works.
Also works in my original folder under the OneDrive path.

edit 2:

Holy gently caress I figured it out. My file was literally named "pi_digits.txt.txt" because windows 10 automatically creates the extension and then HIDES IT. WHAT THE gently caress.

Banjo Bones fucked around with this message at 06:03 on Sep 7, 2019

Dominoes
Sep 20, 2007

Solumin posted:

They basically aren't, as far as I know. Pyenv, for example, downloads and builds the interpreters you want.
Thank you. Hopefully I'll be able to enlist some help to build versions for various OSes as required. Having the user's computer compile is a non-starter, given how finicky that can be.

Solumin
Jan 11, 2013
I think you may have misunderstood me -- pyenv uses a utility they wrote called python-build that compiles Python interpreters using the user's computer. It's not only an option, it's often your best option.

Doing this on Windows..... Yeah. I don't know.

Dominoes
Sep 20, 2007

Solumin posted:

I think you may have misunderstood me -- pyenv uses a utility they wrote called python-build that compiles Python interpreters using the user's computer. It's not only an option, it's often your best option.

Doing this on Windows..... Yeah. I don't know.
That was my interpretation... I don't think that can be relied on. Even on Linux, compiling C's a pain. So far, this has been pretty smooth, but I still had to google errors / install system deps twice. Making and distributing binaries appears to be the only way forward.

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!
You can use docker to compile all your stuff for different things without needing a full fledged ____ installation. It’s pretty neat.

Pyenv will do the compiling for you automatically. The user doesn’t need to know anything about how that works. That’s also what things like homebrew and macports (both for macOS) work for some packages. All the user might experience is that it takes a bit longer than normal and maybe their laptop fans spin up to sound like a harrier jet taking off.

Just out of curiosity, which packages are you looking at that need to be compiled? If it’s any of the numpy/scipy related stuff then honestly,
I would say don’t bother and just point potential users over to conda or something. There are so many bits and pieces regarding compiling and linking those libraries that it just can’t possibly be worth it in any sense.

Dominoes
Sep 20, 2007

Boris Galerkin posted:

You can use docker to compile all your stuff for different things without needing a full fledged ____ installation. It’s pretty neat.
Awesome! Sounds like the way fwd on that.

quote:

Pyenv will do the compiling for you automatically. The user doesn’t need to know anything about how that works. That’s also what things like homebrew and macports (both for macOS) work for some packages. All the user might experience is that it takes a bit longer than normal and maybe their laptop fans spin up to sound like a harrier jet taking off.
Nice! May see if can use that as a dep, or follow whatever logic they used to remove the fiddling and errors.

quote:

Just out of curiosity, which packages are you looking at that need to be compiled? If it’s any of the numpy/scipy related stuff then honestly,
I would say don’t bother and just point potential users over to conda or something. There are so many bits and pieces regarding compiling and linking those libraries that it just can’t possibly be worth it in any sense.
CPython, versions >= 3.4

QuarkJets
Sep 8, 2008

Dominoes posted:

CPython, versions >= 3.4

Why?

Dominoes
Sep 20, 2007

Adding a Python install/version manager to the dep manager.

wolrah
May 8, 2006
what?

bromplicated posted:

Holy gently caress I figured it out. My file was literally named "pi_digits.txt.txt" because windows 10 automatically creates the extension and then HIDES IT. WHAT THE gently caress.

This isn't a Windows 10 thing, this is a Windows Explorer thing that goes all the way back to Windows 95.

https://www.granneman.com/tech/windows/showextensions

Bruegels Fuckbooks
Sep 14, 2004

Now, listen - I know the two of you are very different from each other in a lot of ways, but you have to understand that as far as Grandpa's concerned, you're both pieces of shit! Yeah. I can prove it mathematically.

wolrah posted:

This isn't a Windows 10 thing, this is a Windows Explorer thing that goes all the way back to Windows 95.

https://www.granneman.com/tech/windows/showextensions

The first thing I do with every new windows machine I encounter is turn off that stupid loving "hide extensions for known file types" setting.

CarForumPoster
Jun 26, 2013

⚡POWER⚡
I'd like your experiences using jupyter notebooks for daily or every 30 min cron jobs.

We have some web scraper and API calls that happen at a certain time every day run on an Amazon Workspace.

We're about to refactor some important code related to this and are kinda half .py files and half .ipynb at this point. Generally the .ipynb files have the actual code a human would review and the .py files have scraper templates and 50+ helper functions in them. I really like the idea of using a ipynb for documentation.

What do you guys do? Anyone ditched .py files for ipynbs? We just started down this road, any thoughts on automatically scraping 10+ websites per day using jupyter notebooks?

QuarkJets
Sep 8, 2008

I think I read somewhere that that's how Netflix operates

I think it sounds horrible and some person in 10 years will be cursing your name, but you do you

Sad Panda
Sep 22, 2004

I'm a Sad Panda.
I'm writing a piece of software which will check the contents of a folder for files and display whether they are found or not. It will be used to check if students have submitted their work.

I'm thinking the program should...
1. Read the class info in from the text file (done)
2. User choses which class to search for
3. When you select the class name, it shows the names of all the students in that class
4. Every 5 seconds it checks the specific path to see if the work is there (is their name in the file found)
5. Repeat until all work found / user says stop.
6. Extension - Might show a list of files that don't match any names. This will be useful as there'll be some students who mistyped their name and so know that they need to re-submit with the properly named files (students don't have privy to rename files)


I'm trying to decide the best way to implement the interface. I'll have the normal version of Python, but be unable to pip install any extra packages. I'm thinking that I either...
1) Use Tkinter (no experience)
2) Re-generate the HTML each time and make it automatically reload that page. Would something like <meta http-equiv="refresh" content="10"> refresh to the regenerated HTML?

QuarkJets
Sep 8, 2008

Can you create a virtual environment? Or a miniconda environment?

Dominoes
Sep 20, 2007

CarForumPoster posted:

I'd like your experiences using jupyter notebooks for daily or every 30 min cron jobs.

We have some web scraper and API calls that happen at a certain time every day run on an Amazon Workspace.

We're about to refactor some important code related to this and are kinda half .py files and half .ipynb at this point. Generally the .ipynb files have the actual code a human would review and the .py files have scraper templates and 50+ helper functions in them. I really like the idea of using a ipynb for documentation.

What do you guys do? Anyone ditched .py files for ipynbs? We just started down this road, any thoughts on automatically scraping 10+ websites per day using jupyter notebooks?
I avoid it unless I'm using it to share code that uses plots and/or interactivity. The lack of determinism (eg Cell B relies on cell A, cell A changes, Cell B doesn't necessarily update)'s a trap, and I don't find them convenient to work with in terms of setup/teardown.

Sad Panda
Sep 22, 2004

I'm a Sad Panda.

QuarkJets posted:

Can you create a virtual environment? Or a miniconda environment?

I don't think so. I've made something that generates some ugly html. Feels a hacky way of doing it but seems to work.

QuarkJets
Sep 8, 2008

Sad Panda posted:

I don't think so. I've made something that generates some ugly html. Feels a hacky way of doing it but seems to work.

Is there someone who you can ask?

I wouldn't suggest trying to do this with just core Python

Dominoes
Sep 20, 2007

Dominoes posted:

Adding a Python install/version manager to the dep manager.
To clarify, the way fwd is to get PyPi to officially host binaries, but it's an easier sell with a proof-of-concept and use-case. There are no technical limitations, and some of the internal infrastructure's set up for this. Asking a user's machine to compile code is not a user-friendly option.

Sad Panda posted:

I'm trying to decide the best way to implement the interface. I'll have the normal version of Python, but be unable to pip install any extra packages. I'm thinking that I either...
1) Use Tkinter (no experience)
2) Re-generate the HTML each time and make it automatically reload that page. Would something like <meta http-equiv="refresh" content="10"> refresh to the regenerated HTML?
Is CLI an option? Your HTML hack sounds awful, but may be the best approach!

Dominoes fucked around with this message at 03:52 on Sep 8, 2019

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Sad Panda posted:

I'm writing a piece of software which will check the contents of a folder for files and display whether they are found or not. It will be used to check if students have submitted their work.

I'm thinking the program should...
1. Read the class info in from the text file (done)
2. User choses which class to search for
3. When you select the class name, it shows the names of all the students in that class
4. Every 5 seconds it checks the specific path to see if the work is there (is their name in the file found)
5. Repeat until all work found / user says stop.
6. Extension - Might show a list of files that don't match any names. This will be useful as there'll be some students who mistyped their name and so know that they need to re-submit with the properly named files (students don't have privy to rename files)


I'm trying to decide the best way to implement the interface. I'll have the normal version of Python, but be unable to pip install any extra packages. I'm thinking that I either...
1) Use Tkinter (no experience)
2) Re-generate the HTML each time and make it automatically reload that page. Would something like <meta http-equiv="refresh" content="10"> refresh to the regenerated HTML?

There's several ways to make a HTML file refresh itself. How will people be accessing this file? Via a server or via the file:// protocol?

mr_package
Jun 13, 2000

Thermopyle posted:

pathlib is cool and good. There's a couple of things that are kind of WTF about it, but I can't think of any of them off the top of my head.
For me the main thing is no equivalent to shutil.rmtree(). I find it annoying because cleaning up a temp folder is a really common thing and pathlib has so much functionality but omits the ability to delete a non-empty dir.

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

mr_package posted:

For me the main thing is no equivalent to shutil.rmtree(). I find it annoying because cleaning up a temp folder is a really common thing and pathlib has so much functionality but omits the ability to delete a non-empty dir.

Why not use tempfile for temp dirs? Iirc a tempfile.TemporaryDirectory should automatically clean itself up if you use it in a with statement, or you can call its cleanup() method explicitly.

And if it fails to do so at least it’s in the proper place in your OS for temp stuff and I’m assuming the OS will eventually remove those files/dirs when it does its housekeeping stuff.

Sad Panda
Sep 22, 2004

I'm a Sad Panda.
I can check on Monday, but I think I only have core Python functionality.

What I've put together at the moment...
1. Reads class lists from a text file,
2. Displays a menu in the CLI where you choose a class
3. Loops every 5 seconds to check the folder and creates some HTML. I can make that prettier probably.



The code is at https://github.com/domluther/SubmittorPy

The idea is basically to have it on screen when I ask students to submit work. It will sit there auto-refreshing until I stop it. It will let students and myself see who has/hasn't submitted, and who might need to rename their work because it hasn't found it.

Thermopyle posted:

There's several ways to make a HTML file refresh itself. How will people be accessing this file? Via a server or via the file:// protocol?

It will just be me accessing it and displaying it. The <meta http-equiv="refresh" content="10"> at the top of the HTML file seems to do what I need.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Sad Panda posted:

I'm writing a piece of software which will check the contents of a folder for files and display whether they are found or not. It will be used to check if students have submitted their work.

[...]

I'm trying to decide the best way to implement the interface. I'll have the normal version of Python, but be unable to pip install any extra packages. I'm thinking that I either...
1) Use Tkinter (no experience)
2) Re-generate the HTML each time and make it automatically reload that page. Would something like <meta http-equiv="refresh" content="10"> refresh to the regenerated HTML?

IMO if you're doing this for students and make it good, and you're just their dumb ol teacher that built it, it'll probably get a few of them them interested in coding.

I would accomplish this in using Dash. Dash uses python (Flask) to make pretty Dashboards relatively easily. I'd have a function that does as you say continuously and plots the output.

There's lots of open source examples: https://dash-gallery.plotly.host/Portal/

And components, such as an upload button: https://dash.plot.ly/dash-core-components

...and even makes the HTML layout easy: https://dash.plot.ly/dash-html-components

If you like bootstrap, there's also this: https://dash-bootstrap-components.opensource.faculty.ai/l/components

CarForumPoster fucked around with this message at 13:18 on Sep 8, 2019

Sad Panda
Sep 22, 2004

I'm a Sad Panda.

CarForumPoster posted:

IMO if you're doing this for students and make it good, and you're just their dumb ol teacher that built it, it'll probably get a few of them them interested in coding.

I would accomplish this in using Dash. Dash uses python (Flask) to make pretty Dashboards relatively easily. I'd have a function that does as you say continuously and plots the output.

There's lots of open source examples: https://dash-gallery.plotly.host/Portal/

And components, such as an upload button: https://dash.plot.ly/dash-core-components

...and even makes the HTML layout easy: https://dash.plot.ly/dash-html-components

If you like bootstrap, there's also this: https://dash-bootstrap-components.opensource.faculty.ai/l/components

That looks super pretty, but seems to involve pip install and I still don't think I've got pip install privs.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Sad Panda posted:

That looks super pretty, but seems to involve pip install and I still don't think I've got pip install privs.

I was on a super locked down comp at a large defense contractor and could still pip install things. I had to use the proxy param. Can you browse to pypi.python.org in your browser?

Master_Odin
Apr 15, 2010

My spear never misses its mark...

ladies
Can you not use the --user parameter to also specify only install to your user account? Or just a virtualenv?

You could also go with https://chrome.google.com/webstore/detail/web-server-for-chrome/ofhbbkphhbklhfoeikjpcbhemlocgigb?hl=en and be able to do a bit more fine-grained HTML updating through JS. What I mean here would be that your python code writes the results to a JSON file, an then a basic HTML file that just shows the body and has a JS function that pings the JSON file every x seconds and updates the HTML structure accordingly.

QuarkJets
Sep 8, 2008

Yeah since this is an application that only you are running just set up a miniconda distribution in your home area and then install whatever you need

Adbot
ADBOT LOVES YOU

9-Volt Assault
Jan 27, 2007

Beter twee tetten in de hand dan tien op de vlucht.

CarForumPoster posted:

I'd like your experiences using jupyter notebooks for daily or every 30 min cron jobs.

We have some web scraper and API calls that happen at a certain time every day run on an Amazon Workspace.

We're about to refactor some important code related to this and are kinda half .py files and half .ipynb at this point. Generally the .ipynb files have the actual code a human would review and the .py files have scraper templates and 50+ helper functions in them. I really like the idea of using a ipynb for documentation.

What do you guys do? Anyone ditched .py files for ipynbs? We just started down this road, any thoughts on automatically scraping 10+ websites per day using jupyter notebooks?

Ive done this to create some etl-scripts using Netflix’ papermill library. It works surprisingly well once you’ve got it all setup. The parameters I pass in are used in sql scripts to get some data out of a Postgres database before moving it to a data warehouse. It can of course all be done in a regular python script but I like having errors show up in the notebook, and I also do a bunch of checks on the data for which I create and display some graphs. The processes run daily and I move a copy of the notebooks to a separate folder so we can easily go back and check what happened some days back.

I know it runs counter to what most people consider normal but I’m actually quite happy with it, especially compared to using a tool like SSIS like we used before for ETL jobs. I do share the complaints about notebooks being dangerous if you use it interactively due to our of order execution, but in this scenario the notebooks just runs top to bottom, so that isn’t an issue.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply