|
unpacked robinhood posted:Thanks. Do you simply call uuidN() to generate a value and use it as filename ? Yes, but make sure to preserve the file extension because sometimes it's used to determine the content type. If you want to get fancy you can hash the file using hashlib.sha256 and use that as the identifier, which will also let you de-duplicate the files (if you notice a filename collision, then you can just use the existing file, since it would be the same if it had the same sha256 hash.) Here's an answer on SO that would help you hash files efficiently, if you decide to go that route: https://stackoverflow.com/a/55542529 But if you don't think there will be duplicates, then using a UUID is a more simple way to avoid the problem of filename collisions. xtal fucked around with this message at 15:27 on Aug 6, 2019 |
# ? Aug 6, 2019 15:23 |
|
|
# ? May 15, 2024 06:05 |
|
unpacked robinhood posted:Thanks. Do you simply call uuidN() to generate a value and use it as filename ? import uuid job_id = str(uuid.uuid4()) Then we append the extension to the job_id.
|
# ? Aug 6, 2019 20:26 |
|
From a while back, about task queues for a web appThermopyle posted:95% of the time I prefer python-rq. It's simple with fine docs. Just making a mental note to take your word for it next time!
|
# ? Aug 9, 2019 03:01 |
|
KICK BAMA KICK posted:From a while back, about task queues for a web app This is helpful, I'm starting down the path of a Django project now that will send up with workers calling APIs and I'm pretty new to web app development.
|
# ? Aug 9, 2019 13:49 |
|
Would appreciate anyone willing to ops-test the package manager I posted about earlier. Intended to improve upon the work done by Pipenv and Poetry. Probably buggy. Definitely missing features (Most-notably, being able to run scripts like Ipython, Black etc, and handling `extra` dependencies). Appreciate any and all feedback, esp cases where it won't build or resolve deps. Repo Download Can install by placing either the linux or windows binaries somewhere on the `path`, running the Deb if applicable, or installing via Cargo. Planning on a Windows installer later. Currently on windows, after setting up an environment, you need to install `wheel` in the env manually. eg: `cd __pypackages__/3.7/.venv/bin`, `./python -m pip install wheel`. Highlights: Doesn't use pip with a few exceptions like installing `wheel` and `twine`. Should be (later on...) much faster about resolving dependencies, due to making up for the pypi warehouse's inadequacies via caching on a custom database. Eschews a number of the decisions made by Pipenv, although similar in many ways to Poetry. If you receive connection error panics, try again a few times; likely due to the WIP online cache. Dominoes fucked around with this message at 17:19 on Aug 11, 2019 |
# ? Aug 11, 2019 17:12 |
|
I haven't looked at the package/tool yet, but I would suggest not putting the "Why add another Python dependency manager?" part of the README so far down because that's exactly what I was thinking the entire time I was reading about it. Also, you wrote this in that sectionquote:By not requiring Python to install or run, it remains intallation-agnostic. But you also wrote this at the top quote:Python ≥ 3.4 is required. e: I'd also be interested in a more thorough comparison to Poetry because I gotta say I feel like it does everything that the readme says your package can do. Boris Galerkin fucked around with this message at 17:26 on Aug 11, 2019 |
# ? Aug 11, 2019 17:24 |
|
Boris Galerkin posted:I haven't looked at the package/tool yet, but I would suggest not putting the "Why add another Python dependency manager?" part of the README so far down because that's exactly what I was thinking the entire time I was reading about it. quote:Also, you wrote this in that section Anecdotally, in Windows this isn't a big deal, but in Linux, it's a pain. edit: Case-in-point: I'm unable to get Poetry to work with Python3; it's latched onto my sys Python2. quote:e: I'd also be interested in a more thorough comparison to Poetry because I gotta say I feel like it does everything that the readme says your package can do. Dominoes fucked around with this message at 19:02 on Aug 11, 2019 |
# ? Aug 11, 2019 17:52 |
|
I've never understood that xkcd cartoon. To me, managing Python environments on Linux is easy. - You have the system Python, which you simply shouldn't touch or use unless you're specifically working with operating system files or just need to do something basic and easy (this Python exists to for OS libraries that need a specific version of Python - the needs of the OS do not have to align with your own needs) - You never install a Python.org binary. It's not necessary, just use anaconda. - You have any number of virtual environments that live in their own directories. There is 1 pip and 1 conda per environment. To install new things in the environment, prefer to use conda if possible (search for the package on anaconda cloud), otherwise use pip - You activate whichever environment you want to use at the time, or you don't activate anything if the system python is acceptable (e.g. "I just need to calculate the log10 of something real quick")
|
# ? Aug 11, 2019 20:07 |
|
QuarkJets posted:I've never understood that xkcd cartoon. To me, managing Python environments on Linux is easy. A few cases that may demonstrate the trouble: -Managing deps through two independent repos can be confusing. -You want to use Python3.7, but the OS v is 3.6 -You install a CLI tool that decides to use sys py2, when you want sys py3. -You install a CLI tool. Do you install it for every venv? Sys py? Sys py with user flag? What if it expects sudo access? (ran into this with pgadmin4 recently) -You need to calculate the eigenvalues of something real quick. (Do I go through the overhead of activating an environment? Do I install numpy to sys Py?)
|
# ? Aug 11, 2019 20:23 |
|
Dominoes posted:A few cases that may demonstrate the trouble: quote:-Managing deps through two independent repos can be confusing. 99.9% of the time you can just use anaconda. If you have conflicting requirements for different projects then it doesn't matter how well-designed your package manager is, because at the end of the day you'll have to create separate environments anyway, which conda can do. quote:-You want to use Python3.7, but the OS v is 3.6 Don't even mess with system Python packages. At best you get something that's not quite what you want, at worst you bork the system. It's pointless Install tools in each environment where you want that tool to be accessible, so that your dependencies are managed correctly. Or in /usr/local somewhere if you think that all of your environments should be able to use it (e.g. its dependencies are just Python>=3.X or something) quote:-You need to calculate the eigenvalues of something real quick. (Do I go through the overhead of activating an environment? Do I install numpy to sys Py?) Activating an environment is a 1-liner, the overhead is negligible. If you're never using your system python anyway then you can just modify your .bashrc so that you're always using one of your anaconda environments. If you're using a notebook or an IDE like Spyder or PyCharm (which you should) then you should already be configured to use a good base environment for typical tasks, no additional steps necessary My base anaconda has everything that I need for playing with pandas dataframes, numpy, scipy, astropy, plotting, etc. It has Spyder installed for when I just want to hammer something out real quick and my PyCharm points to it by default when I'm doing actual development QuarkJets fucked around with this message at 00:15 on Aug 12, 2019 |
# ? Aug 11, 2019 21:26 |
|
It sounds like we're comparing different things; ie standard Python/PyPi vs Anaconda ecosystem.
|
# ? Aug 12, 2019 05:08 |
|
I don’t see what the difference is. Install miniconda, set up a default python3.7 environment and keep it activated 100% of the time and install misc stuff to it. Delete it and recreate if you ever feel it gets too bloated. Specific projects I’m working on get their own dedicated environment. Other than typing “conda create …” you wont even know you’re using anaconda. You can still pip install everything and ignore the conda package manager. Edit: I do pretty much the same thing QuarkJets does btw. If I want a permanent CLI tool accessible from anywhere then it gets an environment for it and then it’s just adding one alias to my bashrc (or you can use something like pipx to automate it). If I decide I wanted to build a flask website or whatever, then conda create -n myflaskwebsite python=3.7 and viola. For nonscientific stuff I tend to just use pip install instead of conda install just to help keep things separated in my head. Boris Galerkin fucked around with this message at 06:02 on Aug 12, 2019 |
# ? Aug 12, 2019 05:43 |
|
Dominoes posted:It sounds like we're comparing different things; ie standard Python/PyPi vs Anaconda ecosystem. I see those as a subset of Anaconda. Conda environments are venvs with extra features. And if you need something that conda doesn't provide, that's okay; pip and everything else still works in your environment, everything is nicely sandboxed already so just clone your environment and go nuts
|
# ? Aug 12, 2019 06:51 |
|
QuarkJets posted:I see those as a subset of Anaconda. Conda environments are venvs with extra features. And if you need something that conda doesn't provide, that's okay; pip and everything else still works in your environment, everything is nicely sandboxed already so just clone your environment and go nuts I find that Anaconda is more useful for delivering bullshit configs that can break very easily if you try to do it manually like CUDA and actually being able to talk to the GPUs. I forced myself to use an Apple laptop as my primary work machine for two years and venvs was the only way to run code in shell and not have to upgrade Apple's preferred version of pip. It super hates it when you change that hah.
|
# ? Aug 12, 2019 19:56 |
|
EVIL Gibson posted:I find that Anaconda is more useful for delivering bullshit configs that can break very easily if you try to do it manually like CUDA and actually being able to talk to the GPUs. I develop with CUDA and use GPUs in a professional context, and I don't have any issue using Anaconda. Have you had a specific problem?
|
# ? Aug 12, 2019 20:33 |
|
I've gotten away from using anaconda and just use pyenv.
|
# ? Aug 12, 2019 20:33 |
|
QuarkJets posted:I develop with CUDA and use GPUs in a professional context, and I don't have any issue using Anaconda. Have you had a specific problem? Mainly with GAN networks and running them in windows. If you install python normally, it needs to have a lot of environment settings set. Conda manages all of that and i can simply use one that works and clone it rather than hoping i changed all the settings correctly to use the python enviroment i freshly installed (problem with lots of applications both mac and pc where it leaves cruft behind ) venvs keep all that in one part and conda delivers and packages those enviroments quickly so i spend less time troubleshooting where everything goes wrong. EVIL Gibson fucked around with this message at 20:47 on Aug 12, 2019 |
# ? Aug 12, 2019 20:41 |
|
The real thing that needs solved with python is multiple dependencies that depend on different versions of their own dependencies. Unfortunately, that will likely require core language support for importing specific versions.
|
# ? Aug 12, 2019 20:41 |
|
Thermopyle posted:The real thing that needs solved with python is multiple dependencies that depend on different versions of their own dependencies.
|
# ? Aug 12, 2019 20:55 |
|
EVIL Gibson posted:Mainly with GAN networks and running them in windows. If you install python normally, it needs to have a lot of environment settings set. Conda manages all of that and i can simply use one that works and clone it rather than hoping i changed all the settings correctly to use the python enviroment i freshly installed (problem with lots of applications both mac and pc where it leaves cruft behind ) For sure, it's a godsend. Machine learning software is full of landmines, so much so that I've seen people flat recommended to just use some docker container because they couldn't remember how they set up their configuration
|
# ? Aug 12, 2019 21:20 |
|
EVIL Gibson posted:Mainly with GAN networks and running them in windows. If you install python normally, it needs to have a lot of environment settings set. Conda manages all of that and i can simply use one that works and clone it rather than hoping i changed all the settings correctly to use the python enviroment i freshly installed (problem with lots of applications both mac and pc where it leaves cruft behind ) I dont do it professionally but I recall setting up Tensorflow (GPU version) and getting Cuda to work right on Win 10 at that time (~1 year ago) required a shocking amount of work compared to what I was expecting. I generally prefer conda installing for exactly the reasons above and pip install only when a conda version isnt available.
|
# ? Aug 12, 2019 21:30 |
|
Docker is good and people should use it for things that are hard to configure. Or lots of things that are not that hard to configure. Yesterday I needed to ping a redis server somewhere from a Windows machine that didn't have redis installed and doesn't need redis installed. I just typed: docker run --rm -it redis redis-cli -h thehost ping Downloaded and installed redis in a container, ran the ping command, and then removed the container. tensorflow runs fine in docker and if I was actively doing any tensorflow stuff right now it's what I'd use.
|
# ? Aug 12, 2019 21:41 |
|
Dumb question I can’t figure out: how do you get PyCharm on windows to see packages installed to python on ubuntu/WSL? I found jetbrains’ instructions to get the interpreter setup, and that’s working fine, but PyCharm still can’t resolve a bunch of references because it doesn’t see, for example, that Django is installed. I miss my MacBook.
|
# ? Aug 13, 2019 21:29 |
|
Krakkles posted:Dumb question I can’t figure out: how do you get PyCharm on windows to see packages installed to python on ubuntu/WSL? You don't, you'll have to install those packages in windows as well. There is a way to symlink stuff between WSL and windows, but it'll be a huge pain in the rear end and probably not work right.
|
# ? Aug 13, 2019 22:42 |
|
Hmmm. So even though python is running on WSL, install it (and pip) on windows and just update the requirements.txt regularly? Thank you! That probably isn’t too bad, now that I think about it.
|
# ? Aug 13, 2019 22:59 |
|
Krakkles posted:Dumb question I can’t figure out: how do you get PyCharm on windows to see packages installed to python on ubuntu/WSL? I haven't done it, but I think you just need to set up a remote interpreter, no?
|
# ? Aug 13, 2019 23:31 |
|
So I want to try this keras workspace. https://github.com/sipeed/Maix-Keras-workspace I don't like the instructions in the mobilenet directory. quote:
Is there a way of doing this without altering a systemwide file, and storing the model in my home directory? I know it's a big question, but all I really want to know is whether I should accept my fate or whether I can change this?
|
# ? Aug 14, 2019 01:07 |
|
Thermopyle posted:I haven't done it, but I think you just need to set up a remote interpreter, no? It looks like “install python on windows and maintain the packages in parallel” will probably work, though, so not the end of the world. Frankly, I’m impressed this all interoperates as well as it does.
|
# ? Aug 14, 2019 01:14 |
|
Krakkles posted:Half of it. That’s what I alluded to with the “part that’s working fine”, but it does not appear to support remote installation / verification of packages - so basically everything is highlighted as an invalid reference, because it doesn’t know that Django (for example) or other packages are installed. Well, I just tried it out...and it works fine? I just followed the instructions on that page I linked. It indexed whatever packages I had installed in WSL when I set it up. I typed import django in a file and it highlighted it with a red squiggly as I expected. I pressed Alt-Enter on it, and it, selected install package djang, it asked me for my password in WSL, and then installed it. The red squiggly disappeared. If I go into settings > Project:testproject > Project Interpreter, it shows the packages I have installed in WSL and I can add/remove from there. edit: It sucks that it doesn't support virtualenvs in WSL. Thermopyle fucked around with this message at 02:19 on Aug 14, 2019 |
# ? Aug 14, 2019 02:16 |
|
Weird, I guess I still didn’t figure it out. It appears to be working under windows, though, so could be worse. Edit: just for shiggles, tried again. Switched interpreter back, got the red squiggly, clicked “Install package Django” ... silent failure. (As in, no error message, *nothing* happens) Switch back to windows python where I installed from requirements, all the squigglies disappear. Krakkles fucked around with this message at 03:41 on Aug 14, 2019 |
# ? Aug 14, 2019 03:17 |
|
If you’ve set up the remote interpreter correctly then you shouldn’t even notice that it’s remote, so you must be doing something wrong. Are you on the pro version? Iirc remote interpreters might not be available for the free version.
|
# ? Aug 14, 2019 05:20 |
|
On a maybe related note I just learned what Vagrant is and even aside from replicating the production environment wow is this more convenient than trying to use a spare Raspberry Pi to host Redis and MySQL and run my rq-workers (cause I code on Windows and Windows does not fork()). PyCharm integration is nice too -- paying for Professional has been worth it if only to get me to learn what some of the tools it supports actually do.
|
# ? Aug 14, 2019 05:30 |
|
KICK BAMA KICK posted:On a maybe related note I just learned what Vagrant is and even aside from replicating the production environment wow is this more convenient than trying to use a spare Raspberry Pi to host Redis and MySQL and run my rq-workers (cause I code on Windows and Windows does not fork()). PyCharm integration is nice too -- paying for Professional has been worth it if only to get me to learn what some of the tools it supports actually do. If you think that's nice, wait until you learn about Docker! Krakkles posted:Weird, I guess I still didnt figure it out. It appears to be working under windows, though, so could be worse. I'd first try creating a new project and come up with a minimal reproducible case and then raise an issue on their issue tracker. They're very responsive to bugs, and even if it turns out to be a mistake on your part, they'll tell you what to do.
|
# ? Aug 14, 2019 16:46 |
|
Thermopyle posted:If you think that's nice, wait until you learn about Docker! Can you tell me how i can install Docker without HyperV? I thought it was impossible because only the enterprise edition can install without HyperV but I found a project called DockerToolbox. Then it went ahead and crashed on install because... of HyperV. For your consideration, I would like not to use HyperV because VirtualBox and me having many version of Windows to reverse engineer malware and other sorts of fun things.
|
# ? Aug 14, 2019 20:17 |
|
You can't have hyperv installed at all if you want to run virtual box. Also, docker toolbox is legacy and not recommended.
|
# ? Aug 14, 2019 20:28 |
|
EVIL Gibson posted:Can you tell me how i can install Docker without HyperV? I thought it was impossible because only the enterprise edition can install without HyperV but I found a project called DockerToolbox. I just use HyperV. However, Docker recently announced Docker for WSL 2, which doesn't use HyperV! Of course, it's a tech preview, and you have to be using an Insider Edition of Windows to use WSL 2. I would expect both of those limitations to not last too long. FWIW, I use KVM on my home server to run many different versions of Windows and stuff...you might be in a position to use that?
|
# ? Aug 14, 2019 21:09 |
|
Hey, I've never messed with any sort of image manipulation and I want to dick around with writing my own thresholds/convolutions/filters. Whats a good library for giving me access to an image's pixel's data?
|
# ? Aug 14, 2019 21:33 |
|
ImageMagick?
|
# ? Aug 14, 2019 21:40 |
|
I thought OpenCV was the thing people used. I used it to detect horizontal lines and squares on images once
|
# ? Aug 14, 2019 23:03 |
|
|
# ? May 15, 2024 06:05 |
|
Thermopyle posted:I just use HyperV. Unfortunately, WSL2 does appear to require Hyper V. https://devblogs.microsoft.com/commandline/wsl-2-post-build-faq/
|
# ? Aug 14, 2019 23:51 |