unpacked robinhood posted:

I have a small percentages of files out of a batch that don't parse well when opened with the default open(..)
I've managed to get around this by checking the encoding on each file with filemagic, and setting the encoding parameter accordingly.

Does it feel bad-practicy ?

the yeti posted:

Are there any libs that can actually create encrypted zips? Seems like zipfile and others are explicitly decrypt only (due to...licensing? Technical hurdles?)

OnceIWasAnOstrich posted:

I think libarchive can do that and there is a library with ctypes bindings to libarchive, though I don't know that those function bindings are implemented.

Is there a reason you can't call out to a binary for it?

Boris Galerkin posted:

Are there any websites that let me upload locally built packages (or upload via CI/CD) so that I can pip install them ~from the cloud~ ideally with a password and as cheaply as possible?

chutwig posted:

Does anyone have a pointer to some reasonably thorough benchmarks for evaluating performance when running typical data science/ML tasks? I'm currently looking at https://github.com/numpy/numpy/tree/master/benchmarks, but maybe one of you ML expert types has something even better.

The source of the request is that I'm working on some functional tests for our ML people so that we can establish a known baseline for the performance of their Jupyter notebooks, then start switching things out and quantifying the performance difference (OpenBLAS vs MKL, that sort of stuff). Turns out none of them have an existing set of comprehensive benchmarks for this purpose, and I'm doing a little research/questioning before we start writing our own or packaging up those Numpy benchmarks for our testing.

Falcon2001 posted:

I have a weird python question.

I've been having weird internet outages, so I whipped up a little program that pings a given URL once every second, then logs any errors with a timestamp to a file. For the purposes of having a tiny visual to go along with it, I have it print a '.' for successes, and a '!' for failures to the console. This works fine in, say, the VSCode console, but it doesn't update at all when I run it in the windows commandline or powershell through python.

When I had entire sentences for success/failure, it worked fine. If I make it so it logs a newline after every character, it also works fine (but does seem to only update every two characters?). Googling lead me to some weird regedit discussions on stackoverflow, but it didn't result in anything.

For examples:

Not working:

code:

while True:
    time.sleep(1)
    if ping(target) == None:
        logging.warning('Ping to '+target+' failed!')
        print('!', end="")
    else:
        print('.', end="")

Working:

code:

while True:
    time.sleep(1)
    if ping(target) == None:
        logging.warning('Ping to '+target+' failed!')
        print('!')
    else:
        print('.')

Any ideas?

QuarkJets posted:

Does it work if you use other characters, such as 'a' and 'b'?

cinci zoo sniper posted:

I’d recommend hacking tqdm like so for this use case.

Thermopyle posted:

Yeah, anytime you have weird issues with stuff not updating in the console like you think it should start looking for the flush options.

Foxfire_ posted:

You're writing to stdout and it's line buffered by default. It won't flush to the OS unless a buffer fills up or you write a newline.

use print(whatever, flush=True)

Thermopyle posted:

I don't think there's really an industry-standard way. It kind of just depends on what your infrastructure is like.

If I was looking to make something robust, but on a budget, I'd run an instance of Redis and then use python-rq.

Easy to set up, and easily distributed across as many machines as you'd like.

python-rq's documentation is easy to follow.

Thermopyle posted:

I don't think there's really an industry-standard way. It kind of just depends on what your infrastructure is like.

If I was looking to make something robust, but on a budget, I'd run an instance of Redis and then use python-rq.

Easy to set up, and easily distributed across as many machines as you'd like.

python-rq's documentation is easy to follow.

Thermopyle posted:

I've done both and both are pretty easy.

I don't really think there's any more "figuring out" with either of them.

Lambda cost money of course. Doing it with redis may or may not depending on your current infrastructure.

It's not terribly uncommon to find web sites blocking the IP ranges of Amazon, Digital Ocean, Microsoft, etc, to prevent automated stuff like this.

Falcon2001 posted:

Ah, thanks! That totally fixes it.

Solumin posted:

Can I ask why you're converting a datetime object to a string and back?

i vomit kittens posted:

I'm having some trouble working with datetime in a small app I'm creating. I'm able to convert a datetime object to a string using strftime, but when I try to convert the exact same string back into a datetime using strptime in order to search a database for it, I'm told that the formatting I'm using is not valid even though I literally copy/pasted the format from the strftime function.

The functions are:

new_ev.date_time.strftime("%-m/%d/%y %-I:%M %p")

datetime.strptime(date_time, "%-m/%d/%y %-I:%M %p")

The error I'm getting when the bottom one is called is "ValueError: '-' is a bad directive in format '%-m/%d/%y %-I:%M %p'"

Can anyone point out to me why I'm having this problem?

Solumin posted:

Can I ask why you're converting a datetime object to a string and back?

unpacked robinhood posted:

What's the correct way to keep track of a list of files on disk in a db, with irregular user chosen names ?
I'm pulling attachments from emails and turning them into rows in a table.
There's a string column that could hold a path. Ideally I wouldn't change the schema.

unpacked robinhood posted:

What's the correct way to keep track of a list of files on disk in a db, with irregular user chosen names ?
I'm pulling attachments from emails and turning them into rows in a table.
There's a string column that could hold a path. Ideally I wouldn't change the schema.

xtal posted:

Just stick the path there, relative to some chosen user-media-directory, and make sure that directory is locked down. You can also store the content type and size to avoid looking it up at runtime, but it's not necessary.

CarForumPoster posted:

EDIT: I should mention, we also replaced the filenames with UUIDs when storing because we immediately ran into collisions with filenames.

CarForumPoster posted:

xtal posted: