Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

Seventh Arrow: Jan 26, 2005

Is mask what allows it to apply row 1 to 'Account ID', row 4 to 'Account Name', etc.?

I want to manipulate it such that everything that returns True will push error info to a db table. Don't tell me though, I want to figure it out on my own.

I'm supposed to do this with stored procedures, but I really question whether SQL can do the same kind of logic. I'm pretty sure I'd end up having one SP for Account ID, one SP for Account Name, and so on.

edit: ahhh ok it's not 'mask' but 'col_len_map'...carry on...

Seventh Arrow fucked around with this message at 23:02 on Jan 22, 2023

# ? Jan 22, 2023 03:14

Adbot: ADBOT LOVES YOU

# ? May 16, 2024 02:28

C2C - 2.0: May 14, 2006; Dubs In The Key Of Life; Lipstick Apathy

QuarkJets posted:

Since you are using pycharm, try running your code in debug mode so that you can inspect your objects when an error occurs

This error message is giving an important clue: it is saying that "row" is a dict, not a string. This is because csv.DictReader returns a dict for each row of data. The keys in the dict are the column names.
Python code:
# Access the column labeled "Location" 
location = row["Location"]
# Strip away characters we do not want 
location = location.strip(' ()')
# store the remaining string in a list
locations.append(location) 
Then later you can iterate over these strings and do something with them
Python code:
for location in locations:
    lat, lon = location.split(",")
    print(lat)
    print(lon) 

This did the trick!!! :thanks:

# ? Jan 23, 2023 00:03

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

Really can't stress enough how important it is to be able to figure out how to get your code working in an interactive debugger, even if it's just PDB. There's so many times that print is going to either be misrepresentative, or let you down. There's obviously some weird situations where you can't do it, but a lot of folks (including myself) skip getting their debuggers setup at first because it's just extra time but it more than pays for itself when you need to explore a non-trivial issue, especially API calls or nested complex python objects.

I use iPython for similar concepts and basically have it in every dev environment I can get away with where there isn't some weird conflict preventing me from getting it working.

# ? Jan 23, 2023 02:16

QuarkJets: Sep 8, 2008

Pycharm and VSCode have very intuitive and easy to use debuggers, if you aren't using debuggers now then you should start right away.

# ? Jan 23, 2023 03:04

Data Graham: Dec 28, 2009; 📈📊🍪😋

It took me an embarrassingly long time thinking of the PyCharm debugger as a weird scary area of the app where there be dragons and I should fear to tread, but once I got to know it I would not be parted with it sooner than from one of my limbs.

e: If only I could figure out how to attach the debugger to my iPython session in the terminal I'd be set for life

# ? Jan 23, 2023 03:23

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

QuarkJets posted:

Pycharm and VSCode have very intuitive and easy to use debuggers, if you aren't using debuggers now then you should start right away.

FWIW Some professional environments have a weird enough internal build / dev system that getting a debugger working can be a pain, but it's still worth the up-front effort. I'm shocked at the amount of professional devs who go 'oh wow this is weird, welp better give up.'

Data Graham posted:

It took me an embarrassingly long time thinking of the PyCharm debugger as a weird scary area of the app where there be dragons and I should fear to tread, but once I got to know it I would not be parted with it sooner than from one of my limbs.

e: If only I could figure out how to attach the debugger to my iPython session in the terminal I'd be set for life

if you type %pdb in ipython, it will kick you to the PDB commandline debugger when an error or breakpoint is encountered, which is pretty nice. PDB is a little rough around the edges but is fine for a lot of stuff.

Falcon2001 fucked around with this message at 03:30 on Jan 23, 2023

# ? Jan 23, 2023 03:28

ConanThe3rd: Mar 27, 2009

I'm writing up my first python script using PyCharm and using a virtual environment (Which I might have fouled up as I imported the dependencies via PyCharm).

Is there a guide on how to make python package everything up including its dependencies for out of IDE use?

ConanThe3rd fucked around with this message at 14:16 on Jan 23, 2023

# ? Jan 23, 2023 14:08

Seventh Arrow: Jan 26, 2005

https://www.youtube.com/watch?v=W5p8v4yhxjk

# ? Jan 23, 2023 14:30

ConanThe3rd: Mar 27, 2009

Ah, ok, I should clairfy. I already have the project done and set up as a virtual enviroment and everything, I just need to know how to get the project to, for lack of a better word, compile it to a version I can use outside of PyCharm.

# ? Jan 23, 2023 15:05

Data Graham: Dec 28, 2009; 📈📊🍪😋

PyCharm shouldn't be necessary to run it even now. In the terminal, as long as your virtualenv is activated, you should be able to run the script just with "python myscript.py". Right?

So to "package" it you would generally include instructions on setting up a virtualenv and running it as above. No PyCharm required.

If you're talking about compiling it to an executable binary, or a package you can install by double-clicking something, that's something I don't know much about.

# ? Jan 23, 2023 15:08

12 rats tied together: Sep 7, 2006

zipapp (standard library, official python website), shiv, and pex are the things to google if you want to get started packaging python applications with their dependencies

# ? Jan 23, 2023 18:24

Macichne Leainig: Jul 26, 2012; by VG

I have a function to hit an API and download a bunch of files. It works but it does them one by one and it's pretty slow when you're trying to do thousands of them. What sort of thing should I look into to speed this up? Is this like a multiprocessing thing I'm going to need?

Edit:

I found where a coworker has been using process_map from tqdm.contrib.concurrent, I think I can use that as a learning point, would I be shooting myself in the foot using that?

Macichne Leainig fucked around with this message at 20:20 on Jan 23, 2023

# ? Jan 23, 2023 20:16

12 rats tied together: Sep 7, 2006

tqdm is fine. if you had to start from scratch, asyncio is pretty good these days

# ? Jan 23, 2023 20:23

Slimchandi: May 13, 2005; That finger on your temple is the barrel of my raygun

Macichne Leainig posted:

I have a function to hit an API and download a bunch of files. It works but it does them one by one and it's pretty slow when you're trying to do thousands of them. What sort of thing should I look into to speed this up? Is this like a multiprocessing thing I'm going to need?

Edit:

I found where a coworker has been using process_map from tqdm.contrib.concurrent, I think I can use that as a learning point, would I be shooting myself in the foot using that?

Multiprocessing pool (https://docs.python.org/3/library/multiprocessing.html) is probably a good place to start, with the caveat that the API may rate limit you, and you're much more likely to hit the limit if you are multithreading. So look into a retry decorator with exponential backoff too.

# ? Jan 23, 2023 21:39

Macichne Leainig: Jul 26, 2012; by VG

Thanks, thankfully it's not a third party API, it's one my company owns, the only rate limiting I'm aware of is for authentication and it's pretty easy to just save the token and reuse it. I might play with the multiprocessing pool as a bit more generic option outside of tqdm, though it doesn't matter I guess because tqdm is pretty well used in our dev scripts

# ? Jan 23, 2023 21:56

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

Macichne Leainig posted:

Thanks, thankfully it's not a third party API, it's one my company owns, the only rate limiting I'm aware of is for authentication and it's pretty easy to just save the token and reuse it. I might play with the multiprocessing pool as a bit more generic option outside of tqdm, though it doesn't matter I guess because tqdm is pretty well used in our dev scripts

As someone who interacts with 'internal' APIs a lot, I recommend you ask the owners of the API what their rate limiting is, because you can easily take down a service if you just burst it out immediately, depending on the 'cost' of the API call. It might be nothing and they're just like 'go for it', but certainly there's a lot of folks that skip this step and turn out to be a massive pain for the API team.

# ? Jan 23, 2023 22:13

Macichne Leainig: Jul 26, 2012; by VG

Yeah that's a good point. I don't want to be the guy who took down prod because I decided to thrash an API endpoint

# ? Jan 23, 2023 22:21

ConanThe3rd: Mar 27, 2009

ConanThe3rd posted:

Ah, ok, I should clairfy. I already have the project done and set up as a virtual enviroment and everything, I just need to know how to get the project to, for lack of a better word, compile it to a version I can use outside of PyCharm.

OK, I figured this one out so just to have it here for any other neophytes who are running against this.

What I was having a hard time doing was figuring out where the Virtual Environment fitted into the distributed version of the script when the answer is sort of an obvious "Nowhere" as what I wasn't getting was that the Virtual Environment is basically the python equivalent to nodejs' node_modules folder

Instead, what I needed to do was generate the list of dependencies (requirements.txt as most places note it) with

code:

$pip freeze > requirements.txt

and export that and the script code where anyone downloading it could then have those requirements downloaded with

code:

$pip install -r ./requirements.txt

# ? Jan 24, 2023 08:03

QuarkJets: Sep 8, 2008

ConanThe3rd posted:

OK, I figured this one out so just to have it here for any other neophytes who are running against this.

What I was having a hard time doing was figuring out where the Virtual Environment fitted into the distributed version of the script when the answer is sort of an obvious "Nowhere" as what I wasn't getting was that the Virtual Environment is basically the python equivalent to nodejs' node_modules folder

Instead, what I needed to do was generate the list of dependencies (requirements.txt as most places note it) with
code:
$pip freeze > requirements.txt
and export that and the script code where anyone downloading it could then have those requirements downloaded with
code:
$pip install -r ./requirements.txt

A more robust way to accomplish this is to publish your package to pypi while only defining the minimum set of requirements that your code actually needs. Then someone can just `pip install your_package_name` in their virtual environment. You may be surprised at how easy it is to publish a package

# ? Jan 24, 2023 09:24

FISHMANPET: Mar 3, 2007; Sweet 'N Sour
Can't
Melt
Steel Beams

I've got a computer science background, though out of college I immediately went into the operations side. I've done lots of "scripting" first in PowerShell and have relearned Python. But I find myself still thinking in PowerShell, and missing out on some of the more powerful Python patterns, like passing around functions in variables. Does anyone have a suggestion on some kind of material that might me help think a little bit more in Python? Ideally, I'm picturing some content that lays out a "problem" and solves it in more of a "scripting" way but then shows how to refactor and solve the problem in a more Pythonic way. But anything that pushes me in that direction would be helpful.

# ? Jan 24, 2023 17:13

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

FISHMANPET posted:

I've got a computer science background, though out of college I immediately went into the operations side. I've done lots of "scripting" first in PowerShell and have relearned Python. But I find myself still thinking in PowerShell, and missing out on some of the more powerful Python patterns, like passing around functions in variables. Does anyone have a suggestion on some kind of material that might me help think a little bit more in Python? Ideally, I'm picturing some content that lays out a "problem" and solves it in more of a "scripting" way but then shows how to refactor and solve the problem in a more Pythonic way. But anything that pushes me in that direction would be helpful.

This is probably my personal bias showing, but I'd recommend doing some code katas (codewars or leetcode are two sites in particular) which basically ask you to solve small problems with your language of choice. This will probably help underpin your fundamentals a bit more and get you more used to thinking in terms of 'how do I do this in Python'.

# ? Jan 24, 2023 18:35

Ben Nerevarine: Apr 14, 2006

ConanThe3rd posted:

OK, I figured this one out so just to have it here for any other neophytes who are running against this.

What I was having a hard time doing was figuring out where the Virtual Environment fitted into the distributed version of the script when the answer is sort of an obvious "Nowhere" as what I wasn't getting was that the Virtual Environment is basically the python equivalent to nodejs' node_modules folder

Instead, what I needed to do was generate the list of dependencies (requirements.txt as most places note it) with
code:
$pip freeze > requirements.txt
and export that and the script code where anyone downloading it could then have those requirements downloaded with
code:
$pip install -r ./requirements.txt

Having just gone down this rabbit hole myself, I can�t recommend Poetry highly enough. It handles virtual environments and packaging a lot more gracefully than some older solutions like what you�re describing (and there seem to be quite a few out there, all with their own set of headaches)

For example at the end of everything you tell Poetry to build with a single command, and it creates a couple distribution files, a wheel and I believe a zip file. With that in hand it�s as easy as �pip install *.whl� to install your package locally

# ? Jan 24, 2023 19:15

Jose Cuervo: Aug 25, 2004

Back with a SQLite question. I want to query a table and return all rows where the value in the SID column matches one of the SIDs in a list that I have.

I have tried

Python code:

c.execute("""SELECT SID, start_date FROM start_dates
            WHERE SID IN SIDs_to_use""",
          {'SIDs_to_use': SIDs_to_use}).fetchall()

where 'SIDS_to_use' is a Python list containing 10 SIDs. The error message I get is 'OperationalError: no such table: SIDs_to_use' and I cannot figure out how to get rid of it.

Edit: Figured out I was missing the :

Python code:

c.execute("""SELECT SID, start_date FROM start_dates
            WHERE SID IN :SIDs_to_use""",
          {'SIDs_to_use': SIDs_to_use}).fetchall()

but now the error is 'OperationalError: near ":SIDS_to_use": syntax error'.

Jose Cuervo fucked around with this message at 21:26 on Jan 27, 2023

# ? Jan 27, 2023 21:23

QuarkJets: Sep 8, 2008

Jose Cuervo posted:

Back with a SQLite question. I want to query a table and return all rows where the value in the SID column matches one of the SIDs in a list that I have.

I have tried
Python code:
c.execute("""SELECT SID, start_date FROM start_dates
            WHERE SID IN SIDs_to_use""",
          {'SIDs_to_use': SIDs_to_use}).fetchall()
where 'SIDS_to_use' is a Python list containing 10 SIDs. The error message I get is 'OperationalError: no such table: SIDs_to_use' and I cannot figure out how to get rid of it.

Edit: Figured out I was missing the :
Python code:
c.execute("""SELECT SID, start_date FROM start_dates
            WHERE SID IN :SIDs_to_use""",
          {'SIDs_to_use': SIDs_to_use}).fetchall()
but now the error is 'OperationalError: near ":SIDS_to_use": syntax error'.

It seems like what you're trying to do is create a statement that looks like this:

Python code:

"""SELECT SID, start_date FROM start_dates
            WHERE SID IN (1, 2, 3, ...)"""

But assuming that SIDs are integers, you can't just provide a list and have it auto-convert to a string-encased tuple; the typing is all wrong, SQLite wants you to be a lot more specific than that. You need to have a statement that explicitly maps each SID to a binding as part of a statement:

Python code:

c.execute("""SELECT SID, start_date FROM start_dates
            WHERE SID IN (?, ?, ?, ?)""", [sid1, sid2, sid3, sid4])

Since you just want to query every item in the list, it's pretty easy to generate that mapping yourself:

Python code:

sids = [1, 2, 5, 7, 10]  # for example
sid_bindings = ",".join("?"*len(sids))  # becomes "?,?,?,?,?"
cur.execute(f'select SID, start_date from start_dates
    where SID in ({sid_bindings})', sids).fetchall()

Since we need 1 binding per list entry, we create a string with the correct number of bindings then insert it using an f-string.

# ? Jan 28, 2023 07:42

wash bucket: Feb 21, 2006

I'm hitting the bricks from my old job where we were in "fire-fighting mode" for so long that everything slowly fell out of date. I'm trying to make a fresh start and unlearn some old habits. For Python/Django projects is Pycharm the new gold standard IDE? Or is Eclipse + Pydev still good enough?

Or should I look into getting Sublime Text set up with plug-ins and packages specific to Django?

wash bucket fucked around with this message at 19:43 on Jan 28, 2023

# ? Jan 28, 2023 19:23

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

The impression I have is that pycharm is the best ide, vscode is the most used editor.

# ? Jan 28, 2023 19:53

Jose Cuervo: Aug 25, 2004

QuarkJets posted:

It seems like what you're trying to do is create a statement that looks like this:
Python code:
"""SELECT SID, start_date FROM start_dates
            WHERE SID IN (1, 2, 3, ...)"""
But assuming that SIDs are integers, you can't just provide a list and have it auto-convert to a string-encased tuple; the typing is all wrong, SQLite wants you to be a lot more specific than that. You need to have a statement that explicitly maps each SID to a binding as part of a statement:
Python code:
c.execute("""SELECT SID, start_date FROM start_dates
            WHERE SID IN (?, ?, ?, ?)""", [sid1, sid2, sid3, sid4])
Since you just want to query every item in the list, it's pretty easy to generate that mapping yourself:
Python code:
sids = [1, 2, 5, 7, 10]  # for example
sid_bindings = ",".join("?"*len(sids))  # becomes "?,?,?,?,?"
cur.execute(f'select SID, start_date from start_dates
    where SID in ({sid_bindings})', sids).fetchall()
Since we need 1 binding per list entry, we create a string with the correct number of bindings then insert it using an f-string.

Got it. Is writing basic SQL statements like this the best way to interact with a database, or should I be trying to learn SQLAlchemy or something similar instead?

# ? Jan 28, 2023 19:55

Data Graham: Dec 28, 2009; 📈📊🍪😋

PyCharm is tits, but shell out for the pro version if you can because of the built-in near-zero-effort Django integration. It handles all the common use cases (venv, requirements, runserver in debug) pretty seamlessly. It was a little rough ~3 years ago but they've been actively improving it stride by stride recently.

Next step for them to really get nailed down is Docker integration. It's almost there, but it's got some dumb warts yet.

# ? Jan 28, 2023 20:08

wash bucket: Feb 21, 2006

All this talk of PyCharm, vscode, and Docker lead me to this: Stack Overflow Developer Survey 2022

That seems like a pretty good lay of the land. I suddenly want to look into Docker since it seems like a popular alternative to using virtual machines which I'm more familiar with.

# ? Jan 28, 2023 20:44

QuarkJets: Sep 8, 2008

Jose Cuervo posted:

Got it. Is writing basic SQL statements like this the best way to interact with a database, or should I be trying to learn SQLAlchemy or something similar instead?

I prefer this way personally, I don't use SQLAlchemy unless a project already uses it

# ? Jan 28, 2023 21:02

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

The March Hare posted:

The impression I have is that pycharm is the best ide, vscode is the most used editor.

I use VSCode daily at work and love it; might try out PyCharm now that I understand Python a bit more and can probably appreciate it more, but both of them are what I always see as the clear standout winners; and they conveniently fill the roles of 'free' and 'paid' options.

# ? Jan 29, 2023 02:45

Precambrian Video Games: Aug 19, 2002

Falcon2001 posted:

I use VSCode daily at work and love it; might try out PyCharm now that I understand Python a bit more and can probably appreciate it more, but both of them are what I always see as the clear standout winners; and they conveniently fill the roles of 'free' and 'paid' options.

PyCharm Community is free. JetBrains IDEs are pretty fully featured and there isn't a lot locked behind the paid versions that you'd need for personal use (and for professional use your employer should buy a drat license). The Jupyter integration/scientific view is one useful thing but still has bugs.

I didn't realize Eclipse's Python's plugin was any good... well, is it? I like PyCharm but have been frustrated by their glacial pace in bugfixing. VSCode devs are more responsive but I still feel like it's behind - for example, it was only recently that they added the option to have code outline/structire displayed on the right side (with file explorer/whatever else on the left).

# ? Jan 29, 2023 03:24

QuarkJets: Sep 8, 2008

Iirc the PyCharm community edition doesn't support development over an ssh session, but VSCode does

# ? Jan 29, 2023 04:20

wash bucket: Feb 21, 2006

eXXon posted:

I didn't realize Eclipse's Python's plugin was any good... well, is it?

Maybe once upon a time. Like I said we sank into fire fighting mode for so long we never took time to re-evaluate our workflow so we just kept using what we stared with.

Hence me asking how people actually work in 2023. I'm like a caveman thawed out from a block of ice over here.

# ? Jan 29, 2023 04:31

Precambrian Video Games: Aug 19, 2002

QuarkJets posted:

Iirc the PyCharm community edition doesn't support development over an ssh session, but VSCode does

True, but you can install PyCharm locally with just user access*, whereas as far as I can tell VSCode is impossible to install without root.

Partly for that reason I tried using Eclipse Theia for a while, essentially a FOSS version of VSCode, but... it's rough and hard to recommend unless you're truly dedicated to escaping Microsoft's tracking.

*then use remote desktop, X forwarding, Xpra, whatever. Whether this is better than remote dev via ssh is a matter of taste and connection stability, I suppose.

McCracAttack posted:

Maybe once upon a time. Like I said we sank into fire fighting mode for so long we never took time to re-evaluate our workflow so we just kept using what we stared with.

Hence me asking how people actually work in 2023. I'm like a caveman thawed out from a block of ice over here.

Right, so part of the reason I'm curious is that I used Eclipse CDT a lot and now sometimes work on C++ libraries with Python (pybind11) bindings. In principle, CLion would work since it comes bundled with PyCharm features (whereas PyCharm doesn't have any C++ capabilities), but it's basically built around CMake, which I don't use. I haven't had a great experience with C++ in VSCode so far.

# ? Jan 29, 2023 05:12

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

eXXon posted:

PyCharm Community is free. JetBrains IDEs are pretty fully featured and there isn't a lot locked behind the paid versions that you'd need for personal use (and for professional use your employer should buy a drat license). The Jupyter integration/scientific view is one useful thing but still has bugs.

I work for a major tech company; we have a site license for PyCharm; I just like VSCode (and the remote SSH thing wasn't really functional for PyCharm a year and a half ago when I started I think) so it's mostly a matter of 'I've got projects in flight I don't want to redo the setup for'. We also have a lot of weird internal build systems that there are internal VSCode/PyCharm plugins for, and last I used it the PyCharm one was awful and the VSCode one is great.

# ? Jan 29, 2023 05:50

QuarkJets: Sep 8, 2008

eXXon posted:

True, but you can install PyCharm locally with just user access*, whereas as far as I can tell VSCode is impossible to install without root.

VSCode has a "User Setup" option that lets you install it in a user-owned directory if you're on windows. If you're on linux there's a CLI installer that you can unpack to wherever you want and it has .rpm and .deb files that you unpack in the usual ways if you're not a root user

# ? Jan 29, 2023 06:22

BrainDance: May 8, 2007; Disco all night long!

Anyone know where the thread to ask for people to write simple programs for you is at? I swear I remembered SA having a thread like that but I can't find it.

I need some python to automate some text editing, I got chat logs, they're formatted like this;

code:

Username(userID) 2021-07-14 14:49:22
text text text

Because of the time stamps each username line is different. I wanna automate grabbing the text on the line after a specific username and dump all that out in a text file so I can, say, take everything I've said in a group and have a text file of all that without usernames and other people's text so I can use it to train gpt2.

# ? Feb 3, 2023 04:11

Seventh Arrow: Jan 26, 2005

well if you want to go for maximum irony, you could probably get GPT to also write the program for you.

# ? Feb 3, 2023 04:25

Adbot: ADBOT LOVES YOU

# ? May 16, 2024 02:28

BrainDance: May 8, 2007; Disco all night long!

lol, I thought about that, but I don't have access to chatgpt yet because they only accept American phone numbers on sign up (or, at least, not Chinese ones) and I don't see them opening up to Chinese numbers any time soon.

I could ask gpt-neo but it's not too great at this sort of thing

edit: I just did it with awk was easiest, but still not perfect, now I gotta find a way to remove all lines with Chinese text in them and leave only the English, no clue how to even approach that.
double edit: I just used grep, I guess this is too simple of a problem for python

BrainDance fucked around with this message at 06:05 on Feb 3, 2023

# ? Feb 3, 2023 04:32

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »