Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Nippashish: Nov 2, 2005; Let me see you dance!

psydude posted:

I'm trying to write a simple script in python that effectively just automates a bunch of linux CLI commands and executes some perl/ruby scripts.

As far as I can tell, I can just use subprocess.call to issue commands and then pass any arguments, such as :
code:
subprocess.call(["cd", "~/DEPLOY/"])
subprocess.call(["perl", "/DEPLOY/test.pl"])
subprocess.Popen(["bash", "-c", "~/DEPLOY/vars.rb]
Is this even going to work? I've never used python this way before.

Consider using the sh module, it makes things like this much less ugly.

# ? Oct 9, 2014 20:43

Adbot: ADBOT LOVES YOU

# ? Jun 12, 2024 12:33

good jovi: Dec 11, 2000

psydude posted:

I'm trying to write a simple script in python that effectively just automates a bunch of linux CLI commands and executes some perl/ruby scripts.

As far as I can tell, I can just use subprocess.call to issue commands and then pass any arguments, such as :
code:
subprocess.call(["cd", "~/DEPLOY/"])
subprocess.call(["perl", "/DEPLOY/test.pl"])
subprocess.Popen(["bash", "-c", "~/DEPLOY/vars.rb]
Is this even going to work? I've never used python this way before.

Are you really married to Python for this? The subprocess module is so much more verbose and annoying than similar capabilities in something like perl or ruby. Or, hell, just write a bash script.

# ? Oct 9, 2014 22:31

suffix: Jul 27, 2013; Wheeee!

psydude posted:

I'm trying to write a simple script in python that effectively just automates a bunch of linux CLI commands and executes some perl/ruby scripts.

As far as I can tell, I can just use subprocess.call to issue commands and then pass any arguments, such as :
code:
subprocess.call(["cd", "~/DEPLOY/"])
subprocess.call(["perl", "/DEPLOY/test.pl"])
subprocess.Popen(["bash", "-c", "~/DEPLOY/vars.rb]
Is this even going to work? I've never used python this way before.

There are some problems with these examples:

"cd" isn't actually a callable program, it's a special shell command.
As a program it would only be able to change its own working directory, and not the working directory of the calling program.
To change the working directory of a Python script, use os.chdir(). This will then also be the starting working directory of any subprocesses you create. Or you can pass the "cwd" option to call/Popen to set the working directory for a single call.

The "~" shortcut for your home directory is expanded by the shell, which call/Popen bypasses by default. You can pass "shell=True" to call/Popen, or you can expand the path yourself with os.path.expanduser().

If your Python version is new enough to have it, you should use subprocess.check_call() instead of subprocess.call.
check_call() throws an exception if the command you run fails, while call() just returns the return code, and expects you to check it yourself.
Forgetting to check the return status of commands you run will make errors hard to debug, so you'll want to get in the habit of using check_call.

# ? Oct 9, 2014 23:14

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Karate Bastard posted:

In what way is he a douche? I have no idea who he is. I think I just found him on Google. Or some goon may have recommended the C book to me.

I don't want to derail the thread into a discussion about Zed Shaw but I guess the best example is when he got real mad at someone from yospos who put buttbot[*] on twitter. The buttbot butted one of his tweets. His reaction was to try to dox the person who made the twitter account and send menacing e-mails.

[*] a simple explanation: buttbot is just a silly bot who repeats what you said with one of the words replaced by the word "butt"

He's like a humorless poor man's maddox (who teaches programming).

# ? Oct 10, 2014 02:48

SurgicalOntologist: Jun 17, 2004

hooah posted:

Interestingly, it works on my laptop, but not on my desktop. However, I was able to use the command-line interpreter to correctly print the statement. What does this mean?

If you get different behavior in PyCharm's interpreter and the command-line interpreter, the easiest explanation is that you're using two different interpreters. In PyCharm you can find the interpreter it's using (or trying to use...) under "Project Interpreter" in Settings. On the command-line use which python to find what interpreter you're using there that works.

# ? Oct 10, 2014 03:41

psydude: Apr 1, 2008

Nippashish posted:

Consider using the sh module, it makes things like this much less ugly.

Okay, I'll check it out.

good jovi posted:

Are you really married to Python for this? The subprocess module is so much more verbose and annoying than similar capabilities in something like perl or ruby. Or, hell, just write a bash script.

Not particularly. I know a bit of Ruby, and most of our shop is using Ruby, but I'm just much more comfortable with Python. I could try re-writing it in Ruby tomorrow.

suffix posted:

There are some problems with these examples:

"cd" isn't actually a callable program, it's a special shell command.
As a program it would only be able to change its own working directory, and not the working directory of the calling program.
To change the working directory of a Python script, use os.chdir(). This will then also be the starting working directory of any subprocesses you create. Or you can pass the "cwd" option to call/Popen to set the working directory for a single call.

The "~" shortcut for your home directory is expanded by the shell, which call/Popen bypasses by default. You can pass "shell=True" to call/Popen, or you can expand the path yourself with os.path.expanduser().

If your Python version is new enough to have it, you should use subprocess.check_call() instead of subprocess.call.
check_call() throws an exception if the command you run fails, while call() just returns the return code, and expects you to check it yourself.
Forgetting to check the return status of commands you run will make errors hard to debug, so you'll want to get in the habit of using check_call.

Thanks for the input. I'll try modifying it tomorrow.

# ? Oct 10, 2014 04:28

hooah: Feb 6, 2006; WTF?

SurgicalOntologist posted:

In PyCharm you can find the interpreter it's using (or trying to use...) under "Project Interpreter" in Settings.

This was the problem - there was no interpreter selected. Thanks for the help!

# ? Oct 10, 2014 13:07

Karate Bastard: Jul 31, 2007; Soiled Meat

I've been using PyDev a great deal in the past, but I see PyCharm recommended a lot here. What's the selling point compared to PyDev?

# ? Oct 11, 2014 09:01

QuarkJets: Sep 8, 2008

Karate Bastard posted:

I've been using PyDev a great deal in the past, but I see PyCharm recommended a lot here. What's the selling point compared to PyDev?

PyCharm is built by the same team that built IntelliJ. It's a very nice and polished IDE

PyDev is basically an Eclipse addon. A lot of people dislike Eclipse, but if you like Eclipse then PyDev is probably going to feel pretty good

# ? Oct 11, 2014 10:25

Dominoes: Sep 20, 2007

I don't use inheritance much. I have three classes that have a similar structure, but have substantial differences in the details of many of their methods. There are a few methods that can be used for all three classes. I have a parent class that contains the shared methods and nothing else; the three others inherit from it.

Is there a better way to do this? Pycharm doesn't like it, so I assume this is non-standard; it complains about unresolved attributes for all self. variables and methods in the parent class. A different approach would be to include the shared methods in one of the three classes, then have the other two inherit it. What do you think?

# ? Oct 12, 2014 21:06

KICK BAMA KICK: Mar 2, 2009

In the class holding the shared methods, add an __init__ method in which you initialize the attributes used by the shared methods. Then call super().__init__() in the constructor of each of the inheriting classes -- PyCharm will even remind you to do this.

For unresolved methods, if they're present but implemented differently in each child class, make them abstract methods of the parent class. Can't link now but look at the abc module.

KICK BAMA KICK fucked around with this message at 21:17 on Oct 12, 2014

# ? Oct 12, 2014 21:14

Dominoes: Sep 20, 2007

Thanks. That made PyCharm stop flagging the code. It added boilerplate. Worth it for clarity?

Dominoes fucked around with this message at 22:43 on Oct 12, 2014

# ? Oct 12, 2014 21:44

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

Thanks. That made PyCharm stop flagging the code. It added boilerplate. Worth it for clarity?

I often initialize all instance variables in __init__ just for clarity. Helps you figure out how the class works.

# ? Oct 12, 2014 23:11

Liam Emsa: Aug 21, 2014; Oh, god. I think I'm falling.

I'm using cssselect to strip information from a site. I'm having a problem where I grab an address, but the street and city get clumped together. In the HTML code, they're separated by a tag, example:

code:

<td width="205" align="left" valign="top"><font face="Verdana,Arial,Helvetica,Sans-serif" size="2">4770 Landau Dr<br>Plano, TX 75024</font></td>

The code I'm using does this:

code:

    while c < 10:
        cells = rows[i].cssselect('td')
        info.append(cells[2].text_content().strip())
        i += 1
        cells = rows[i].cssselect('td')
        info.append(cells[3].text_content().strip()) // <---- this line
        i += 4
        c += 1

The line I indicated is the one that grabs the street address/state. And it gets stored like this:

code:

'4770 Landau DrPlano, TX 75024',

When I want it preferably like this:

code:

4770 Landau Dr
Plano, TX

# ? Oct 13, 2014 21:33

accipter: Sep 12, 2003

Liam Emsa posted:

I'm using cssselect to strip information from a site. I'm having a problem where I grab an address, but the street and city get clumped together. In the HTML code, they're separated by a tag, example:
code:
<td width="205" align="left" valign="top">4770 Landau Dr Plano, TX 75024</td>
The code I'm using does this:
code:
 while c < 10:
 cells = rows[i].cssselect('td')
 info.append(cells[2].text_content().strip())
 i += 1
 cells = rows[i].cssselect('td')
 info.append(cells[3].text_content().strip()) // <---- this line
 i += 4
 c += 1
The line I indicated is the one that grabs the street address/state. And it gets stored like this:
code:
'4770 Landau DrPlano, TX 75024',
When I want it preferably like this:
code:
4770 Landau Dr
Plano, TX

Why don't you focus more on your DOTA feeding and less on this? From this code is seems like info is being initialized as a string (e.g., info = ''), you could either initialize it as an empty list (e.g., info = list()) and then join the final address with new lines (e.g., '\n'.join(info)), or just add a line break after the first append:

code:

info.append('\n' + cells[3].text_content().strip())

# ? Oct 13, 2014 23:41

EAT THE EGGS RICOLA: May 29, 2008

I can't believe I'm asking this, but I need to write unit and functional tests for a bottle application without using any other modules. That's insane, right?

# ? Oct 14, 2014 00:59

good jovi: Dec 11, 2000

EAT THE EGGS RICOLA posted:

I can't believe I'm asking this, but I need to write unit and functional tests for a bottle application without using any other modules. That's insane, right?

Assuming you mean without using any other non-stdlib modules, no, thats fine. Unittest is perfectly serviceable.

# ? Oct 14, 2014 01:53

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

I am wondering if anybody here is comfortable interfacing django with a Microsoft SQL database using a system DSN and pyodbc. I am following the django 1.4 tutorial--it's what I have to work with--and I have finally hit the point where it's time to toy with the database. I am doing this to simplify a problem I'm having working with somebody else's django code. I get the same error, which makes life a little easier:

pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect); [01000] [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (Connect()). (2)')

I can use pyodbc to toy with the database fine:

(I am using Python 2.7)

code:

>>> import pyodbc
>>> cnxn = pyodbc.connect("DSN=Django_DSN")
>>> cursor = cnxn.cursor()
>>> cursor.tables()
>>> rows = cursor.fetchall()
>>> for row in rows:
...     print row.table_name
...
(table names comes out)

The System DSN is set with:
Name: Django_DSN
Description: django tutorial database
Server: localhost\SQLEXPRESS

And I selected to Change the default database to: django_tutorial

The MS SQL Server Management Studio has for my machine\SQLEXPRESS a database named django_tutorial. All good there.

My django database configuraton is this:

code:

DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        'NAME': 'django_tutorial',
        'OPTIONS': {'DSN': r'Django_DSN'},

    }
}

Most of my issues with the System DSN being incorrectly configured melt away when I tried the pyodbc test. I made sure to create it using the 64-bit tool, not the 32-bit one. It can successfully run a test connection. I'm assuming I have no configured it through django correctly. The only odd thing that kind of comes up is some people have Unicode issues and have to switch drivers. All the examples for that are for Linux, not Windows. Could I be suffering something similar? Is there a way for me to log the database connection command without having to wade through internals and guess?

# ? Oct 14, 2014 02:28

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

good jovi posted:

Assuming you mean without using any other non-stdlib modules, no, thats fine. Unittest is perfectly serviceable.

bottle is that web framework that does a bunch of magic stuff on import.

# ? Oct 14, 2014 05:10

sadus: Apr 5, 2004

Rocko Bonaparte posted:

Is there a way for me to log the database connection command without having to wade through internals and guess?

I have never tried using SQL with Python, but some related random thoughts:
* SQL Express doesn't come with SQL Profiler but this free thing works OK and says it logs login attempts too: https://expressprofiler.codeplex.com/
* Might create the DSN in the 32bit cliconfg.exe too just in case? I usually try to keep them both matching out of paranoia / just in case anyway.

# ? Oct 14, 2014 05:15

EAT THE EGGS RICOLA: May 29, 2008

good jovi posted:

Assuming you mean without using any other non-stdlib modules, no, thats fine. Unittest is perfectly serviceable.

I can test a lot of it without anything else, but not being able to use something like WebTest is causing me a ton of grief.

# ? Oct 14, 2014 15:39

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

sadus posted:

I have never tried using SQL with Python, but some related random thoughts:
* SQL Express doesn't come with SQL Profiler but this free thing works OK and says it logs login attempts too: https://expressprofiler.codeplex.com/
* Might create the DSN in the 32bit cliconfg.exe too just in case? I usually try to keep them both matching out of paranoia / just in case anyway.

I put on my big boy pants this morning and stepped through the Django code with a debugger. Guess what I figured out? The options field in the database configuration has to have all keys lowercase, not uppercase.

Wrong:

code:

DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        'NAME': 'django_tutorial',             # Or path to database file if using sqlite3.
        'OPTIONS': {'DSN': r'Django_DSN'},

    }
}

Right:

code:

DATABASES = {
    'default': {
        'ENGINE': 'sql_server.pyodbc',
        'NAME': 'django_tutorial',             # Or path to database file if using sqlite3.
        'OPTIONS': {'dsn': r'Django_DSN'},

    }
}

Because I used "DSN" instead of "dsn", it never detected the option, and instead tried to connect with the other options alone, which were insufficient to resolve the database.

I feel compelled to go on their mailing lists or whatever and asked what motivated that stupid decision. I mean, look at the configuration options one level above. They are all capitalized! DSN is an acronym! But nooooo! Options must be all lowercase because

# ? Oct 14, 2014 17:18

Hughmoris: Apr 21, 2007; Let's go to the abyss!

Does anyone have any simple projects they recommend to learn and apply the principles of OO programming? For the life of me I can't get a good grasp on using objects and classes etc...

# ? Oct 15, 2014 02:18

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Hughmoris posted:

Does anyone have any simple projects they recommend to learn and apply the principles of OO programming? For the life of me I can't get a good grasp on using objects and classes etc...

Everyone has a problem with "getting" OO. Then, after you get it, you'll have a hard time understanding why you had difficulty.

There's lots of examples if you google something like "python OO example".

The issue is that people have strong opinions about what is a good way to teach object-oriented concepts.

I guess what I would do if I were you (and is basically what I did 6 or 7 years ago) is look at a ton of examples and try implementing them. Then one day you'll just get it.

# ? Oct 15, 2014 02:48

namaste friends: Sep 18, 2004; by Smythe

No joke, the sun Java tutorial was where I "got" oo.

# ? Oct 15, 2014 02:51

FoiledAgain: May 6, 2007

I was working on something one day when I thought "Boy, it sure would be useful if I could put all this code together in one place, kinda like a function, but with attribu...OH that's what a class is for!". Same thing with decorators. Until you actually have the need for them, you can read all the tutorials on the internet, they still won't make any sense. Focus on accomplishing things with your code. Do projects. Eventually you'll need a class and you'll know it.

# ? Oct 15, 2014 04:05

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

FoiledAgain posted:

I was working on something one day when I thought "Boy, it sure would be useful if I could put all this code together in one place, kinda like a function, but with attribu...OH that's what a class is for!".

You can actually use attributes with functions! :science:

# ? Oct 15, 2014 05:05

Dominoes: Sep 20, 2007

Thermopyle posted:

Everyone has a problem with "getting" OO. Then, after you get it, you'll have a hard time understanding why you had difficulty.

There's lots of examples if you google something like "python OO example".

The issue is that people have strong opinions about what is a good way to teach object-oriented concepts.

I guess what I would do if I were you (and is basically what I did 6 or 7 years ago) is look at a ton of examples and try implementing them. Then one day you'll just get it.

Don't use them at first. Just use functions. You'll get to a point where your code gets complex and messy, and you're passing variables around in arguments in a confusing way. Then it'll make sense.

# ? Oct 15, 2014 07:29

tef: May 30, 2004; -> some l-system crap ->

Symbolic Butt posted:

You can actually use attributes with functions!

similarly, you don't need to do factories in python, constructors work like functions do, and you can substitute one for the other. you don't need to do builders in python, functions take named arguments. instead of design patterns, python has language features: functions, or iterators, generators, descriptors, with statements, named arguments, etc.

Hughmoris posted:

the principles of OO programming

you'll get a lot more use out of learning the python builtins, and you can actually get really far in python with a handful of classes to keep the application state together, and a few named tuples :3:

Hughmoris posted:

For the life of me I can't get a good grasp on using objects and classes

at a very rough level objects are a collection of procedures with shared state (variables) between them, i.e taking a set of functions, a handful of globals, and packing them together in an object. you can spend a lot of time reading up on design patterns and what not, but tbh, i think you'll get a lot more from diving into a python library that you use, checking out bits of the standard library to see how they work, and code review, than you will from a design textbook. that said, go and learn about state machines: these come up all the goddam time in various guises. all the goddam time.

the objects you write will depend on the problem you are solving. if anything learning design is not so much learning about the tools, but being able to deconstruct the problem into smaller chunks. don't worry about getting all of the technical jargon into your head. sometimes it pays to explore an idea through code. taste in design is often learned through mistakes. unfortunately, we don't tend to document the mistakes.

fwiw i hear refactoring is a good book.

Hughmoris posted:

Does anyone have any simple projects they recommend to learn and apply the principles of OO programming?

build a web crawler. then make it multi threaded/multi process.

# ? Oct 15, 2014 08:11

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

Don't use them at first. Just use functions. You'll get to a point where your code gets complex and messy, and you're passing variables around in arguments in a confusing way. Then it'll make sense.

And then a lot of people overdo it and use classes for everything.

Once you have classes everything looks like an object.

# ? Oct 15, 2014 08:11

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

For reference, it's still a judgment call. Sometimes I get away with just using dicts / lists, and then introduce a class for something later on. You don't need to make an up-front decision about it in Python as you need to do in some other crazier languages, and if you chose wrong, you can fix it.

# ? Oct 15, 2014 08:36

QuarkJets: Sep 8, 2008

Dominoes posted:

Don't use them at first. Just use functions. You'll get to a point where your code gets complex and messy, and you're passing variables around in arguments in a confusing way. Then it'll make sense.

I like this idea. I have a younger guy on my team who did this exact thing

Thermopyle posted:

And then a lot of people overdo it and use classes for everything.

Once you have classes everything looks like an object.

And then you become a Java programmer

# ? Oct 15, 2014 18:21

Liam Emsa: Aug 21, 2014; Oh, god. I think I'm falling.

Appending onto my earlier question, let's say I have a list of addresses in a text file that looks like this:

code:

Cedar Street Baptist Church of God
2301 Cedar StRichmond, VA 23223
New Hope Community Church
11731 SE Stevens RdPortland, OR 97266
Mountain Creek Community Church
5950 Eagle Ford DrDallas, TX 75249
Broomfield United Methodist Church
545 W 10th AveBroomfield, CO 80020
True Vine Baptist Church
435 S Ellison DrSan Antonio, TX 78245
Bettendorf Christian Church
3487 Towne Pointe DrBettendorf, IA 52722
Northminster Presbyterian Church
2450 E Ft Lowell RdTucson, AZ 85719
Raleigh First Church of the Nazarene
1329 US-70 WGarner, NC 27529

I want to put this into a csv file where the cells are NAME, STREET, CITY, STATE, and ZIP.

NAME is easy, as it's every other line starting with the 0th.

ZIP is easy, as it's just the last 5 characters.

STATE is easy, as it's just the 8th and 7th to last characters.

However, I'm trying to find out how on earth I can separate STREET and CITY. I was thinking of trying to find a way to search for every time there's a lowercase letter with an uppercase letter following it. This seems to work for almost every one, except for a few edge cases (like the last entry on that list). Any ideas?

# ? Oct 17, 2014 21:08

KICK BAMA KICK: Mar 2, 2009

Liam Emsa posted:

Any ideas?

This looks like it works on the sample you posted (and handles ZIP+4). Could easily have an error in an edge case -- I'm not even sure how I got the greediness right, just experimented a bunch.

e: Yeah, the lookahead for ', ' is unnecessary, and it doesn't seem to capture ZIP+4 after all. This seems to work.

KICK BAMA KICK fucked around with this message at 21:43 on Oct 17, 2014

# ? Oct 17, 2014 21:37

hooah: Feb 6, 2006; WTF?

I'm liking PyCharm so far, but there is one niggling annoyance I've encountered: after I debug a program, when I hit the "run" hotkey, it debugs again, rather than running as normal. If I instead run the program via the menu, it runs normally. Is there a setting for this or something, or is it just an oddity of PyCharm?

# ? Oct 17, 2014 21:45

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

hooah posted:

I'm liking PyCharm so far, but there is one niggling annoyance I've encountered: after I debug a program, when I hit the "run" hotkey, it debugs again, rather than running as normal. If I instead run the program via the menu, it runs normally. Is there a setting for this or something, or is it just an oddity of PyCharm?

Which "run" hotkey?

# ? Oct 17, 2014 21:50

hooah: Feb 6, 2006; WTF?

Thermopyle posted:

Which "run" hotkey?

I chose Visual Studio-like hotkeys when I first ran it, so ctrl+f5.

# ? Oct 17, 2014 22:36

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

KICK BAMA KICK posted:

This looks like it works on the sample you posted (and handles ZIP+4). Could easily have an error in an edge case -- I'm not even sure how I got the greediness right, just experimented a bunch.

e: Yeah, the lookahead for ', ' is unnecessary, and it doesn't seem to capture ZIP+4 after all. This seems to work.

Whoa, regex websites are getting better and better.

# ? Oct 18, 2014 01:41

Hed: Mar 31, 2004; Fun Shoe

Yeah I was going to say I've never seen that one before but :iia:

# ? Oct 18, 2014 12:04

Adbot: ADBOT LOVES YOU

# ? Jun 12, 2024 12:33

hooah: Feb 6, 2006; WTF?

Alright, forgoing the hotkey thing for a bit, why doesn't this work?

Python code:

def nested_sum(list):
    total = 0
    for x in list:
        if type(x) == 'list':  # Not too familiar with 'type', tried just list also.
            total += sum(x)
        else:
            total += x
    return total

print nested_sum([1, 2, [1, 2]])

I get a runtime error when x = [1, 2] because the if fails. Why doesn't type(x) return list (or 'list')?

# ? Oct 19, 2014 01:26

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »