Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

shrike82: Jun 11, 2005

HumbleBundle is offering a bunch of Python books (mostly newbie/intermediate level) on the cheap -
https://www.humblebundle.com/books/python-book-bundle

# ? Apr 6, 2017 09:05

Adbot: ADBOT LOVES YOU

# ? Apr 18, 2024 23:11

QuarkJets: Sep 8, 2008

shrike82 posted:

HumbleBundle is offering a bunch of Python books (mostly newbie/intermediate level) on the cheap -
https://www.humblebundle.com/books/python-book-bundle

Seems like a good stack of stuff for up to $15. I checked a couple of the prices on Amazon and they were all at least $20 each, albeit an ebook sometimes isn't as convenient as a paperback

# ? Apr 6, 2017 10:40

oliveoil: Apr 22, 2016

The Python documentation says "Long integers are now stored internally either in base 2**15 or in base 2**30": https://docs.python.org/2.7/whatsnew/2.7.html#optimizations

What does that mean? I have never seen "base 2 to some power" used before. 2**15 = 32,768 and I really doubt this means that Python uses a base-32,768 number representation.

# ? Apr 7, 2017 04:36

breaks: May 12, 2001

I haven't looked at the implementation and I don't know anything about how such things usually work, but having read that I'd guess that's exactly what it means. With a system that can handle goofy stuff like int("9"*100000) it seems reasonable that it might make sense to convert to some ridiculous base like that so that a 16/32bit int is a digit. My guess would be they only use 15/30 bits of the number so that they can do normal math with the ints, peek at the most/two most significant bits, and handle carrying into the next digit with speed. Uh I guess I can't think of why 30 instead of 31 off the top of my head, but I'm sure there is some good reason.

Or I could be totally off base.

breaks fucked around with this message at 06:46 on Apr 7, 2017

# ? Apr 7, 2017 06:40

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

oliveoil posted:

The Python documentation says "Long integers are now stored internally either in base 2**15 or in base 2**30": https://docs.python.org/2.7/whatsnew/2.7.html#optimizations

What does that mean? I have never seen "base 2 to some power" used before. 2**15 = 32,768 and I really doubt this means that Python uses a base-32,768 number representation.

The general programming thread is a better place to ask this probably.

# ? Apr 7, 2017 14:47

Proteus Jones: Feb 28, 2013

oliveoil posted:

The Python documentation says "Long integers are now stored internally either in base 2**15 or in base 2**30": https://docs.python.org/2.7/whatsnew/2.7.html#optimizations

What does that mean? I have never seen "base 2 to some power" used before. 2**15 = 32,768 and I really doubt this means that Python uses a base-32,768 number representation.

http://stackoverflow.com/questions/9860588/maximum-value-for-long-integer

# ? Apr 7, 2017 14:51

Tigren: Oct 3, 2003

I've been reading Fluent Python based on this thread and Eela6's recommendation. Is there a better/more Pythonic way to do this? I'm parsing 'yum list <package>' for Installed and Available packages.

code:

[tigren@python ~]$ yum list httpd*
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.linuxfix.com
 * epel: mirrors.cat.pdx.edu
 * extras: mirrors.cat.pdx.edu
 * updates: mirrors.syringanetworks.net
Available Packages
httpd.x86_64                                                                           2.4.6-45.el7.centos                                                                     base
httpd-devel.x86_64                                                                     2.4.6-45.el7.centos                                                                     base
httpd-itk.x86_64                                                                       2.4.7.04-1.el7                                                                          epel
httpd-manual.noarch                                                                    2.4.6-45.el7.centos                                                                     base
httpd-tools.x86_64                                                                     2.4.6-45.el7.centos                                                                     base

Python code:

        allowed_strings = ['Available Packages', 'Installed Packages', package_name]

        for line in out:
            if not any([x in line for x in allowed_strings]):  #Ignores lines like 'Loaded plugins: fastestmirror'
                pass
            elif 'Available Packages' in line:
                add_to = available
            elif 'Installed Package' in line:
                add_to = installed
            else:
                add_to.append(line)

        return available, installed

# ? Apr 7, 2017 20:55

Space Kablooey: May 6, 2009

I'd indent the other conditions and invert the any to get rid of the pass, otherwise it looks fine to me.

# ? Apr 7, 2017 21:06

Tigren: Oct 3, 2003

HardDiskD posted:

I'd indent the other conditions and invert the any to get rid of the pass, otherwise it looks fine to me.

Fluent Python has got me looking for tricks everywhere!

Python code:

        allowed_strings = ['Available Packages', 'Installed Packages', package_name]

        for line in out:
            if any([x in line for x in allowed_strings]):  #Ignores lines like 'Loaded plugins: fastestmirror'
                if 'Available Packages' in line:
                    add_to = available
                elif 'Installed Package' in line:
                    add_to = installed
                else:
                    add_to.append(line)

        return available, installed

# ? Apr 7, 2017 22:01

onionradish: Jul 6, 2006; That's spicy.

You can even drop the list brackets within any() as most functions that take an iterable work with a generator expression too in 3.x:

code:

if any(x in line for x in allowed_strings):
    ...

# ? Apr 7, 2017 22:19

Tigren: Oct 3, 2003

onionradish posted:

You can even drop the list brackets within any() as most functions that take an iterable work with a generator expression too in 3.x:
code:
if any(x in line for x in allowed_strings):
    ...

I'm working with minimum python 2.4. I even have to implement my own any.

:negative:

Tigren fucked around with this message at 23:58 on Apr 7, 2017

# ? Apr 7, 2017 22:22

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

Wouldn't it be better to use a dictionary of allowed strings mapping to their lists? So instead of checking the line is in an allowed string, and then comparing it to those strings again by hand, you can just look up each line in the dictionary and append to its related list, with an if value or try/except bit for handling the missing lines

(Or at least use a constant if you're comparing the same string literal multiple times, for your health!)

# ? Apr 7, 2017 23:40

SurgicalOntologist: Jun 17, 2004

Edit: nevermind I see what you're doing. Carry on.

SurgicalOntologist fucked around with this message at 00:48 on Apr 8, 2017

# ? Apr 8, 2017 00:45

OnceIWasAnOstrich: Jul 22, 2006

I am completely baffled by this thing I have just discovered. I am loving around with making NNTP proxies (yeah yeah, don't start) and I'm modifying an old package called Papercut which came with a buggy mostly-formed proxy hidden in there as a plugin.

The basic idea is that this simply uses nntplib to re-create the requests the server receives and then reformat them back into the raw NNTP packets and send them back. I ran into an issue where articles would become slightly corrupted while doing this. I was making nntplib.NNTP.article() requests. This function returns a 4-tuple, where the 4th item is a list containing the lines in the article. I started comparing this list of bytestrings between raw nntplib requests to the underlying NNTP server vs my proxied server, and found a few lines in a few articles that were subtly different. In particular, if the line bytestring began with the two bytes 2e2e (hex format, 2e==".") both the proxy and the raw NNTP query would receive the line properly, but after being re-sent through the proxy you would lose one of those bytes, making the total string one byte shorter with no other changes. After testing this on a large number of articles I have not found any other corruption at all.

Why on earth is the SocketServer.StreamRequestHandler or the nntplib slightly mangling my data when there is a newline followed by two periods? I threw in a loop to add an additional period to any strings starting with two periods...and it worked, no more corruption.

Code to pull data and reformat into one long bytestring

Python code:

    def get_ARTICLE(self, group_name, id):
        #WHYYY?
        resp, nr, id, headerlines = self.nntp.head(id)
        resp, nr, id, articlelines = self.nntp.article(id)
        dobreak = 0
        #Get rid of the headers so we can jam them back on later?
        while 1:
            if articlelines[0] == "":
                dobreak = 1
            del articlelines[0]
            if dobreak:
                break
        #What the gently caress am I doing here? This poo poo is bananas.
        for i in xrange(len(articlelines)):
            if articlelines[i][0]=='.':
                articlelines[i]='.'+articlelines[i]

        return ("\r\n".join(headerlines), "\r\n".join(articlelines))

Code that handles the server response. This is a method in a SocketServer.StreamRequestHandler object. send_response() uses self.wfile which is the object provided by the StreamHandlerRequest for outputting data. I don't see any way that this gets modified.

Python code:

    def do_ARTICLE(self):

        if len(self.tokens) == 2 and self.tokens[1].find('<') != -1:
            # Message ID specified
            for b in backends.values():
                self.tokens[1] = self.get_number_from_msg_id(self.tokens[1], b)
                result = b.get_ARTICLE(self.selected_group, self.tokens[1])
                if result:
                    backend = b
                    # article_info = b.get_article_number(self.tokens[1])
                    article_info = (self.selected_group, self.tokens[1])
                    break

        if result is None:
            self.send_response(ERR_NOSUCHARTICLENUM)
        else:
            response = STATUS_ARTICLE % (article_info[0], article_info[1])
            self.send_response("%s\r\n%s\r\n\r\n%s\r\n." % (response, result[0], result[1]))

    def send_response(self, message):
        self.wfile.write(message + "\r\n")
        self.wfile.flush()

While writing this up I decided to investigate, and found this:

RFC977 posted:

Text is sent only after a numeric status response line has been sent
that indicates that text will follow. Text is sent as a series of
successive lines of textual matter, each terminated with CR-LF pair.
A single line containing only a period (.) is sent to indicate the
end of the text (i.e., the server will send a CR-LF pair at the end
of the last line of text, a period, and another CR-LF pair).

If the text contained a period as the first character of the text
line in the original, that first period is doubled. Therefore, the
client must examine the first character of each line received, and
for those beginning with a period, determine either that this is the
end of the text or whether to collapse the doubled period to a single
one.

I guess this answers why I have to do this. The nntplib client has stripped extra periods from the lines, and I need to regenerate them. I guess I don't need to post this, but :justpost:

# ? Apr 8, 2017 01:02

Ghost of Reagan Past: Oct 7, 2003; rock and roll fun

What's a good library for drawing an array of pixels? They would change over time (I'm not animating figures, though), and I plan on eventually feeding into some LEDs, but for now I'd rather prototype without wiring up a bunch of LEDs. I've thought of just doing it with Flask + Javascript but I'm wondering if there's some other good solution.

I'm semi-willing to use some other language, but Python's the language I use daily so I'd rather just stick with that unless I can't find an adequate solution.

# ? Apr 8, 2017 01:23

OnceIWasAnOstrich: Jul 22, 2006

Ghost of Reagan Past posted:

What's a good library for drawing an array of pixels? They would change over time (I'm not animating figures, though), and I plan on eventually feeding into some LEDs, but for now I'd rather prototype without wiring up a bunch of LEDs. I've thought of just doing it with Flask + Javascript but I'm wondering if there's some other good solution.

I'm semi-willing to use some other language, but Python's the language I use daily so I'd rather just stick with that unless I can't find an adequate solution.

numpy for the array and matplotlib imshow() to render?

# ? Apr 8, 2017 01:44

huhu: Feb 24, 2006

Ghost of Reagan Past posted:

What's a good library for drawing an array of pixels? They would change over time (I'm not animating figures, though), and I plan on eventually feeding into some LEDs, but for now I'd rather prototype without wiring up a bunch of LEDs. I've thought of just doing it with Flask + Javascript but I'm wondering if there's some other good solution.

I'm semi-willing to use some other language, but Python's the language I use daily so I'd rather just stick with that unless I can't find an adequate solution.

https://python-pillow.org/

Edit: How many LEDs are you talking about by the way? Are you planning to use Arduino, Raspberry Pi, or something else?

# ? Apr 8, 2017 01:45

Ghost of Reagan Past: Oct 7, 2003; rock and roll fun

huhu posted:

https://python-pillow.org/

Edit: How many LEDs are you talking about by the way? Are you planning to use Arduino, Raspberry Pi, or something else?

32x32 and a Pi for the prototype. We'll see whether a Pi can do what I actually have in mind at the end of the day, but I'll get there when I get there. But on a computer I can have a much higher resolution I'm probably gonna start with that. I'm okay with this taking a lot of iteration and work.

OnceIWasAnOstrich posted:

numpy for the array and matplotlib imshow() to render?

Well that's pretty easy. I could probably write it to render at 20fps or whatever.

praise be to numpy

# ? Apr 8, 2017 01:59

breaks: May 12, 2001

Tigren posted:

Fluent Python has got me looking for tricks everywhere!

Python code:

        allowed_strings = ['Available Packages', 'Installed Packages', package_name]

        for line in out:
            if any([x in line for x in allowed_strings]):  #Ignores lines like 'Loaded plugins: fastestmirror'
                if 'Available Packages' in line:
                    add_to = available
                elif 'Installed Package' in line:
                    add_to = installed
                else:
                    add_to.append(line)

        return available, installed

1) There is a potential future problem that results from the string literals being duplicated in two different places.

2) What happens if package_name is seen before either of the strings?

3) Why is the if any line needed at all?

# ? Apr 8, 2017 02:04

Tigren: Oct 3, 2003

breaks posted:

1) There is a potential future problem that results from the string literals being duplicated in two different places.

2) What happens if package_name is seen before either of the strings?

3) Why is the if any line needed at all?

Guess I should have posted more of the code:

Python code:

        available = add_to = []
        installed = []
        allowed_strings = ['Available Packages', 'Installed Packages', package_name]

        command = BASE_YUM_CMD + 'list %s*' % package_name
        out, err = run_command(command)
        for line in out:
            if any([x in line for x in allowed_strings]):
                if 'Available Packages' in line:
                    add_to = available
                elif 'Installed Package' in line:
                    add_to = installed
                else:
                    if 'x86_64' in line:
                        add_to.append(line)

1. What's the problem here? I'm not seeing it.
2. package_name won't be seen before either of the strings, but if it is, add_to is set to available to start.
3. So I don't capture lines that don't have one of those three allowed strings.

My user will pass in a package_name and get a list of available and installed packages matching that name.

# ? Apr 8, 2017 02:10

breaks: May 12, 2001

You're missing an 's' in the second installed packages line which is not immediately going to cause a problem but should probably be cleaned up.

You could turn the last else into an elif and then the any line is redundant.

Then your list can be turned into a list of categories instead of a list of categories and package names, and you can solve problem 1 by only having the category names in one place. As someone else suggested that place could also be a dict mapping category names to lists that you will append to.

# ? Apr 8, 2017 02:41

huhu: Feb 24, 2006

Ghost of Reagan Past posted:

32x32 and a Pi for the prototype. We'll see whether a Pi can do what I actually have in mind at the end of the day, but I'll get there when I get there. But on a computer I can have a much higher resolution I'm probably gonna start with that. I'm okay with this taking a lot of iteration and work.

Well that's pretty easy. I could probably write it to render at 20fps or whatever.

praise be to numpy

Definitely go Python. It works very well with the Pi, if you didn't already know that.

If you got money to burn, a guy at my hackspace built a really nice Neopixel 10x10 grid.
https://www.adafruit.com/products/1138

# ? Apr 8, 2017 03:31

Dex: May 26, 2006; Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

Tigren posted:

3. So I don't capture lines that don't have one of those three allowed strings.

echoing the dict suggestion above

Python code:

def get_packages(pkg):
    sections = ('Available Packages', 'Installed Packages')
    section = None
    packages = {}

    outtxt = run_command('yum list %s' % pkg) 

    for line in outtxt:
        line = line.strip()
        if line in sections: 
            section = line
            continue
        if pkg in line and section: 
            packages.setdefault(section, []).append(line)
    return packages

# ? Apr 8, 2017 06:39

Eela6: May 25, 2007; Shredded Hen

OnceIWasAnOstrich posted:

numpy for the array and matplotlib imshow() to render?

A uint8 array of dimensions [m, n, 3] works well to represent images on a pixel level. The library skimage (installed by default in anaconda) works well for basic image manipulation as well. E.G:

Python code:

import numpy as np
from skimage.io import imshow
RED, GREEN, BLUE = 0, 1, 2
img = np.zeros([100, 100, 3], dtype='uint8')
img[0:30, :, RED] = 255
img[31:61, :, GREEN] = 255
img[62:, :, BLUE] = 255
imshow(img)

[timg]http://i.imgur.com/bccSZGh.png][/timg]

Eela6 fucked around with this message at 16:57 on Apr 8, 2017

# ? Apr 8, 2017 16:54

Space Kablooey: May 6, 2009

Can someone explain to me what's going on here:

Python code:

Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = []
>>> x.append(x)
>>> x
[[...]]
>>> x[0]
[[...]]
>>> x[0][0]
[[...]]
>>> x[0][0][0]
[[...]]
>>>

I ran into this when I was mistakenly appending a list to itself, but I'm intrigued as to what's going on here.

Edit: nvm, it's pretty obvious.

Space Kablooey fucked around with this message at 18:31 on Apr 10, 2017

# ? Apr 10, 2017 18:25

shrike82: Jun 11, 2005

At work, I typically deal with small datasets pulled thru financial APIs (100K records at most) that I interact with using numpy/pandas/flat files.

I'm hobby tinkering around with IoT stuff on a Raspberry Pi to log metered data (e.g., temperature, humidity, MACs connected to home wifi) as well as logging other stuff. I'm using ThingSpeak and it works well for simple logging.

What's a solution for storing something that generates ~3K records every 5 minutes? I'd default to python-sqlalchemy-sqlite with the DB file backed up every day to DropBox. Is there a nicer free(-ish) online solution?

# ? Apr 11, 2017 05:07

breaks: May 12, 2001

Any of you dudes use an assertion library? I've looked at Sure and PyHamcrest and would be interested in alternatives, especially stuff with a different philosophy from those since they are a bit similar. This is for QA work, main goals are informative and readable errors and readability of the assertions themselves wrt less technical people.

# ? Apr 11, 2017 05:55

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Temperature and humidity tracking automatically makes me think of a time series database.

I haven't used it but I've heard of opentsdb. Or even good old RRDTool. You could probably set up your own service for either of those on a 5/month DigitalOcean server.

But then again, maybe a time series database isn't what you want, it's hard to say without knowing what you'll do with the data. No matter what you do I bet the DigitalOcean server idea would serve you well if you're really wanting a cloud/remote thingy.

InfluxDB is another good option, I'm pretty sure there's hosted versions of that out there if you didn't want to set up your own server. Then you could do sweet visualizations with Grafana.

# ? Apr 11, 2017 06:03

Cingulate: Oct 23, 2012; by Fluffdaddy

SurgicalOntologist posted:

I don't fully understand the algorithm but the first step is to try to do this with vectors operation. This might not work as written but may only need small tweaks. At least it should illustrate the idea of vectorizing the operation.
Python code:
x, y = np.meshgrid(range(0, nummonths), range(0, nummonths))
cohorts = data['nps'] * data['duration']**(y - x)

What's the point of the '0's here? Is this best behavior somehow?

# ? Apr 11, 2017 11:55