Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

Tigren: Oct 3, 2003

Thermopyle posted:

The one thing I don't like about more advanced typing annotations is that they can make the signature too long and noisy and those are the signatures they're most useful with. Syntax highlighting and other IDE features help with that, though.
Python code:
def hi(foo: Union[Callable[str, bool], MyOwnType]], bar: Sequence[Callable[[Union[str, int], Tuple[str, str], bool):

I know you know better, but one might suggest you could either get rid of the type hints or clean up that signature with a better function name.

# ¿ Mar 9, 2017 22:40

Adbot: ADBOT LOVES YOU

# ¿ Apr 29, 2024 09:37

Tigren: Oct 3, 2003

Jose Cuervo posted:

Got it, thanks!

And remember, in Python 3.6, dicts are now ordered.

Python Changelog posted:

New dict implementation

The dict type now uses a �compact� representation based on a proposal by Raymond Hettinger which was first implemented by PyPy. The memory usage of the new dict() is between 20% and 25% smaller compared to Python 3.5.

The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5).

# ¿ Mar 14, 2017 21:55

Tigren: Oct 3, 2003

Pissing me off today: back porting scripts to support RHEL5. Just, like, basic poo poo is missing from python 2.4 that I rely on all the time. Set comprehensions, any(), conditional expressions, argparse! We've still got about 20% of our infrastructure running on this 10 year old OS.

At least it's Friday. :guinness:

# ¿ Mar 24, 2017 22:45

Tigren: Oct 3, 2003

Thermopyle posted:

Yeah, you need a server running. Basically shopify will call an url on a server you control when orders are placed. Then your code does something when that happens. It will work fine on a raspberry pi.

Flask and Django both are both capable of doing this is like 20 lines of code or something. The hardest part will be learning how to do it because for some reason understanding how the web works is not easy despite it being pretty simple.

This is the perfect use case for Flask. You don't need the bells and whistles of Django.

Python code:

from flask import Flask, request

from raspberry_pi_funcs import blink_bright_light() # I don't know how the raspberry pi functions work

app = Flask(__name__)

@app.route('/', methods=['POST'])
def index():
        data = request.get_json()
	if data['total_line_items_price'] > 100:
		blink_bright_light()
	return "Success"

# ¿ Mar 31, 2017 17:28

Tigren: Oct 3, 2003

xpander posted:

Echoing the other replies that Flask/Django are perfect for this. On the operational side, check out Zappa for some sweet serverless action. This will let you use some AWS services(API Gateway, Lambda) to run the application code without having to keep a server online. You'll fall well within the free tier for development usage, and probably well into their production load as well.

How do you trigger local raspberry pi functions with Lambda?

# ¿ Apr 1, 2017 06:15

Tigren: Oct 3, 2003

I've been reading Fluent Python based on this thread and Eela6's recommendation. Is there a better/more Pythonic way to do this? I'm parsing 'yum list <package>' for Installed and Available packages.

code:

[tigren@python ~]$ yum list httpd*
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.linuxfix.com
 * epel: mirrors.cat.pdx.edu
 * extras: mirrors.cat.pdx.edu
 * updates: mirrors.syringanetworks.net
Available Packages
httpd.x86_64                                                                           2.4.6-45.el7.centos                                                                     base
httpd-devel.x86_64                                                                     2.4.6-45.el7.centos                                                                     base
httpd-itk.x86_64                                                                       2.4.7.04-1.el7                                                                          epel
httpd-manual.noarch                                                                    2.4.6-45.el7.centos                                                                     base
httpd-tools.x86_64                                                                     2.4.6-45.el7.centos                                                                     base

Python code:

        allowed_strings = ['Available Packages', 'Installed Packages', package_name]

        for line in out:
            if not any([x in line for x in allowed_strings]):  #Ignores lines like 'Loaded plugins: fastestmirror'
                pass
            elif 'Available Packages' in line:
                add_to = available
            elif 'Installed Package' in line:
                add_to = installed
            else:
                add_to.append(line)

        return available, installed

# ¿ Apr 7, 2017 20:55

Tigren: Oct 3, 2003

HardDiskD posted:

I'd indent the other conditions and invert the any to get rid of the pass, otherwise it looks fine to me.

Fluent Python has got me looking for tricks everywhere!

Python code:

        allowed_strings = ['Available Packages', 'Installed Packages', package_name]

        for line in out:
            if any([x in line for x in allowed_strings]):  #Ignores lines like 'Loaded plugins: fastestmirror'
                if 'Available Packages' in line:
                    add_to = available
                elif 'Installed Package' in line:
                    add_to = installed
                else:
                    add_to.append(line)

        return available, installed

# ¿ Apr 7, 2017 22:01

Tigren: Oct 3, 2003

onionradish posted:

You can even drop the list brackets within any() as most functions that take an iterable work with a generator expression too in 3.x:
code:
if any(x in line for x in allowed_strings):
    ...

I'm working with minimum python 2.4. I even have to implement my own any.

:negative:

Tigren fucked around with this message at 23:58 on Apr 7, 2017

# ¿ Apr 7, 2017 22:22

Tigren: Oct 3, 2003

breaks posted:

1) There is a potential future problem that results from the string literals being duplicated in two different places.

2) What happens if package_name is seen before either of the strings?

3) Why is the if any line needed at all?

Guess I should have posted more of the code:

Python code:

        available = add_to = []
        installed = []
        allowed_strings = ['Available Packages', 'Installed Packages', package_name]

        command = BASE_YUM_CMD + 'list %s*' % package_name
        out, err = run_command(command)
        for line in out:
            if any([x in line for x in allowed_strings]):
                if 'Available Packages' in line:
                    add_to = available
                elif 'Installed Package' in line:
                    add_to = installed
                else:
                    if 'x86_64' in line:
                        add_to.append(line)

1. What's the problem here? I'm not seeing it.
2. package_name won't be seen before either of the strings, but if it is, add_to is set to available to start.
3. So I don't capture lines that don't have one of those three allowed strings.

My user will pass in a package_name and get a list of available and installed packages matching that name.

# ¿ Apr 8, 2017 02:10

Tigren: Oct 3, 2003

chutwig posted:

If you have a long list of strings, like
code:
["blah.example.com", "blorp.example.com", "burp.example.com"]
...then I'd probably just do
code:
[x.replace('example.com', 'sub.example.com') for x in your_list_of_strings]
...assuming you're not worried about where "example.com" appears and whether there might be multiple instances of it (i.e., you don't need to anchor to the right).

Yep, that's where I'd go too.

code:

def filterIP(fullList):
   regexIP = re.compile(r'\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}$')
   return filter(lambda i: not regexIP.search(i), fullList)

groups = {key : [domain.replace('example.com', 'sub.example.com') for domain in filterIP(list(set(items)))] for (key, items) in groups.iteritems() }

# ¿ Apr 15, 2017 00:51

Tigren: Oct 3, 2003

Vivian Darkbloom posted:

Confused about an issue with passing functions. This might be related to tkinter weirdness, but I might just be doing something else wrong.
code:
import tkinter as tk
root = tk.Tk()
for i in range(10):
    tk.Button(root,text=i,command=lambda:print(i)).pack(side=tk.LEFT)
root.mainloop()
When I click any button it prints 9, even though each button was initialized with its own print function. Why is it behaving like this, and not instead printing the appropriate number for each button?

IMO, if you're going to be learning the GUI library from scratch you'd be much better served going with a small Flask app and learning the basics of HTML/CSS/JS.

That's an actual marketable skill with virtually infinite learning resources vs some weird, arcane language on top of a language that produces interfaces from 1995.

# ¿ Apr 26, 2017 15:33

Tigren: Oct 3, 2003

QuarkJets posted:

These descriptions both describe Flask

Touche. But you're not going to find a lot of employers impressed by your Tcl/Tk skills.

# ¿ Apr 27, 2017 02:01

Tigren: Oct 3, 2003

Thermopyle posted:

PyCon 2017 videos are up.

Awesome, thanks for the heads up. I really wish my company would send me to PyCon, or at least not make me use vacation days to go. I look forward to the videos every year and often go back and find valuable talks from years past.

I watched Tim Head's talk on MicroPython, which I've tried out before and really enjoyed. The talk made me want to sink my teeth into microcontrollers again. I've been meaning to do something with a bird house camera.

I'm also excited to see a few talks on async, which seems like something I should become more familiar with in TYOOL 2017. Kelsey Hightower's Kubernetes for Pythonistas also looks like a great talk.

# ¿ May 22, 2017 05:58

Tigren: Oct 3, 2003

LochNessMonster posted:

Practical example question about working with functions.

Say I want to talk to APIs and run a few calls against it. My idea would be to create a function with requests to set up a connection. After the connection would be established I'd like to use that to run several different functions I write for the different api calls I want to do.

How would I go about setting something like that up?

Sounds like what you want is a Session.

http://docs.python-requests.org/en/master/user/advanced/#session-objects

quote:

The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance, and will use urllib3's connection pooling. So if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection).

Thanks to urllib3, keep-alive is 100% automatic within a session! Any requests that you make within a session will automatically reuse the appropriate connection!

Tigren fucked around with this message at 22:26 on Jun 2, 2017

# ¿ Jun 2, 2017 22:23

Tigren: Oct 3, 2003

outlier posted:

A Flask question, maybe just a style one.

I'm tidying up a Flask web-service and was just going to stick in the footer:

- The current number of records ("Currently storing 323,000 foos!")
- A link to the latest news item (which are also stored in the flask db)

But it's not kosher to stick this sort of logic in footer template. And it's another two queries for each page. And there's no need to call (and recall) these two queries for every page, as they don't have to be precisely up to date.

So what's the neatest way to do this?

Like Thermopyle said, you want to use a cache for this. Depending on what kind of traffic your Flask app will see, you can either use memcached or Werkzeug has a built in SimpleCache object.

# ¿ Aug 7, 2017 16:17

Tigren: Oct 3, 2003

Hughmoris posted:

Does anyone have any good examples/blogs/libraries using functional programming in Python?

https://www.dontdoit.com

# ¿ Oct 27, 2017 01:39

Tigren: Oct 3, 2003

Mr Crucial posted:

I'm not sure if this is the right place for this question, but here goes. How do I add a CA certificate so that Python and Python-based apps trust TLS certs generated by that CA?

I'm running Ansible on a Centos 7 box. I have playbooks that connect to Windows devices over WinRM protected by a cert generated by the CA. Ansible works fine but the CA is not trusted so I get a lot of verification errors every time I run the playbooks.

Various googling has led me down the path of running
code:
python -c "import ssl; print(ssl.get_default_verify_paths())"
to find the location of the default trust store (/etc/pki/tls/certs/ca-trust.crt in my case), and adding a PEM-encoded copy of the CA public cert to it using openssl. This I've done, and using "openssl verify" to test confirms that openssl at least trusts the CA.

Ansible, however, stubbornly refuses to verify the certs. I'm at a bit of a loss as to how to get this working now. Can anyone point me in the right direction?

I haven't used this authentication method myself, but do the following host vars help?

code:

ansible_connection: winrm
ansible_winrm_cert_pem: /path/to/certificate/public/key.pem
ansible_winrm_cert_key_pem: /path/to/certificate/private/key.pem
ansible_winrm_transport: certificate

Source

Tigren fucked around with this message at 18:12 on Nov 30, 2017

# ¿ Nov 30, 2017 18:10

Tigren: Oct 3, 2003

Mr Crucial posted:

They don't unfortunately, because I'm not using certificate authentication, I'm using CredSSP auth over HTTPS which is a different matter. I did try adding the ansible_winrm_cert_pem like so, but it didn't work:
code:
ansible_port=5986
ansible_connection=winrm
ansible_winrm_scheme=https
#ansible_winrm_server_cert_validation=ignore
ansible_winrm_cert_pem=/etc/pki/tls/cert.pem
ansible_winrm_transport=credssp
It's frustrating as poo poo because all of the official Ansible documentation suggests using self-signed certificates on a WinRM target and turning off certificate validation - there's not a single word about how to do this properly.

Does setting the REQUESTS_CA_BUNDLE environment variable help? It looks like CredSSP auth is handled by requests-credssp.

I love that Ansible even just tells you to ignore cert validation. Super secure!

quote:

When the Ansible controller is running on Python 2.7.9+ or an older version of Python that has backported SSLContext (like Python 2.7.5 on RHEL 7), the controller will attempt to validate the certificate WinRM is using for an HTTPS connection. If the certificate cannot be validated (such as in the case of a self signed cert), it will fail the verification process.

To ignore certificate validation, add ansible_winrm_server_cert_validation: ignore to inventory for the Windows host.

Tigren fucked around with this message at 20:53 on Nov 30, 2017

# ¿ Nov 30, 2017 20:35

Tigren: Oct 3, 2003

Hughmoris posted:

I'm new to Pycharm and utilizing virtual environments, and I'm running Windows 10.

When creating a new project in Pycharm, I can't find a module that I want to install (https://github.com/divijbindlish/parse-torrent-name). Is my next best option to open up a console window, activate the virtual environment and install the module? Or is there a way to help Pycharm find the module for installation?

Phone posting, but you should be able to open the project interpreter settings and install packages there.

https://www.jetbrains.com/help/pycharm/installing-uninstalling-and-upgrading-packages.html

# ¿ Dec 2, 2017 00:48

Tigren: Oct 3, 2003

Hughmoris posted:

Thanks. That was the initial route I pursued but the package I needed wasn't in the available list.

Weird, works for me.

What is listed when you click on that "Manage Repositories" button?

Mine has https://pypi.python.org/simple listed.

# ¿ Dec 2, 2017 01:33

Tigren: Oct 3, 2003

Hughmoris posted:

How is python+selenium for filling out lots of repetitive forms? I noticed that some people on my project team are manually entering in the data for 2000+ users in to a web portal. They've asked for help but my eyes will fall out of my head if I have to manually type in crap.

I have all of the user data in a clean csv file. The steps that are needed are basically:

I log in to the web portal (just once)

Click on search field and enter user name

Click on said user

Fill in a couple of text boxes, check a couple of boxes, select values from a drop down list

save form

GOTO search for a user

I used AutoIT for a similar job a few years ago but I figured I'd give Python a try for this (plus I forgot AutoIT).

Do you think you could simplify this even more by just making a bunch of POST requests? See if you can spy on the transaction in Chrome Dev Tools.

[Edit]
Oh, another page...

# ¿ Dec 5, 2017 04:23

Tigren: Oct 3, 2003

Tigren posted:

And remember, in Python 3.6, dicts are now ordered.

This seems like it's no longer just an implementation detail, but Guido proclaiming dicts are now ordered by design starting in 3.7.

Guido van Rossum posted:

Make it so. "Dict keeps insertion order" is the ruling. Thanks!

https://mail.python.org/pipermail/python-dev/2017-December/151283.html

# ¿ Dec 20, 2017 00:27

Tigren: Oct 3, 2003

You might also want to look into Folium. It's a quick, easy way to build a slippy LeafletJS based map using OpenStreetMap tiles from Python.

https://folium.readthedocs.io/en/latest/quickstart.html#getting-started

# ¿ Jan 2, 2018 19:46

Tigren: Oct 3, 2003

Hadlock posted:

Is there a package/library similar to Go's "Cobra"? Command line flagging coffee organizational thing.

I'm not sure you're gonna find something exactly like cobra, but maybe check out Click

Python code:

import click

@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
              help='The person to greet.')
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    for x in range(count):
        click.echo('Hello %s!' % name)

if __name__ == '__main__':
    hello()

Bash code:

#And what it looks like when run:

$ python hello.py --count=3
Your name: John
Hello John!
Hello John!
Hello John!

# It automatically generates nicely formatted help pages:

$ python hello.py --help
Usage: hello.py [OPTIONS]

  Simple program that greets NAME for a total of COUNT times.

Options:
  --count INTEGER  Number of greetings.
  --name TEXT      The person to greet.
  --help           Show this message and exit.

# ¿ Jan 30, 2018 21:10

Tigren: Oct 3, 2003

SnatchRabbit posted:

Crossposting from the AWS thread but I'm a Python noob and I'm trying to parse some JSON data inside a notification event.

Has anyone used Lambda to parse SNS event? I've been trying to parse an AWSConfig rule's SNS events and using the data to do various things. I'm able to parse the data up to a point, but I can't seem to get the message JSON data that I want. I'm using python 2.7, and json, boto3 imports. I think my issue is the JSON is being read as a single key. Anyone know how to do this?
code:
from __future__ import print_function
import boto3
import json

def lambda_handler(event, context):
   message = event['Records'][0]['Sns']['Message']
   message = json.loads(message)
   print(message)
output:
code:
{u'configRuleNames': [u'restricted-sshv2'], u'awsRegion': u'us-west-2', u'messageType': u'ConfigRulesEvaluationStarted', u'recordVersion': u'1.0', u'notificationCreationTime': u'2018-03-29T22:08:34.631Z', u'awsAccountId': u'########'}
event data: https://dumptext.com/jyTfBcNl

What are you hoping for the end result to be? message = json.loads(message) returns a dict, and printing it will display the whole thing as a string.

# ¿ Mar 30, 2018 00:29

Tigren: Oct 3, 2003

Thermopyle posted:

Python 3.7 out.

Some of the new features :

http://w4t.pw/z7

Love this part:

quote:

The Order of Dictionaries Is Guaranteed

In Python 3.6, this ordering was just a nice consequence of that implementation of dict. In Python 3.7, however, dictionaries preserving their insert order is part of the language specification. As such, it may now be relied on in projects that support only Python >= 3.7 (or CPython >= 3.6).

# ¿ Jun 28, 2018 18:17

Adbot: ADBOT LOVES YOU

# ¿ Apr 29, 2024 09:37

Tigren: Oct 3, 2003

Bundy posted:

MariaDB and I think postgres let you easily implement versioning on tables. I personally go to json, SQL and toml for most of my needs.

If it's a simple CRUD app and you don't need truly high concurrency, and you need a DB, use sqlite. It's a single file, which can of course be versioned. There's also sqldiff which lets you diff two sqlite databases. If you don't need an RDBMS, then just store your poo poo in a json file. Even if you don't need a relational DB, use sqlite. There are times when it's even faster than writing to the filesystem.

# ¿ Mar 22, 2019 23:50

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python