Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

joebuddah posted:

I am trying to save data to a custom name to a default location.

fn = "order + order + ".txt"
fn = "'"+"C:/"+fn+ "'"
myfile = open(fn, 'w')

I am using ignition which uses python 2.5
Using import.os didn't work any hints would be appreciated. Thanks

which os commands were you trying? instead of C:/etc/etc, you should use / and the os.path.join function to build your path:

>>> import os
# Assuming we want /mydir or C:\mydir, leave the OS decide that
>>> target = os.path.join('/', 'mydir')
# Check if our target folder exists, if not make it.
# Do this for every 'new' folder in your target path, it won't automatically create the entire tree for you.
>>> if not os.path.exists(target):
... os.mkdir(target)
...
# Move to folder.
>>> os.chdir(target)
# Print out current working directory.
>>> print(os.getcwd())
c:\mydir

Adbot
ADBOT LOVES YOU

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

joebuddah posted:

That worked thank you.

I honestly don't know what I tried. Instead of commenting out what didn't work, I just kept trying different things. Thanks again.

one other thing from your snippet, you should use python's context managers('with' statements) for file I/O instead of remembering to try->do thing->finally close everywhere:

code:
with open('myfile.txt', 'wt') as outfile:
    outfile.write('blahblahblah')
once execution leaves the nested block, the file is automatically closed so you don't have to worry about it

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

baka kaba posted:

mkdirs will make all the parent folders though!

i should really read docs more often, i never knew this was a thing, ta

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

Dominoes posted:

Hey, this is probably old news to most of y'all, but using py.test is a clear winner over the builtin unittest. No imports, classes, or inheritance required, just write functions with name starting with 'test', and use an assert statement in the func. Run with 'py.test filename.py'

Compared to builtin unittests, immediate advantages are lack of boilerplate, and not acting finicky about directory structure. I spent a while troubleshooting issues stemming from unittest being picky about relative imports; no issue with py.test.

hypothesis has some weird thing about pytest fixtures that i can't remember off the top of my head so i don't use pytest at all, i'd be curious what you're doing that unittest is boilerplate for you compared to writing your own assertion logic for everything though

for anyone who works heavily with remote APIs, have you used betamax for unit/integration testing? i only discovered it recently but i'm in love, just wondering if anybody has some horror stories with it or if i'm justified in raving about it to everyone

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

Dominoes posted:

Python code:
import unittest

class Tests(unittest.TestCase):

    def test_things(self):
        self.assertEqual(5 == 5)
Python code:
def test_things():
    assert 5 == 5
The code present in the first example but not the second (ie class setup and method names) doesn’t contribute to the code's meaning, and requires lookup/memorization.

Python code:
import unittest

class TestThing(unittest.TestCase):

    def test_tuples_equal(self):
        self.assertTupleEqual(tuple1, tuple2)

Python code:
    def test_tuples_equal(self):
	# Spot the bug.
        assert all([a==b for a,b in zip(tuple1,tuple2)])
slightly fairer example and not even really demonstrating why you'd use unittest. use whatever makes you happy, but it's worth keeping in mind that unittest is still used for a reason

SurgicalOntologist posted:

What's it for, I've always wondered. When choosing between two APIs, and one is built around subclassing, I'm choosing the other one 90% of the time. If the advantage is to be able to use things like setup and teardown methods, I'd rather do it pytest's way. If it turns out I need something like that, add a fixture. No need to built up something more complicated from the beginning just because you might need it later.

depends mostly on preference, occasionally what you use in testing(that's the thing i couldn't think of earlier), and what your org is like i guess. i work in a mixed environment so xunit-style tests are a plus, if pytest rocks your world then that's cool too

edit: also depends on your views on subclassing. a lot of my code is made of subclasses so i don't really have a problem doing it in my tests

Dex fucked around with this message at 23:38 on Aug 22, 2016

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

SurgicalOntologist posted:

But why not just assert a == b?

Anyways I'm specifically curious what is gained from a subclass API, not necessarily trying to join into a py.test vs unittest debate.

depends on your test data whether that'll take an age. if you can always, always rely on a == b then unittest's assertions have nothing to offer you though, of course. if you do, then you run the risk of screwing up your assertions like i did in the second block and you now have a passing test for something that isn't right, and might not be immediately obvious to anybody why that is. there's other stuff in unittest too, it's just one of the main reasons i like it

not really sure what your problem with subclassing is so i'm going to say that's really just a preference thing on your part? i prefer subclassing and using setUpClass and setUp over scoping fixtures but it's not better, it's just how i do things. if unittest offers you nothing and pytest suits your needs, happy days, i just don't think a single import and (unittest.TestCase) is significant enough boilerplate to avoid unittest completely

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

SurgicalOntologist posted:

I still don't understand. What does assertTupleEqual do differently? From the docs I can't figure out any differences including efficiency. Your zip example makes me think that you have in mind short-circuiting, but a quick test shows that Python does short circuit when testing tuple equality (and I'll bite, the bug was not testing the length of the tuples. Although I still have no idea idea the point you're trying to make as to why you'd ever want to write your test like that).

i've seen more or less that assert in a codebase before, is why i included it. did i write it? nope. would i write it? probably, but i'm going to pretend not. would it make through peer review? you'd hope not, but poo poo happens. as it was, that test was green and edited by somebody who didn't _quite_ understand what he was rewriting originally, so the simplified code _should_ have meant a simplified test once it started failing. alas, mistakes were made but green is green. if the standard in place is "use assertTupleEquals when asserting tuples are equal" and so on, it acts as documentation for anybody new scanning through the tests before they do anything. i think this aspect of tests gets overlooked a lot

also i just thought asserting 5 == 5 was a bit of a bullshit example when i asked "what does pytest give you that takes a load of boilerplate in unittest", so i figured i was allowed to use one too :) i've used pytest before and never really had a problem with it, it just doesn't work for me these days so i use unittest.

quote:

Anyways, I feel like a case could be made that in general, APIs should not be based on inheritance, but I am not prepared to make that case myself.

inheritance is about sharing functionality imo, nothing more. if you need functions from three classes in your class, why not just inherit from all of them(assuming your mro isn't haywire)? if those functions could serve the codebase better ripped out of their classes and put somewhere saner, do that instead. if your class requires the user to override stuff and does nothing on its own, then sure, you've probably taken a weird design decision somewhere since that's more of an interface thing. i'm sure somebody else has a fantastic use case for fully abstract classes too, i just don't use any in my own stuff and tend to manage without them

quote:

I want to find out what these advantages of unittest are that are being implied but not really spelled out.

it's built-in, structured nicely and saves me a lot of boilerplate would be the core reasons i like it - i work mostly with json apis now, stuff like self.assertDictContainsSubset is just way simpler to scan through in a PR than converting .keys and .values to sets then using issubset/issuperset, or iterating through expected and comparing to actual, or whatever other method somebody wants to use to say a[keys1-15] are somewhere in dict b and we don't care that the rest doesn't match. if you think the structure sucks and it causes you to write tons of stuff you don't need then that's fine too.

i didn't say i thought unittest uses classes for a reason, i said people use unittest for a reason :) (if it does i'm not aware of it, never really thought too strongly about it tbh), but i do think unittest being class based is an xunit thing more than any core philosophy - which has its benefits when you work with people(or are a person) working on other languages more often than not. i think it was some recent episode of python testing(great podcast even if i'm misremembering why i started thinking about it) that had me considering whether or not the xunit origin is a bad thing for people who work mostly in python though, since a lot of the conventions violate what you'd do in your production code

edit: i should refresh pages more often

quote:

I believe you and Dex when you say it does add something, but I would like to know what that is, for my own education. (having never really used unittest but skipped right to pytest).

if you're using pytest and it's all going well you have no real reason to change imo. if you find yourself writing a lot of complicated asserts it might be worth looking at the ones built into unittest instead. that's about it. it can be worthwhile to flip between different test frameworks on different projects just for the sake of it though - doctest is pretty useless for my current work, but it was nice for another project where having the documentation executing itself gave the people using it a degree of confidence that it a) worked b) was documented clearly enough that they didn't have to learn the whole thing just to make a couple of changes later

Dex fucked around with this message at 04:19 on Aug 23, 2016

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

mike12345 posted:

I can't access it from Windows, can access it from inside the vm. I found this blog http://gisellezeno.com/tag/virtualenv.html
that says to forward it using ssh? ssh -L 54321:localhost:54321 user@server not sure but anyway it's not working for me

forwarding from something bound to localhost only like that requires your firewall and selinux policies to allow it. the official docs might make more sense https://ipython.org/ipython-doc/3/notebook/public_server.html#notebook-public-server

but yeah, virtualenv has nothing to do with ports which is why google was leading you nowhere

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

LochNessMonster posted:

I don't think I'll make it multithreaded, definately not straight away. I'm familiar with databases and can make intermediate sql queries so I might take a look at sqlalchemy as well.

The last time I did some programming was in college (java, 15 years ago or something). So besides the general concepts I'm completely new. Especially to python.

Is it ok to post my code here to let others have a look at it and point out design flaws and/or improvements?

go for it, if it's big enough that somebody who doesn't care is going to need to scroll past it then maybe make a project.log thread and just link it here when you want feedback.

the sqlalchemy tutorial uses an in-memory sqlite database, so it's worth checking out http://docs.sqlalchemy.org/en/latest/orm/tutorial.html to get familiar with how ORM works. you can of course just use raw sql queries everywhere if you want to, but that's no fun

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

LochNessMonster posted:

Thanks for the advice. I did look at the project.log but felt kinda silly for starting one as it's my first ever project.

I currently have 1 long python script that scrapes a site, turns it to a beautifulsoup object and parses information I want.

For now I print the output to stdout but I want to start storing it, hence my db question.

The code is starting to get longer so I was contemplating on making different scripts. One scraper, one paraser and ome that writes stuff to database. I don't want to hammer sites and scrape 10 pages 20 times in a row to test parsing code for example.

My biggest question is if I should create methods and/or classes to seperate code. I havent quite figured out when I should use those.

first project or five thousandth, there's no entry criteria for project.log.

start with functions, classes can come later(if they ever need to). try to break down your functions until they're doing one thing you can easily test within them, and/or replace easily later. for example, if my_parser.py contains five functions that have absolutely nothing in common other than being generally related to parsing, then there's no need for a MyParser class. once you start breaking things up in a way that makes sense to you, you'll figure out for yourself how your code should be structured. there's no hard and fast rules around "this MUST be a class", and you could easily write an entire app without using any.

also do yourself a favour and try to get in the mindset of test driven development. it might seem like serious overkill for a first project("why would i need a test for something that just returns a url string? waste of time!"), but i find writing tests up front helps me reason about the flow of an application a lot easier than banging out a bunch of code that just does it first and trying to make it sane later.

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

LochNessMonster posted:

Any guides/tutorials/guidelines I can look into on that? What I've done so far is just test every minor thing I'm trying to build as a seperate standalone app with static and/or test data.

the official docs are a good starting point: http://docs.python-guide.org/en/latest/writing/tests/ . basically let the code tell you my_cool_function is returning 'i am very cool' by calling it and asserting the value, instead of printing it out and reading it yourself. this is one of the reasons why TDD helps me design my applications, too - instead of "oh i need an app that does x, y, z and also saves the world", break it down to one function you need _now_, write tests for that, then implement it.

don't get too hung up on which testing framework you want to use. i use unittest most of the time, because it suits me. if you experiment with a few of them and find one that fits you better, roll with it

quote:

I'm not sure how vcr will help me do this easier (without making it too complex for me). Looking at your code snippets confuses me as I have no idea how to use it in my current code. I really am a beginner at the bottom level here.

i actually posted in this thread earlier asking about https://betamax.readthedocs.io/en/latest/ , which is based on vcr. it's very, very cool if your app makes a lot of web requests and you don't want to use stubs or mocks, but might be a bit in-depth if you're not up on unit testing your code yet. keep in mind for now i guess, but the basic idea for your use case would be:

- you have a function that gets a web page, parses it, then fetches the other eight pages(making up numbers, shh)
- if you have a unit test for this function, it's going to run really slowly since you're waiting for nine network requests to complete
- if you're using vcr/betamax and you've already run that test, the requests(headers, bodies, status codes, the lot) are already written to disk so it'll just replay those instead of making actual web calls

so, an automatic way of doing what you're already doing by saving the files locally. from onionradish's example, synopsis.yaml would contain the request for 'http://www.iana.org/domains/reserved'.

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!
i've never used pyaudio, but from a quick look, instead of blocking during recording you can use a callback: http://people.csail.mit.edu/hubert/pyaudio/#wire-callback-example

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

JVNO posted:

I'm curious why the blocked form of the code wouldn't be valid? The intention is to record the entire PyAudio stream from the experiment (up to about 30 minutes) and dump it to a single wave file.

i'm just guessing from the code you posted(it's monday and i'm ill so reading is for nerds), but in her example she's reading before the questions are presented. in the code block you posted, you're reading afterwards. the 'hang' you're referring to is possibly just the recording actually starting after the questions have been posted, since:

Python code:
for j in range(0, int(RATE / CHUNK * RECORD_SECONDS)):
    data = stream.read(CHUNK)
...reads as "from now until however many seconds, start recording", to me, whereas the callback example has a start_stream which should kick off that reading in the background(assuming your callback function is doing that)

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

LochNessMonster posted:

Thank you, this what I had in mind as a concept, but I had no clue if it would be a good and/or efficient way of doing it. I also didn't know what type I should've used. Thanks for helping me figuring this out!

After I manage to do this, I can create a loop based on Suspicious Dishs sql query to write to a database.

you could also use sqlalchemy for this. first create your database entity:

Python code:
from sqlalchemy import Column, String, Integer
from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class Motorcycle(Base):
	__tablename__ = 'motorcycles'
	id = Column(Integer, primary_key=True)
	brand = Column(String)
	mileage = Column(Integer)
	year = Column(Integer)
	dealer = Column(String)
then you can use Motorcycle for your inserts:

Python code:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from yourmodule import Motorcycle

engine = create_engine('sqlite:///yourdb')
Session = sessionmaker(bind=engine)
# This creates your table, if it doesn't exist.
Motorcycle.metadata.create_all(engine)

# Populate this with actual data
results = [Motorcycle(brand='whatever', mileage=1, year=2000, dealer='whoever'),
           Motorcycle(brand='welp', mileage=10, year=2000, dealer='whoever')]
session = Session()
for bike in results:
    session.add(bike)
session.commit()
bit more overhead but it can be easier to maintain if your project starts growing.

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

LochNessMonster posted:

I'm pretty sure I'll be running into a lot of design flaws in a while, but figuring those out seems like half the fun.

i don't think i've ever got through an entire project without rethinking something i did at the start and could have done better. getting something working and being happy with how it works are two totally different things, i barely ever accomplish the latter. it's good to know when to just leave something be though otherwise you'll never finish anything!

sqlalchemy is definitely overkill for what you're trying to do, it's just handy to keep in mind if things start growing. the more you understand using database connections and sql directly, the easier the sqlalchemy stuff is to figure out, though, so there's no real rush to start using it - was just throwing it out there as an option to look into if you're curious

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

LochNessMonster posted:

yeah, you're right. It's a list containing dictionaries.

I'm really new to this, and still struggling a bit with the different types of data structures (and how to proces them)

since you're iterating through a list, just use arcticzombie's example for each dict in it:
Python code:
for testDict in mylist:
    cursor.execute("INSERT INTO motorcycles VALUES(?, ?, ?, ?, ?, ?, ?)",
        testDict["DealerId"],
        testDict["Dealer"],
        testDict["Brand"],
        testDict["Type"],
        testDict["Price"],
        testDict["Milage"],
        testDict["Year"]

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!
py3's open function handles unicode encoding for you, py2 does not. the codecs package can do it though:

Python code:
import codecs

s = u'\xa3'
with codecs.open('output.txt', 'w', encoding='utf-8') as f:
    f.write(s)
...gets me a lonely little pound sterling in a file:

$ cat output.txt
£

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

LochNessMonster posted:

I was considering this when looking into the differences between django and flask. And making each functionality a microservice (sorry for buzzwords) actually sounds like a great idea since I have no clue what I'll be doing next. It's just a hobby project I started and keep expanding. Making components small and (relatively) easy to change sounds like a good idea. That said, I'll first try and get a default flask application running. After that I'm gonna experiment with a RESTful api. When I'm ready I think I'll need to head over to the front end thread and start a religious war by asking whether to use React or Angular.

falcon is pretty handy for rest apis

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

huhu posted:

After hours of frustration with a script I'm writing, I've come to realize I could have been using CSVs instead of Excel files. So, lesson learned, if you can deal with CSV files instead of Excel do it. If you're wondering why, Excel columns can start at 1 or "A" however everything in Python starts at 0.

i use openpyxl for excel stuff

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!
[flake8]
max_complexity = 10
max_line_length = 9999

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

lifg posted:

Does anyone have experience or recommendations on building Python software with virtualenv and Docker?

I've used them both for toy applications, but not together. They seem to solve the same problem, just at different levels, so I don't *think* there's a benefit to using virtualenv if I'm also using Docker.

I've never used virtualization (of any sort) for software development, so this is all new to me.

what are you using for your builds? we use a swarm and yet another docker plugin to run our builds - each build spawns a brand new container so there's no need for virtualenv. if you're intending to use docker compose to stand up a bunch of build slaves that'll be persisted for more than a single build, you'd want virtualenv

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!
i haven't used openshift but i see you have a couple of options there - custom dockerfiles, hooking in with jenkins pipelines(which i've worked with _a lot_), and more. so it's really a question of how you want to work. if you just want to code in your ide and run your tests etc locally, and ignore the final artifact that pops out of your build system, then you don't need docker locally. your build process could just be running a custom dockerfile that installs your deps and runs your unit tests as part of every commit, but only pushes the image on merges to master. so something like

code:
FROM my-base-python-image:whatever

RUN mkdir -p /opt/myapp
ADD . /opt/myapp/
WORKDIR /opt/myapp/
RUN pip install -r requirements.txt && python -m unittest discover tests/ && flake8 && rm -rf tests/
CMD ["howeveryourunyourapp"]
so if you wanted a local image, you'd just run "docker build -t mydevimage ." in your app directory and it would execute the above, which you could then run with "docker run mydevimage". or you could just leave your build systems do the docker bits that they need for deployment, and maintain your local dev environment however you want.

Dex fucked around with this message at 22:39 on Dec 3, 2016

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!
from some stuff i looked at last year, go's concurrency for cpu intensive threads was around 50x cpython's. however, go has some seriously nasty leaks that are completely beyond your ability to control, so you will almost certainly end up with memory issues that are near impossible to isolate, so if you have even the slightest reason to care about memory consumption, it seems like a terrible choice

i'd aim for pypy before running python in go if i was willing to make tradeoffs on certain libs since it's apparently significantly more performant than cpython, but i don't actually need to so i've only done some light reading on it

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!
Popen works as a context manager, fyi, you can use "with Popen(blah) as proc:" to get your waiting and file descriptor closing and so on for free

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

socialsecurity posted:

I've started looking into this but it seems it would require quite a bit of rewrite I didn't think sending a json back to a html post would be this much of an undertaking but thank you I think in the end going down this path would be easier to work on down the road.

falcon is my go-to for quick and easy apis:

Python code:
import json
from wsgiref import simple_server

import falcon


class MyThing(object):
    def on_post(self, req, resp):
        data = req.stream.read(req.content_length or 0)
        # do stuff with your data
        resp.status = falcon.HTTP_200
        resp.body = json.dumps({'stuff': 'here'})


app = falcon.API()
mything = MyThing()
app.add_route('/', mything)

if __name__ == '__main__':
    httpd = simple_server.make_server('0.0.0.0', 8000, app)
    httpd.serve_forever()
gives us:
code:
$ curl -X POST http://localhost:8000
{"stuff": "here"}
you could run it with gunicorn or whatever wsgi you want, too

Dex fucked around with this message at 02:38 on Jan 29, 2017

Adbot
ADBOT LOVES YOU

Dex
May 26, 2006

Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

Boris Galerkin posted:

I've been trying out pytest and it's really neat. My test codes look much cleaner than with unittest, but what I like the most is the colored red/green output. I do something like TDD so I'm constantly alt-tabbing or tmux pane switching etc to another terminal to re-run tests. Is there already an established way to tell pytest something like "monitor test folder x and re-run your tests every single time source code in folder y changes"?

pytest-xdist gives you a watch option(-f).

quote:

And going further (but maybe too Vim specific) is there a way to have pytest color my vim status bar red/green depending on status so I don't even have to look over to the other terminal unless I need to read the status messages?

e: what would be even better is if it would add information to my vim status line, just something short like: "b3,r66; b8,r23", telling me that one test failed in buffer 3 on line 66 so I could just switch my vim buffer directly to "3" and then see which test failed.

i haven't used vim for actual dev in a while, but are you using some plugin like python-mode or just rolling your own config? maybe do something like run pytest on save and check for v:shell_error instead of watching?

personally i just use pycharm and toggle auto-test on though

  • Locked thread