|
BigRedDot posted:Let me know if I can answer any questions I will, thanks!
|
# ? Oct 17, 2016 18:42 |
|
|
# ? May 9, 2024 03:53 |
|
Xarn posted:Need help with boto3 again . How much is it costing you? I can't imagine the deserializing/serializing loop takes any noticeable amount of time compared to the vast cost of talking to CloudWatch and writing to persistent store. edit: it looks to me like boto doesn't allow you to do this but all AWS services are ultimately just REST endpoints so you could always do it yourself with requests and don't deserialize the response. I'd be very surprised if this was actually a problem though.
|
# ? Oct 17, 2016 23:33 |
|
LochNessMonster posted:So now I've built my scraper/parser and put the data in a sqlite3 db, but now comes the presentation. If you really wanted to expand your tech exposure you could expose your db as an HTTP web service (in which case I'd recommend Flask and the plugin Flask-RESTful) and build the website client-side with React or Angular or something. I wouldn't necessarily recommend this over a server-side web site but it's a different approach that is widely used in industry today.
|
# ? Oct 17, 2016 23:38 |
Tons of problems involve people importing data from a CSV where the first row is the LABELS of the CSV. I always find myself converting these labels into namedtuples so I can access the CSV's info either positionally or by keyword. So I wrote some small helper functions to easily read and write labeled csvs. This is not high-level python, but hopefully it's useful or interesting to someone!
|
|
# ? Oct 18, 2016 07:28 |
|
Dr Monkeysee posted:If you really wanted to expand your tech exposure you could expose your db as an HTTP web service (in which case I'd recommend Flask and the plugin Flask-RESTful) and build the website client-side with React or Angular or something. I wouldn't necessarily recommend this over a server-side web site but it's a different approach that is widely used in industry today. I was considering this when looking into the differences between django and flask. And making each functionality a microservice (sorry for buzzwords) actually sounds like a great idea since I have no clue what I'll be doing next. It's just a hobby project I started and keep expanding. Making components small and (relatively) easy to change sounds like a good idea. That said, I'll first try and get a default flask application running. After that I'm gonna experiment with a RESTful api. When I'm ready I think I'll need to head over to the front end thread and start a religious war by asking whether to use React or Angular.
|
# ? Oct 18, 2016 08:07 |
|
LochNessMonster posted:I was considering this when looking into the differences between django and flask. And making each functionality a microservice (sorry for buzzwords) actually sounds like a great idea since I have no clue what I'll be doing next. It's just a hobby project I started and keep expanding. Making components small and (relatively) easy to change sounds like a good idea. That said, I'll first try and get a default flask application running. After that I'm gonna experiment with a RESTful api. When I'm ready I think I'll need to head over to the front end thread and start a religious war by asking whether to use React or Angular. falcon is pretty handy for rest apis
|
# ? Oct 18, 2016 08:18 |
|
Can someone explain to me why Python uses a mutex (the GIL) when you're green threading? Like, the best I can guess is that maybe the cpython has a risk of creating race conditions due to threads of cpython running faster than Python code in the process? Also, is Python itself already green threaded? It looks like, when you make a user thread in Python, you're actually threading into Python's underlying threading. I don't know, the poo poo confuses me. It's some next level inception bs. All I know is that Python seems to get increasingly slower the more I use threading.
|
# ? Oct 19, 2016 11:10 |
|
BedBuglet posted:Can someone explain to me why Python uses a mutex (the GIL) when you're green threading? Like, the best I can guess is that maybe the cpython has a risk of creating race conditions due to threads of cpython running faster than Python code in the process? You won't get a performance boost with threading alone. You need to either use the multiprocessing module or you need to unlock the GIL (using numba or Cython or something else) If you're interested here's a good post talking about the GIL and effective use of threading in Python: http://python-notes.curiousefficiency.org/en/latest/python3/multicore_python.html
|
# ? Oct 19, 2016 11:49 |
|
You seem to be swapping terminology a bit. A green thread isn't the same thing as a thread. Threads are provided by the OS, green threads are provided by you (well usually by a library you're using like eventlet/gevent/asyncio) and all of them run in a single traditional thread. I haven't really thought about it, but I can't think of any reason the GIL will have any effect on green threads since they all run in one regular thread. Here's a quickly googled and skimmed explanation of different ways of doing concurrency in Python: http://stupidpythonideas.blogspot.com/2015/01/greenlets-threads-and-processes.html (I think I got that all right, haven't done anything requiring threading in a couple years. I forget everything if I haven't used it in 6 months or more) Thermopyle fucked around with this message at 19:12 on Oct 19, 2016 |
# ? Oct 19, 2016 19:08 |
If you're looking for more information on Python re: concurrency, threads, async, and parallelism, the Python Cookbook (3rd ed), Fluent Python, and Essential Python all cover it pretty well.
|
|
# ? Oct 19, 2016 20:12 |
|
I have two machines, and for each machine I have the intervals during which it was working as a list of the start points and list of the end points of the intervals (for example [0, 5, 9] and [3, 6, 13] would indicate that the machine was working from time 0 to 3, 5 to 6, and 9 to 13). I would like to determine how to calculate the start and end points of intervals where exactly one machine was working and exactly two machines were working are the same time. For instance if the start and end points of the second machine intervals were [0, 7] and [2, 10] then the start and end points of intervals where exactly one machine was working are [2, 5, 7, 10] and [3, 6, 9, 13] and the start and end points of intervals where exactly two machines were working are [0, 9] and [2, 10]. I would like to be able to generalize this to three or more machines as well. Any pointers on how to get started?
|
# ? Oct 21, 2016 20:20 |
|
I haven't had an opportunity to use it but I came across this library recently that appears to be perfect for your problem.
|
# ? Oct 21, 2016 20:54 |
|
Jose Cuervo posted:I have two machines, and for each machine I have the intervals during which it was working as a list of the start points and list of the end points of the intervals (for example [0, 5, 9] and [3, 6, 13] would indicate that the machine was working from time 0 to 3, 5 to 6, and 9 to 13). mystes fucked around with this message at 21:38 on Oct 21, 2016 |
# ? Oct 21, 2016 21:36 |
|
mystes posted:Just store a position in each list and state for each computer and for each iterations take the smallest value from the values at the current positions and toggle the state of the computer that list belongs to and increment the position for that list. (If multiple lists have that value at the current position then toggle/increment all of them.)The value you took is then the current time and you have the state of all the computers at that time. Repeat until you have reached the end of all the lists. I think I understand what you described, and implemented it as follows: Python code:
Python code:
Python code:
|
# ? Oct 22, 2016 02:20 |
|
Seriously check out traces. Your problem is literally their first example on the first page of their documentation.
|
# ? Oct 22, 2016 04:11 |
|
Oh I misread your earlier post to be saying you had lists like [start1, end1, start2, end2] rather than like [[start1, start2], [end1, end2]]. Anyway, using traces as suggested by SurgicalOntologist:code:
code:
mystes fucked around with this message at 15:13 on Oct 22, 2016 |
# ? Oct 22, 2016 12:45 |
|
Does anyone have experience in hosting their own private PyPI setup? Ideally, this would be something I could run completely via a python script that doesn't require some apache/nginx configuration as I'm not sure I have access to that on the cloud provider I'm hoping to use for this.
|
# ? Oct 22, 2016 17:02 |
|
I have a quick question involving Halloween, a Raspberry Pi, and a doorbell. I'm building a Halloween "prank" doorbell that involves a Raspberry Pi with a servo controller, some relays and a pushbutton - essentially what happens is that my plain old doorbell is replaced with a doorbell that activates a Raspberry Pi, which performs one of five spoopy activities randomly when people come to trick or treat. These spoops invove sounds, lights and servos. The whole show is run in Python, which I'm very new to. What I'm having an issue with is the behavior of "while True :" loops, and the use of functions/GPIO calls - so I wasn't sure whether to post here or in the Raspberry Pi thread but I thought I'd start here. Further up in the code, I've built my functions; the five "doorbell_#" functions (where # is 1-5), the "ringring" function which selects one of the five randomly, and another function which pulses the LED on the doorbell itself (this is to mimic a heartbeat or breathing or something). It looks very cool when it works; the PWM controller board I have attached to the Pi gives it a very cool looking up & down ebb - plus I changed the LED in the doorbell itself from white to red. The issue is that I can either: 1). get my doorbell "heartbeat" function to loop properly - but then the doorbell button only registers presses in between loops, or 2). get my doorbell "heartbeat" to play one time only, and the doorbell button works any time after the "heartbeat" finishes It looks like the issue is that the "doorbellheartbeat()" function running is preventing the next line (waiting for the button to be pushed) from running. With the code as below (using RPi.GPIO) the heartbeat runs once and then the button works immediately afterwards (scenario #2 above): code:
code:
|
# ? Oct 22, 2016 17:07 |
|
It's been awhile since I used GPIO but I think you want to look into GPIO.add_event_detect. wait_for_edge blocks, which prevents the heartbeat from running. Your second attempt only checks the state at one specific time. Whereas add_event_detech basically gives you a flag as to whether an event occurred any time since you last checked. Edit: The more robust option is to do everything in asyncio but it looks like add_event_detect gives you some basic asynchronous functionality which is all you need. SurgicalOntologist fucked around with this message at 17:23 on Oct 22, 2016 |
# ? Oct 22, 2016 17:19 |
|
Two things I'd suggest looking into are for loops which will run a loop a set amount of times and interupts which can stop other functions. I'm new to gpio stuff so you'll have to do some searching.
|
# ? Oct 22, 2016 17:21 |
|
Oops
|
# ? Oct 22, 2016 17:36 |
with regards to jose quervo's question: This is how I would do it. Included is a test for your example. This is pure python - not familiar with the library mentioned. (Please indulge my love for namedtuples.) Python code:
code:
Eela6 fucked around with this message at 20:46 on Oct 22, 2016 |
|
# ? Oct 22, 2016 20:30 |
|
I've never used Pandas, but I'm giving it a go (mainly to spit out some charts) and I'm wondering what the general approach is for this problem I have some JSON data that's a mix of values and dicts, and I'm using pd.read_json to turn it into a dataframe. I want to filter rows based on a value in the dicts, but it looks like df[df.thing['thing_id']] won't work, because it's using the column data as hashed keys and you can't do that with a dict? I could filter the incoming rows by hand before creating a dataframe, but that seems a bit silly since pandas is entirely about working with datasets to get exactly what you want. And I could manually expand all the dicts into individual columns, but that seems a bit awkward too. What's the general approach to filtering this kind of thing? I'd post some of the JSON but the API site is now down But hopefully you get what I mean
|
# ? Oct 22, 2016 23:02 |
|
The correct way to do what you're trying to do is df[df.thing == 'thing_id']. Or do you have dictionaries in each cell? If so you need to flatten that out before constructing the dataframe.
|
# ? Oct 22, 2016 23:16 |
|
(Sorry, I meant df[df.thing['thing_id'] == 123], filtering on rows that contain a dict with a specific value on a key) Yeah each JSON object is like JSON code:
I mean it makes sense, I just want to do it the 'right' way - pandas looks cool, but most of it is way over my head, it feels like a lot of documentation takes your understanding of data analysis for granted
|
# ? Oct 22, 2016 23:43 |
|
Yes, pandas is going to have a hard time with such a complex structure. In particular it can't handle dictionaries in each cell very well. I mean you could do something likePython code:
and if that can't handle JSON in your specific structure you'll have to write a custom flattener. Shouldn't be too difficult and will give you the most control.
|
# ? Oct 23, 2016 00:33 |
|
sarcastx posted:I have a quick question involving Halloween, a Raspberry Pi, and a doorbell. It sounds like what you want is to listen for doorbell inputs while running the doorbell heartbeat simultaneously. One way to accomplish this is with threads: Python code:
|
# ? Oct 23, 2016 02:08 |
|
QuarkJets posted:
That did it! Thanks QuarkJets!
|
# ? Oct 23, 2016 15:00 |
|
SurgicalOntologist posted:But you're going to have a much nicer time if you flatten. Try this: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.json.json_normalize.html Yeah this looks workable once I get the hang of it, thanks!
|
# ? Oct 23, 2016 19:22 |
|
sarcastx posted:That did it! Thanks QuarkJets! Hey no problem I wrote that up after a few beers and looking back at my implementation, you really only need to create one thread; you could stick the heartbeat calls in a spawned thread and keep the doorball listening in a while loop in main(). In other words, the blurb I wrote uses 3 threads (the main thread + 2 spawned threads) but you really only need 2. Although there's no harm in having 3!
|
# ? Oct 23, 2016 20:18 |
|
SurgicalOntologist posted:Seriously check out traces. Your problem is literally their first example on the first page of their documentation. mystes posted:Oh I misread your earlier post to be saying you had lists like [start1, end1, start2, end2] rather than like [[start1, start2], [end1, end2]]. Anyway, using traces as suggested by SurgicalOntologist: Python code:
Eela6 posted:with regards to jose quervo's question:
|
# ? Oct 24, 2016 18:00 |
Makes sense! A well-regarded library is always a good way to go. Besides being a little bulky, there is probably some corner case I've missed in my implementation. That said, I enjoyed writing it and figuring out the 'tricks ' (like ignoring zero-width intervals and merging intervals of the form n: (a, b) , n:(b, c) --> n:(a, c)
|
|
# ? Oct 24, 2016 18:15 |
Is there a laconic way to return every X out of every Y elements of the list? Say, with X=2 and Y=4, output of foo([1, 2, 3, 4, 5, 6], X, Y) would be [1, 2, 5, 6]. I can write a for loop doing but, but it's going to be way too slow.
|
|
# ? Oct 27, 2016 13:53 |
|
cinci zoo sniper posted:Is there a laconic way to return every X out of every Y elements of the list? Say, with X=2 and Y=4, output of foo([1, 2, 3, 4, 5, 6], X, Y) would be [1, 2, 5, 6]. I can write a for loop doing but, but it's going to be way too slow. If I understand your question, you're looking for the basic slicing feature of python lists. Python code:
Tigren fucked around with this message at 15:10 on Oct 27, 2016 |
# ? Oct 27, 2016 15:07 |
|
cinci zoo sniper posted:Is there a laconic way to return every X out of every Y elements of the list? Say, with X=2 and Y=4, output of foo([1, 2, 3, 4, 5, 6], X, Y) would be [1, 2, 5, 6]. I can write a for loop doing but, but it's going to be way too slow. code:
|
# ? Oct 27, 2016 15:08 |
Tigren posted:If I understand your question, you're looking for the basic slicing feature of python lists. overeager overeater posted:
|
|
# ? Oct 27, 2016 15:46 |
Overager's solution is good but creates a new list. You can use a generator comprehension to lazily yield new entries, which avoids a lot of extra overhead from creating an intermediate list. If you do need an explicit list rather than iterator, you can just call list() on the generator, or use a list comprehension later. This is accomplished by replacing the brackets [] with parentheses () Python code:
|
|
# ? Oct 27, 2016 20:28 |
Eela6 posted:Overager's solution is good but creates a new list. You can use a generator comprehension to lazily yield new entries, which avoids a lot of extra overhead from creating an intermediate list. If you do need an explicit list rather than iterator, you can just call list() on the generator, or use a list comprehension later.
|
|
# ? Oct 27, 2016 20:35 |
|
I'm trying to generate a page for each dealer in my sqlite db, which kinda works, but when trying to display all vehicles for that dealer I'm running into some issues. I think my 2nd sql query isn't picking up the dealer name. And if it would, I'm not sure if it'd actually return all vehicles for each page. I'm kinda stuck on how to figure this out. Python code:
|
# ? Nov 2, 2016 20:21 |
|
|
# ? May 9, 2024 03:53 |
|
LochNessMonster posted:I'm trying to generate a page for each dealer in my sqlite db, which kinda works, but when trying to display all vehicles for that dealer I'm running into some issues. Your string substitution is out of place. dealerVehicles = query_db("select * from motorcycles where dealer = %s" % (dealer)) If you're using python3, take a look at .format() instead. https://pyformat.info/ Tigren fucked around with this message at 21:17 on Nov 2, 2016 |
# ? Nov 2, 2016 21:11 |