|
quote not edit
|
# ? Mar 23, 2017 21:04 |
|
|
# ? May 2, 2024 21:56 |
|
Nope, you shouldn't need to figure out the html variable. response.text in your second block is the same as html.read() in your first. In other words, scrapy gets the html for you and passes it in as response. That is, if I'm understanding the scrapy API correctly based on what you posted.
|
# ? Mar 23, 2017 21:12 |
|
Yeah so I imagined this would work:Python code:
Also, I know I'm going to need to write come exceptions into that method later, once I know this will work. It ran without error, but didn't write to the CSV file! Getting close. FingersMaloy fucked around with this message at 21:48 on Mar 23, 2017 |
# ? Mar 23, 2017 21:39 |
|
It's throwing a syntax error because it's missing the except clause. By the way, you can rewrite any file opening operation as: Python code:
|
# ? Mar 23, 2017 21:45 |
As a small note, this part of your code:Python code:
Python code:
edit: EFB
|
|
# ? Mar 23, 2017 21:47 |
|
You've got a few for loops in there where you repeatedly assign the same variables, so they'll end up with whatever the last value was. If you're trying to write multiple records to the CSV file, you need to write one inside the loop each time around (or store them in say a list and then write the whole lot at the end)
|
# ? Mar 23, 2017 22:25 |
|
Yeah, you probably want to (a) decide if you want to write a new CSV for each page that is scraped, or just append to a single file; and (b) call writerow more than once (i.e. put it inside a loop).
|
# ? Mar 23, 2017 22:54 |
|
Thanks everyone! This is very helpful. I've reworked the CSV part to:Python code:
baka kaba posted:You've got a few for loops in there where you repeatedly assign the same variables, so they'll end up with whatever the last value was. If you're trying to write multiple records to the CSV file, you need to write one inside the loop each time around (or store them in say a list and then write the whole lot at the end) I think you mean these two sections: Python code:
I'm going to work making the whole thing create a list and then write the list to the CSV, if I can figure that out. Thanks again all!
|
# ? Mar 23, 2017 23:32 |
|
HardDiskD posted:It's throwing a syntax error because it's missing the except clause. except clauses are not required if there's a finally. code:
|
# ? Mar 24, 2017 00:51 |
|
FingersMaloy posted:
What I mean is you're doing a for loop, which assumes you're (possibly) working with multiple items. So in your first bit of code, you call that method that finds all the elements with a "timeago" class. Then the for loop iterates over each of those elements in turn, tries to get the value of its "datetime" attribute, and assigns that to your date_posted variable So if you have multiple "timeago"-class elements, you'll rewrite the value of date_posted each time, and in the end it will be set to whatever the last element said, and you'll lose the previous values. If that's what you want to do, there are better ways to go about getting the last element of a sequence. If you're only expecting one "timeago" element on the page, so you'll only write to your variable once, you don't need the for loop! (plus your intent is clearer) Python code:
baka kaba fucked around with this message at 10:45 on Mar 24, 2017 |
# ? Mar 24, 2017 10:42 |
|
Lysidas posted:except clauses are not required if there's a finally. Neat. I didn't know about that. Space Kablooey fucked around with this message at 13:45 on Mar 24, 2017 |
# ? Mar 24, 2017 13:40 |
|
What're my best options for platform-independent MP3 and FLAC playback in Python? I'm writing a media player to serve as a reference implementation for a new metadata format I'm writing, and most of the stuff that handles MP3 or FLAC either a. only works on Windows or b. is bundled in with a bunch of poo poo focused around game development that I'd rather not distribute with my program (like pygame). Right now I'm settled on pyglet (smaller and simpler than pygame at least) but it would be really keen if I just had a teeny-tiny package (or two, for each format) that did nothing but play back media, modulate the volume, etc.
|
# ? Mar 24, 2017 14:25 |
^^ No idea, but I would love to know the answer. I gave my second talk at San Diego Python last night! It went really well. Who in the thread is going to PyCon this year? It might be fun to have a goon Python dev lunch.
|
|
# ? Mar 24, 2017 18:03 |
|
Pissing me off today: back porting scripts to support RHEL5. Just, like, basic poo poo is missing from python 2.4 that I rely on all the time. Set comprehensions, any(), conditional expressions, argparse! We've still got about 20% of our infrastructure running on this 10 year old OS. At least it's Friday.
|
# ? Mar 24, 2017 22:45 |
|
Some stuff you may be able to pull out of __future__; according to the docs List Comprehensions were in 2.4, so presumably you can just run a list comprehension and then convert to a set
|
# ? Mar 24, 2017 23:16 |
|
QuarkJets posted:Try using a comma instead of a colon in the typehinting Forgot to say that this is what the problem was. Thanks!
|
# ? Mar 27, 2017 15:06 |
|
Hi I am a new python user just now working my way through Learn Python The Hard Way I'm wondering what a good framework for python is? Especially for beginners if that's a thing. I have 2.x and 3.x installed but I'm reading in a lot of places that 3.x is the way to go so I'm gonna need a framework that's for 3.x work. edit: Crap... is Atom text editor what I'm looking for? It says to download it but that's just a text editor. I'm guessing all I need is that at this point in my quest for programming skills. tricksnake fucked around with this message at 00:51 on Mar 28, 2017 |
# ? Mar 28, 2017 00:41 |
|
All you need is a text editor at this point. Don't worry about getting fancy too quick. (I followed the same book and I and several pros I know recommend it to a lot of people, it's a very good intro) Shaw suggesting Atom is a new one, but I just checked and I guess he's updated it since I last looked at the starting chapters. When you say "framework" I assume you mean an IDE - definitely overkill for you right now (unless you're used to using IDEs with other programming languages - but if you've been programming before, you probably already have a preferred text editor installed)
|
# ? Mar 28, 2017 01:19 |
|
Epsilon Plus posted:All you need is a text editor at this point. Don't worry about getting fancy too quick. (I followed the same book and I and several pros I know recommend it to a lot of people, it's a very good intro) Shaw suggesting Atom is a new one, but I just checked and I guess he's updated it since I last looked at the starting chapters. Ok thanks dude I really appreciate it. I'll just use Atom for now.
|
# ? Mar 28, 2017 01:30 |
tricksnake posted:Hi I am a new python user just now working my way through Learn Python The Hard Way I would recommend downloading Anaconda. It comes with a lightweight IDE, Spyder, that I find excellent for beginners to-intermediate programmers. In addition, Anaconda includes many of the most useful packahes and dependencies 'in-box', some of which can be frustrating to install otherwise. It has decent autocomplete and will warn you about syntax errors, and it has a very easy REPL work flow. Other good choices for IDEs are PyCharm and Visual Studio Code, but you might find them a little heavyweight for what you're doing right now. Good luck with Python! I hope you enjoy it
|
|
# ? Mar 28, 2017 02:52 |
|
I'm mostly repeating what Eela and Epsilon said: It doesn't matter what tools you use at this point. Atom and Visual Studio Code are simple, good text editors. Spyder's an IDE that's easy to use; you could try that as well: 'pip install spyder' in a terminal. You might like it because it has a console built in, and buttons to run your code. Don't mess with PyCharm yet; it can be overwhelming while you're learning. Installing Anaconda will make installing third-party packages easier.
|
# ? Mar 28, 2017 03:02 |
|
Yea I'll just stick with Atom until i start getting into the heavy duty stuff. Thanks for the recommendation I'll check out Anaconda.
|
# ? Mar 28, 2017 16:59 |
|
What's the current hotness for type annotations and checking? Mypy or something else?
|
# ? Mar 28, 2017 19:07 |
|
a witch posted:What's the current hotness for type annotations and checking? Mypy or something else? mypy
|
# ? Mar 28, 2017 19:15 |
|
Thermopyle posted:mypy Does mypy have support for pandas objects? So I could set the return/input type to be pd.Series or pd.DataFrame?
|
# ? Mar 28, 2017 19:50 |
|
Would this be the best way to log errors on a script that I'm running?code:
|
# ? Mar 28, 2017 20:39 |
|
huhu posted:Would this be the best way to log errors on a script that I'm running? Check out the logging module that comes in the standard library. It offers a cleaner interface for this type of stuff.
|
# ? Mar 28, 2017 20:43 |
huhu posted:Would this be the best way to log errors on a script that I'm running? Python has a standard logging module . However, if you're just doing something very basic, this is the 'pythonic' way to do what you're asking. Python code:
Python code:
|
|
# ? Mar 28, 2017 20:49 |
|
Awesome thanks for the quick replies!
|
# ? Mar 28, 2017 21:12 |
|
Eela6 posted:This is syntatically equivalent to the following: Just a small correction, you actually meant this: Python code:
|
# ? Mar 28, 2017 21:12 |
Indeed I did. Thank you. Edit: this is actually a great example of why to use 'with' over manual open/close: even with experience it's easy to gently caress it up doing it manually. Eela6 fucked around with this message at 21:41 on Mar 28, 2017 |
|
# ? Mar 28, 2017 21:33 |
|
vikingstrike posted:Does mypy have support for pandas objects? So I could set the return/input type to be pd.Series or pd.DataFrame? https://github.com/pandas-dev/pandas/issues/14468
|
# ? Mar 28, 2017 21:51 |
|
huhu posted:Would this be the best way to log errors on a script that I'm running? I know this is already answered, but here's an entire blog post on the topic of logging exceptions, with some helpful examples for Python 2 and 3. It includes how to log the stack trace. https://realpython.com/blog/python/the-most-diabolical-python-antipattern/
|
# ? Mar 29, 2017 12:54 |
|
Thanks for inadvertently bringing f-strings to my attention. I always use str.format() but that seems like a better way to do that...
|
# ? Mar 29, 2017 19:33 |
|
Baby Babbeh posted:Thanks for inadvertently bringing f-strings to my attention. I always use str.format() but that seems like a better way to do that... They're the best. It's weird that .format() has always bothered me as much as it has, but dang it sucks.
|
# ? Mar 29, 2017 20:16 |
|
Curious if anyone would be interested in a Flask thread? I'm starting a new job working full time with it and would love a place to discuss it more.
|
# ? Mar 30, 2017 04:59 |
Why not? I'd love to know
|
|
# ? Mar 30, 2017 06:23 |
|
Eela6 posted:Why not? I'd love to know Me too.
|
# ? Mar 30, 2017 13:44 |
|
nvm
Space Kablooey fucked around with this message at 15:32 on Mar 30, 2017 |
# ? Mar 30, 2017 15:23 |
|
|
# ? May 2, 2024 21:56 |
|
The Django thread barely gets enough traffic to justify its existence. You may be better off just talking Flask in here?
|
# ? Mar 30, 2017 15:34 |