laxbro posted:

Newbie question: I'm trying to build a web scraper with Beautiful Soup that pulls table rows off of the roster pages for a variety of college sports teams. I've got it working with regular HTML pages, but it doesn't seem to work with what appear to be Angular pages. Some quick googling makes it seem like I will need to use a python library like selenium to virtually load the page in order to scrape the html tables on the page. Would a work around be to first use Beautiful Soup, but if a table row returns as None, then call a function to try scraping the page using something like selenium. Or, should I just try to scrape all of the pages with selenium or a similar library?

laxbro posted:

Newbie question: I'm trying to build a web scraper with Beautiful Soup that pulls table rows off of the roster pages for a variety of college sports teams. I've got it working with regular HTML pages, but it doesn't seem to work with what appear to be Angular pages. Some quick googling makes it seem like I will need to use a python library like selenium to virtually load the page in order to scrape the html tables on the page. Would a work around be to first use Beautiful Soup, but if a table row returns as None, then call a function to try scraping the page using something like selenium. Or, should I just try to scrape all of the pages with selenium or a similar library?

baka kaba posted:

Also Selenium was a real pain last time I used it, they updated something so the Firefox webdriver wasn't included anymore and everything broke. There might be a better scraper out there?

baka kaba posted:

do I need to screw around parsing numbers from filenames here

Symbolic Butt posted:

Most likely yes. If you make it explicit that the number part is an integer you'll get the sorting you want:

>>> ('file', 2) < ('file', 10)
True
>>> ('file', '2') < ('file', '10')
False

baka kaba posted:

pathlib is cool, Path.iterdir() is not because it's not sorted. Is there a natural sort I'm missing (where 'file2' < 'file10') or do I need to screw around parsing numbers from filenames here

Eela6 posted:

pathlib Paths are hashable and orderable (and therefore sortable) so you can just call sorted!
*snip*

Death Zebra, can you give us the full stack trace?

Eela6 posted:

Death Zebra, can you give us the full stack trace?

HardDiskD posted:

How does it deal with a folder that has file1, file9, file10 and so on?

baka kaba posted:

Badly, I was using sorted() until I realised my data was out of order, tracked it down to the files coming in all file1, file10, file11. That's why I was hoping a natural sort was lurking in the standard library

Oh well, maybe I can work around it with that split tuple suggestion up there, thanks! It just feels bad because it's brittle and Windows does natural sort ordering, missed opportunities

Eela6 posted:

baka kaba posted:

CheckIo is pretty great for problems once you get into it. I tried HackerRank for a bit with Java 8 and some of the questions were shockingly bad, like they were taking community submissions with no quality control. No idea what the Python section's like though!

But those kinds of things tend to be isolated problems, Eela's suggestion is good because you get practice actually designing and building an application in Python, working with classes and messing with command-line args and whatever else. So you're ready to just dive in and build whatever the interviewer wants, without being unfamiliar with some of the basic language stuff

HardDiskD posted:

Just had a quick glance at it, but you could use os.path.splitext to check the file extension. use the suffix property from Path.

Vivian Darkbloom posted:

I wish I had any completed applications to port! I've been messing around with small projects so I won't have nothing to show off, but I have not been great, historically, at finishing personal projects.

ynohtna posted:

Here's my common snippet to convert an iterable into natural sort order:

Python code:

import re
def sorted_nicely(iter):
    """ Sort the given iterable in the way that humans expect."""
    convert = lambda text: int(text) if text.isdigit() else text
    alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ]
    return sorted(iter, key=alphanum_key)

Usage is:

Python code:

    for p in sorted_nicely(seen_paths):
        print(p)

dougdrums posted:

code:

$ pip3 search make sandwich

pysandwich (0.1.2) 		python sandwiches.
panini (2.4.6) 	 		The Simple Sandwich Maker. ([url]https://panini.io[/url])
libsandwich (0.1)  		Wrapper for libsandwich
sandwich-python (3.13.2) 	Python module for crossway multi-aspect production patterns.
yasm (1.4) 			Yet Another Sandwich Maker
pysw (1) 			makes yto samwitch!!!
sandwichmaker (2.0) 		easy to use sandwich maker

...

Vivian Darkbloom posted:

I'm finding how hard it is to make a cross-platform console interface that isn't horrible for a little strategy game project, so it's time to get a basic GUI framework set up. I just read the Python Wiki page on GUI programming and I am looking at pyjs as a possibility, since targeting web browsers might be easier than rolling my own GUI entirely. Not sure, though -- ideally I'd like something that's programmer-friendly and doesn't require a lot of setup for a user running my program. Any recommendations on what might be the best GUI choice?

accipter posted:

What are you envisioning in terms of the GUI?

Vivian Darkbloom posted:

A central canvas, a few lines of text underneath, and a few buttons horizontally arranged beneath the text. The canvas would need to be updated to recolor different areas on the board. Don't really need the ability to read clicks from the canvas or interact with any fancy widgets -- just whipping up a demo.

e: I am thinking tkinter will be ok for what I need.

ynohtna posted:

Here's my common snippet to convert an iterable into natural sort order:

Python code:

import re
def sorted_nicely(iter):
    """ Sort the given iterable in the way that humans expect."""
    convert = lambda text: int(text) if text.isdigit() else text
    alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ]
    return sorted(iter, key=alphanum_key)

Usage is:

Python code:

    for p in sorted_nicely(seen_paths):
        print(p)

QuarkJets posted:

Can tkinter handle those canvas operations you described? That sounds more like something you'd use Qt for (which is multi-platform but would create a pyqt dependency)

Vivian Darkbloom posted:

Confused about an issue with passing functions. This might be related to tkinter weirdness, but I might just be doing something else wrong.

code:

import tkinter as tk
root = tk.Tk()
for i in range(10):
    tk.Button(root,text=i,command=lambda:print(i)).pack(side=tk.LEFT)
root.mainloop()

When I click any button it prints 9, even though each button was initialized with its own print function. Why is it behaving like this, and not instead printing the appropriate number for each button?

Tigren posted:

That's an actual marketable skill with virtually infinite learning resources vs some weird, arcane language on top of a language that produces interfaces from 1995.

huhu posted:

I've just finished my first Flask project for my new job which is basically a page with a form that allows users to do CRUD operations on a database table. It took me about a week to create. Of course there was a bunch of stuff like having to learn MySQL, JavaScript form validation, navigate their code base, etc. along the way so it took a bit of time. I was just able to recreate what took me about a week with Flask Admin in an hour and was even able to add additionally functionality like a list view, searching, and filtering. I feel like this was deceptively simple though and when I start working on more complex projects I'd be better off writing the SQL and JS myself. Is that correct?

QuarkJets posted:

These descriptions both describe Flask

ynohtna posted:

Python's closures are late binding: the values of variables used within them are looked up at the time the inner function/closure is called.

It feels a bit of a hack, but I've seen a default argument used as a common workaround (the default value is bound at the time of definition), i.e.:

Python code:

for i in range(10):
    tk.Button(root, text=i, command=lambda i=i: print(i)).pack(side=tk.LEFT)

functools.partial is probably the proper way to do it, but may reduce readability in some cases.