Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
CarForumPoster
Jun 26, 2013

⚡POWER⚡

i vomit kittens posted:

It's just so confusing to me because of how variable it is. It will work fine for 100 questions, maybe even 300, and then just abruptly fail on one. It's also not failing trying to use an element after the button click, it's the button click itself that is throwing the error.

code:
    for x in range (0, 250):
         paragraphs = wait.until(EC.visibility_of_all_elements_located((By.ID, "para")))
        ---logic for determining which paragraphs are part of the question, answers, or neither and writing them to the appropriate place---
        answerRadio = wait.until(EC.element_to_be_clickable((By.ID, "SelectedAnswerId")))
        driver.execute_script("arguments[0].checked = true;", answerRadio)
        answerRadio.click()
        //the click is necessary because the "Show Answer" button does not become available until the focus changes.
        showAnswer = wait.until(EC.element_to_be_clickable((By.ID, "aShowAnswerCQTab")))
        showAnswer.click() *****
        time.sleep(2)
        paragraphs = wait.until(EC.presence_of_all_elements_located((By.TAG_NAME, "p")))
        //The paragraphs on the answer page do not have the class name "para" like on the question page.
        ---logic for determining which paragraph is the one that says the correct answer and writing it to the file.---
        nextQuestion = wait.until(EC.element_to_be_clickable((By.ID, "aSubmitCQTab")))
        nextQuestion.click() *****
The lines where I put ***** are where the stack traces point to the problem being. Like I said, the click does actually go through, because the page will change to either show the answer or go to the next question, but the script throws an error immediately after the fact saying that it was unable to find the button that allows it to do what it just did.

I'm not great at selenium but I've made ~25ish selenium based scrapers by now. I find that adding "padding waits" solves a lot of my waiting for poo poo to show up problems. So I might wait for something to be clickable then wait an extra half second to make sure. Usually I'm scraping poo poo in the middle of the night to be used when someone gets in a few hours later so time isn't critical.

Adbot
ADBOT LOVES YOU

breaks
May 12, 2001

i vomit kittens posted:

The lines where I put ***** are where the stack traces point to the problem being. Like I said, the click does actually go through, because the page will change to either show the answer or go to the next question, but the script throws an error immediately after the fact saying that it was unable to find the button that allows it to do what it just did.

That you're getting errors around the sleep, or that it's there at all, suggests that you're not waiting on the right thing(s). You should never wait X time, you should wait on some condition. Do whatever you need to do to get rid of the sleep and that will probably help solve this problem. Without runnable code that repros the issue, there's no way to give you anything other than a vague guess. Whatever you're automating could be doing anything; you haven't pasted whatever specific exceptions you're getting; even if you did it would be of limited help without any knowledge of whatever you're automating. Sorry not trying to sound like a dick, but what you're describing sounds to me like general UI automation hassles which are very particular to the app being automated.

breaks fucked around with this message at 04:58 on Jan 15, 2020

i vomit kittens
Apr 25, 2019


The sleep is left over from before I added the wait.until() functions. I'm not sure if it's actually necessary anymore. The wait.until() should satisfy that need of waiting for a condition since it's looking for those buttons to be clickable, no? Also, while I may not have used the exact wording of the exception, I did say that Selenium is telling me it can't find the element.

code:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"aShowAnswerCQTab"}
I can't really just paste a link to the website here and tell you to go run it. I'm collecting these questions from a practice test for a state licensing exam in order to analyze them as part of a research project. Access to the site is either hundreds of dollars or only available to recent graduates/faculty of our program.

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!
What's the correct way of setting an empty string for a default value with dataclasses?

code:
@dataclasses.dataclass
class Foo:
    text: str = str()
code:
@dataclasses.dataclass
class Foo:
    text: str = dataclasses.field(default_factory=str)

cinci zoo sniper
Mar 15, 2013




Boris Galerkin posted:

What's the correct way of setting an empty string for a default value with dataclasses?

code:
@dataclasses.dataclass
class Foo:
    text: str = str()
code:
@dataclasses.dataclass
class Foo:
    text: str = dataclasses.field(default_factory=str)

Both will work, but I think that factory is preferred style for when your default is an output of a 0-argument function.

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!
Yeah I see them use it for things like a default empty list so wasn't sure why it didn't extend to other things like str or int. Also is the Optional typing class ever really needed? I don't understand the difference between these two:

code:
...
something: typing.Optional[str] = None
...
code:
...
something: str = None
...

Mr. Angry
Jan 20, 2012

Boris Galerkin posted:

Yeah I see them use it for things like a default empty list so wasn't sure why it didn't extend to other things like str or int. Also is the Optional typing class ever really needed? I don't understand the difference between these two:
str and int are immutable and are always passed by value so you can simply set a default value like "" and it'll work. Lists, dicts and other objects are always passed by reference and setting a default value without a factory function will result in every instance having a reference to the same object. It's for the same reason you don't use empty lists or other objects in function arguments.

PEP 484 used to allow both of your examples to work but now the Optional type must be explicit and type checkers should raise an error if you don't include it.

cinci zoo sniper
Mar 15, 2013




Boris Galerkin posted:

Yeah I see them use it for things like a default empty list so wasn't sure why it didn't extend to other things like str or int. Also is the Optional typing class ever really needed? I don't understand the difference between these two:

code:
...
something: typing.Optional[str] = None
...
code:
...
something: str = None
...

Optional[str] is shorthand for Union[str, NoneType], meaning that the object is string or None. The latter example thus should be flagged by any type checker as incorrect.

Data Graham
Dec 28, 2009

📈📊🍪😋



Mr. Angry posted:

str and int are immutable and are always passed by value so you can simply set a default value like "" and it'll work. Lists, dicts and other objects are always passed by reference and setting a default value without a factory function will result in every instance having a reference to the same object. It's for the same reason you don't use empty lists or other objects in function arguments.

PEP 484 used to allow both of your examples to work but now the Optional type must be explicit and type checkers should raise an error if you don't include it.

Waiwaiwait. I feel like this means I might have been doing poo poo wrong for years then.

Does this mean

code:
def foo(blah, my_list=[]):
Or

code:
foo(blah, [])
are wrong? What do I need to be doing (and fixing in ... like .. all of my code ever)?

cinci zoo sniper
Mar 15, 2013




Data Graham posted:

Waiwaiwait. I feel like this means I might have been doing poo poo wrong for years then.

Does this mean

code:
def foo(blah, my_list=[]):
Or

code:
foo(blah, [])
are wrong? What do I need to be doing (and fixing in ... like .. all of my code ever)?

Yes, you might want to avoid that.

As in, it works, but so long as you remember to specify my_list each time a new, e.g. to only ever invoke foo with foo(blah, []).

cinci zoo sniper fucked around with this message at 14:52 on Jan 15, 2020

mbt
Aug 13, 2012

Data Graham posted:

Waiwaiwait. I feel like this means I might have been doing poo poo wrong for years then.

Does this mean

code:
def foo(blah, my_list=[]):
Or

code:
foo(blah, [])
are wrong? What do I need to be doing (and fixing in ... like .. all of my code ever)?
:staredog:

Create a new array within the function not passed to the function is easiest. It's just one of those things you hardcode into your brain

E: or yeah remember to always pass an empty array, which iirc clears the old one with your new one. Now I'm curious what it looks like on the back end

Data Graham
Dec 28, 2009

📈📊🍪😋



Okay, well as long as passing in an empty array to a new function invocation is kosher, that's not as bad as I thought.

A lot of times I've put the my_list=[] in the function definition just so there would always be something to iterate over within the function instead of throwing None errors if I didn't specify the kwarg in the function call (usually this was the result of me tacking on new functionality to legacy code where there are a lot of calls to the function and I want to seamlessly add handling for an optional param without having to refactor every single call).

I could have just done "if my_list:" but hey

Boris Galerkin
Dec 17, 2011

I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Data Graham posted:

Waiwaiwait. I feel like this means I might have been doing poo poo wrong for years then.

Does this mean

code:
def foo(blah, my_list=[]):
Or

code:
foo(blah, [])
are wrong? What do I need to be doing (and fixing in ... like .. all of my code ever)?

All you'd need to do is

code:
def foo(blah, my_list=None):
    my_list = my_list or list()
    ...

cinci zoo sniper
Mar 15, 2013




Data Graham posted:

Okay, well as long as passing in an empty array to a new function invocation is kosher, that's not as bad as I thought.

A lot of times I've put the my_list=[] in the function definition just so there would always be something to iterate over within the function instead of throwing None errors if I didn't specify the kwarg in the function call (usually this was the result of me tacking on new functionality to legacy code where there are a lot of calls to the function and I want to seamlessly add handling for an optional param without having to refactor every single call).

It is a functioning approach, but that does not make it anywhere near kosher. It’s a really bad code smell, and you should do what Myers suggests. Second part of your post I thankfully don’t understand fully as it sounds horrifying.

Data Graham
Dec 28, 2009

📈📊🍪😋



Like for example I want to take

code:
def foo(blah):
    return True
And turn it into

code:
def foo(blah, my_list=[]):
    for item in my_list:
        do_stuff()
    return True
Where the code has a ton of existing calls to foo(blah). So if I did my_list=[], it would work, whereas my_list=None would throw TypeErrors.

I guess it felt cleaner and saved a line of code to not have to check whether my_list was truthy, but what you guys are saying makes sense. I like the "my_list = my_list or list()" style.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

I used to sometimes see mutable default arguments used for memoization.

Python code:
def memoized(value, stuff={}):
    try:
        return stuff[value]
    except KeyError:
        blah = expensive_stuff(value)
        stuff[value] = blah
        return blah
Don't see that much anymore. People will use a custom class or lru_cache more often nowadays.

cinci zoo sniper
Mar 15, 2013




Thermopyle posted:

I used to sometimes see mutable default arguments used for memoization.

Python code:
def memoized(value, stuff={}):
    try:
        return stuff[value]
    except KeyError:
        blah = expensive_stuff(value)
        stuff[value] = blah
        return blah
Don't see that much anymore. People will use a custom class or lru_cache more often nowadays.

Yeah this is my approach for problems like going through a few gigabytes of XML documents where 1 or 2 samples may be broken.

SurgicalOntologist
Jun 17, 2004

Data Graham posted:

Like for example I want to take

code:
def foo(blah):
    return True
And turn it into

code:
def foo(blah, my_list=[]):
    for item in my_list:
        do_stuff()
    return True
Where the code has a ton of existing calls to foo(blah). So if I did my_list=[], it would work, whereas my_list=None would throw TypeErrors.

I guess it felt cleaner and saved a line of code to not have to check whether my_list was truthy, but what you guys are saying makes sense. I like the "my_list = my_list or list()" style.

One way to avoid the if statement is to make the default value an empty tuple. Since tuples are immutable, it doesn't have the same problem.

But in general you shouldn't hesitate to write if arg is None: often.

Data Graham
Dec 28, 2009

📈📊🍪😋



SurgicalOntologist posted:

One way to avoid the if statement is to make the default value an empty tuple. Since tuples are immutable, it doesn't have the same problem.

But in general you shouldn't hesitate to write if arg is None: often.

Yeah, good call. I like the previous suggestion a little more because it also saves an indent, but I'll bear that in mind.

The tuple thing sure feels like inside baseball though. One of the joys of python is how much of it Just Works the way you'd expect it to on the surface, and I guess I'd tricked myself into thinking passing-by-value-or-reference was one of those things one didn't have to think about in day-to-day code, since that pattern I was using didn't seem to cause any issues.

Live and learn. That'll make a drat fine interview question though

mbt
Aug 13, 2012

Data Graham posted:

Yeah, good call. I like the previous suggestion a little more because it also saves an indent, but I'll bear that in mind.

The tuple thing sure feels like inside baseball though. One of the joys of python is how much of it Just Works the way you'd expect it to on the surface, and I guess I'd tricked myself into thinking passing-by-value-or-reference was one of those things one didn't have to think about in day-to-day code, since that pattern I was using didn't seem to cause any issues.

Live and learn. That'll make a drat fine interview question though

Mutability of python data structures is extremely fundamental to the language. Learning what can and cant be modified as well as the difference between copy / deepcopy is very important.

And yes it is a great interview question lol

E: please dont take this as a dig at your abilities, I dont intend to admonish anyone for not knowing it

Data Graham
Dec 28, 2009

📈📊🍪😋



Oh sure, I get it. You can imagine how it feels learning fundamentals after umpty-ump years doing this though :v:

But having thought about this a little, yeah, the classical form the interview question would take would be like "What's the difference between a list and a tuple". But with this spin it would become more like:

Suppose you're writing a function that takes a list as a parameter. Why would it be OK to write def foo(my_list=()): and not OK to write def foo(my_list=[]):?

The answer I would LIKE to see would be "It wouldn't, because that's weird and obscures readability and tricks a coder into thinking the function is supposed to take a tuple instead of a list"

But "it's immutable and therefore will allow you to iterate over my_list within the function without worrying about whether it's getting passed by reference" definitely demonstrates more internalized understanding than I had an hour ago


e: To be clear I knew about mutability, just not this aspect of it. You know, definition vs. practice stuff.

Data Graham fucked around with this message at 17:30 on Jan 15, 2020

SurgicalOntologist
Jun 17, 2004

A stupid detail but you have to write empty tuples like (,)

Technically it's the comma that defines a tuple not the parentheses. The parentheses are just needed for grouping in most places (but not all).

SurgicalOntologist
Jun 17, 2004

Edit: (haha a reverse quote != edit)

Data Graham posted:


The answer I would LIKE to see would be "It wouldn't, because that's weird and obscures readability and tricks a coder into thinking the function is supposed to take a tuple instead of a list"

I see where you're coming from, but I would disagree. In this (quite common) case, the function actually expects any iterable, there is no reason to prefer a list. If it actually needs a list because it uses a list method or something, then this approach won't work anyway.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

SurgicalOntologist posted:

A stupid detail but you have to write empty tuples like (,)

Technically it's the comma that defines a tuple not the parentheses. The parentheses are just needed for grouping in most places (but not all).

Python code:
type(())
<class 'tuple'>
type((3))
<class 'int'>
type((3,))
<class 'tuple'>

SurgicalOntologist
Jun 17, 2004

Huh I wonder where I got that from.

\/ yep that must be it

SurgicalOntologist fucked around with this message at 18:16 on Jan 15, 2020

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

SurgicalOntologist posted:

Huh I wonder where I got that from.

It's ok, it was my first thought as well.

I think the single-element tuple requiring the comma that first led me to have that instinct.

Data Graham
Dec 28, 2009

📈📊🍪😋



I’m in the habit (usually) of putting trailing commas in everything, but especially tuples because wacky breakage happens in the case of length = 1 if you don’t.

I like trailing commas in dicts and lists too because it makes adding/removing elements safer when they’re all on individual lines.

E: ^^ yeah that

mbt
Aug 13, 2012

Data Graham posted:

Suppose you're writing a function that takes a list as a parameter. Why would it be OK to write def foo(my_list=()): and not OK to write def foo(my_list=[]):?

The answer I would LIKE to see would be "It wouldn't, because that's weird and obscures readability and tricks a coder into thinking the function is supposed to take a tuple instead of a list"

But "it's immutable and therefore will allow you to iterate over my_list within the function without worrying about whether it's getting passed by reference" definitely demonstrates more internalized understanding than I had an hour ago

Would it make sense if I told you everything in python is passed by reference? Therefore no matter what you need to find some way around the mutability. Passing a tuple in the definition works, doing x = list(mylist) works, there are a lot of ways around it, but everything is always passed by reference

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Data Graham posted:

I like trailing commas in dicts and lists too because it makes adding/removing elements safer when they’re all on individual lines.

Plus it makes diff's better.

duck monster
Dec 15, 2004

M. Night Skymall posted:

Look, we can't all be the financial industry and run our critical services on *checks notes* giant excel spreadsheets.

When I was doing stuff with the CSIRO, all the science dudes would basically use a combination of Python and giant hosed up excel spreadsheets with , like, half a bajillion rows all generated by some messed up visual basic instrument API.

Somehow it all actualy worked

pmchem
Jan 22, 2010


duck monster posted:

When I was doing stuff with the CSIRO, all the science dudes would basically use a combination of Python and giant hosed up excel spreadsheets with , like, half a bajillion rows all generated by some messed up visual basic instrument API.

Somehow it all actualy worked

can confirm this is often how science is done

teen phone cutie
Jun 18, 2012

last year i rewrote something awful from scratch because i hate myself
Crossposting from the web development thread:

Spent my morning reading some of the OAuth 2 RFC.

Can anyone recommend me tutorials that walk you through implementing an authentication server from start to finish, preferably in Python?

OnceIWasAnOstrich
Jul 22, 2006

pmchem posted:

can confirm this is often how science is done

Had a collaboration where we had a pipeline that generated a ton of numerical data tables and were trying to standardize storage onto some Apache Arrow based format, either Feather or Parquet. This was vetoed by some PIs because they wanted "something standard that we can open up in Excel when you all leave". Massive CSV files it is.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Grump posted:

Crossposting from the web development thread:

Spent my morning reading some of the OAuth 2 RFC.

Can anyone recommend me tutorials that walk you through implementing an authentication server from start to finish, preferably in Python?

HAHAHAHAHHAHAH

A few years ago I did this exact thing and man it's irritating. I was never able to find a good single source of information. I ended up having a dozen tabs that I'd reference while trying to figure out why all of them contradicted each of them in some small way.

Unfortunately I didn't save any of those.

In the end I got it working.

One thing I did that helped was keep the source of one of the django oauth2 packages handy so I could reference how they handled things.

duck monster
Dec 15, 2004

Grump posted:

Crossposting from the web development thread:

Spent my morning reading some of the OAuth 2 RFC.

Can anyone recommend me tutorials that walk you through implementing an authentication server from start to finish, preferably in Python?

It usually starts with consuming an entire quart of the cheapest mass produced suck-and-chuck whiskey you can find, and it all goes down hill from there. The end result is you wake up in the morning, your wife has taken the kids to her mothers, everyone hates you on facebook now and you cant remember why, and running on your laptop is a functional OAUTH2 server.

teen phone cutie
Jun 18, 2012

last year i rewrote something awful from scratch because i hate myself
took a step back from the OAuth stuff and am trying to go down a couple levels. This weekend I'm going to try and immerse myself in advanced Python usage and the Flask docs. I think my tentative plan here is to create a simple Flask API that scrapes a webpage and returns JSON at different endpoints.

Quick question about Flask configuration variables: what's the most painless way possible to work with env variables with easy switching between production/dev vars? The Flask docs recommend a few ways, but what's most common? Also, what should/shouldn't be committed to Git? Secret key and DB Login seems obvious, but is everything else fair game?

DearSirXNORMadam
Aug 1, 2009
Quick question on python multiprocessing module:

What is the most pythonic way to take a large array, and have an object perform an operation on each element concurrently?

Eg, I have some statistical model represented by an object, and it contains numpy arrays representing means, covariances, etc. I need it to estimate the likelihood of a bunch of array elements. When I just did pool.map_async over the array, I would occasionally get hangs, which I suspect were deadlocking because two processes were trying to read the object properties or something, but I don't have a clear idea of how the pool map method works under the hood, so maybe that's not the case?

vikingstrike
Sep 23, 2007

whats happening, captain

Mirconium posted:

Quick question on python multiprocessing module:

What is the most pythonic way to take a large array, and have an object perform an operation on each element concurrently?

Eg, I have some statistical model represented by an object, and it contains numpy arrays representing means, covariances, etc. I need it to estimate the likelihood of a bunch of array elements. When I just did pool.map_async over the array, I would occasionally get hangs, which I suspect were deadlocking because two processes were trying to read the object properties or something, but I don't have a clear idea of how the pool map method works under the hood, so maybe that's not the case?

Could use concurrent.futures and the Process Pool Executor, then just executor.map(fn_name, array) maybe? Depending on how long each process is, may not be worth the overheard, just depends.

https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor

Might also be able to use numpy too:

https://stackoverflow.com/questions/35215161/most-efficient-way-to-map-function-over-numpy-array/35216364

QuarkJets
Sep 8, 2008

Mirconium posted:

Quick question on python multiprocessing module:

What is the most pythonic way to take a large array, and have an object perform an operation on each element concurrently?

Eg, I have some statistical model represented by an object, and it contains numpy arrays representing means, covariances, etc. I need it to estimate the likelihood of a bunch of array elements. When I just did pool.map_async over the array, I would occasionally get hangs, which I suspect were deadlocking because two processes were trying to read the object properties or something, but I don't have a clear idea of how the pool map method works under the hood, so maybe that's not the case?

there are a number of possible sources for hangs, but multiprocessing copies the relevant objects to each process, so it definitely isn't an issue with mutexes under the hood or whatever

the likeliest option is that some array elements actually just take longer to work with, for whatever reason. Have each process self-report its computation time and then try to perform the computation serially on those elements + their neighboring elements to see for yourself whether this is the case, if you're curious

another option is that your resources are overtasked. Maybe you're asking for too many workers, or if your processes require significant IO then maybe one process is getting temporarily locked out of IO for whatever reason

But the most pythonic way is to just use multiprocessing.map or map_async, in any case

Adbot
ADBOT LOVES YOU

DearSirXNORMadam
Aug 1, 2009

QuarkJets posted:

there are a number of possible sources for hangs, but multiprocessing copies the relevant objects to each process, so it definitely isn't an issue with mutexes under the hood or whatever

the likeliest option is that some array elements actually just take longer to work with, for whatever reason. Have each process self-report its computation time and then try to perform the computation serially on those elements + their neighboring elements to see for yourself whether this is the case, if you're curious

another option is that your resources are overtasked. Maybe you're asking for too many workers, or if your processes require significant IO then maybe one process is getting temporarily locked out of IO for whatever reason

But the most pythonic way is to just use multiprocessing.map or map_async, in any case

Definitely not a long time on computations because when I used regular ol' map instead of map_async, it was taking less long than the hangs. Maybe IO issues, will investigate.

Actually, one more thing: so pool was throwing errors around using an object method instead of a function in map_async, so the way I was dealing with it was that I was passing the object to each worker via the repeat iterator from itertools. I'm starting to realize this may have actually been the bigger issue...

Edit:
So the pseudocode would be like
https://pastebin.com/JScMEukA

DearSirXNORMadam fucked around with this message at 01:14 on Jan 21, 2020

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply