|
Thanks. I've got one last question for today. I use Flask to expose a web interface and run methods on my backend thing. At the moment flask instantiates the backend, and answers requests from the interface by calling backend methods and returning the result. I'm not sure it's good design but it seems to works ok. Can the backend run http requests on the flask instance too, as a way to send status updates to be consumed from the web side ? At the moment I'd like to show a progress bar while a request is processed, which can take up to a few minutes. (I've never done web stuff and it shows )
|
# ? May 19, 2018 22:55 |
|
|
# ? Jun 3, 2024 23:58 |
|
Insert-web-framework-here is just a function. All the views and decorators and middleware and templates...all of it boils down to The Function. From the standpoint of you, the developer, there is no server. Nothing running in a loop waiting to handle web requests. There is just The Function. You can do whatever you want in The Function. The only things required of this function is that it takes in the data from the request (in python the format of this data is a standard for frameworks called WSGI) and that it returns a string that can be interpreted by clients as a valid response. By itself, Flask or Django or whatever can only handle one request/response at a time. However, in production and in development, the server you are running is not a Flask or Django server. It is another multithreaded or async server that can handle multiple requests and responses at a time, and that is what allows you to make calls "to yourself" from within The Function. If you make a request from inside The Function to http://127.0.0.1:8000/some/path/here/, your web server (nginx or runserver or run or gunicorn or blah blah blah) will call The Function again and now The Function is running in two different threads. One thread is waiting on the second thread to give a response. All that being said, what is usually done is that you just do whatever from right inside your view rather than calling another url. For example, if you have an endpoint for POSTing new Foobars which creates a row in a database table and as part of another endpoint Foobars get created, it's lower overhead to just do the database calls from inside that second view rather than POSTing to your dedicated view for creating Foobars.
|
# ? May 19, 2018 23:55 |
|
Sad Panda posted:Very. I'll be logging in to each site and there's not going to be much standardisation. I do exactly this with a different, easier, non-python solution. For what industry and purpose are you doing this project?
|
# ? May 20, 2018 14:00 |
|
Thermopyle posted:Kenneth Reitz did a talk on Pipenv that might be useful for anyone wondering why they should use Pipenv. Thanks, this was neat and I'll give it a shot.
|
# ? May 20, 2018 17:38 |
|
CarForumPoster posted:I do exactly this with a different, easier, non-python solution. For what industry and purpose are you doing this project? It's personal use, no form of industry. I'm logging into betting websites. What's the easier option? I considered a Pyautogui setup which is just recording a series of clicks and using OCR + Image Recognition, but Selenium seemed like an interesting way to start. I managed to get the login working on 145/155 websites. The other ones are throwing up Selenium errors. A few of them because elements are tagged as invisble so Selenium doesn't want to interact with them. Python code:
|
# ? May 20, 2018 18:20 |
|
Sad Panda posted:It's personal use, no form of industry. I'm logging into betting websites. What's the easier option? I considered a Pyautogui setup which is just recording a series of clicks and using OCR + Image Recognition, but Selenium seemed like an interesting way to start. Some learning curve to this, decent documentation, and $150/month but many good features. It was much mroe robust and allowed faster deployment than trying anything with BeautifulSoup and the like. https://www.parsehub.com/ EDIT: Its free to use with a limited feature set and what not but I ended up needing several of their not free features. E.G. I dont want my projects to become public. CarForumPoster fucked around with this message at 18:36 on May 20, 2018 |
# ? May 20, 2018 18:28 |
|
Thanks for the write up Thermopyle.
|
# ? May 20, 2018 20:29 |
|
Anyone experienced with Selenium? I'm trying to perform a hover, but it doesn't seem to work. https://www.galabingo.com/ That's the site in question. The login button does not seem to want to be clicked on, and the username + password boxes only appear when that is hovered over. Maybe this can be achieved by Javascript but I'm lost on that too. Python code:
|
# ? May 20, 2018 22:23 |
|
I didn't try it because I don't have Selenium up to date at home, but from looking at the page you probably want to hover the div.fn-show-login-tooltip element instead.
|
# ? May 20, 2018 23:24 |
|
breaks posted:I didn't try it because I don't have Selenium up to date at home, but from looking at the page you probably want to hover the div.fn-show-login-tooltip element instead. Thanks! That worked perfectly. As someone said to me.. "Selenium is probably not too bad if using it for what its intended use is... (automated testing of websites you are devving), but for automating sites we dont own it does become trickier to use." 149/155 sites isn't bad. 2 with invisible elements that I can't interact with. 2 more with 'cannot focus element' and 2 more that don't like being opened in Chrome (throws a CAPTCHA) but Firefox works just fine and spoofing the User-Agent doesn't work so I'm not sure what it's using to detect that.
|
# ? May 20, 2018 23:39 |
|
Linear Zoetrope posted:In numpy (or some interoperable library like pandas) is there a way to create basic shapes like triangles or circles in an ndarray/tensor/table/whatever? I'm featurizing a space of entities represented like {pos = (x,y) , shape = Triangle{base_len = 1.5}, ...} or whatever and while I'm entirely capable of writing this myself* it would be nice to be able to just have python do it. I'm not really sure I understand what you're asking. My first thought for a grid of triangles is you're looking for a mesh but I also have no idea what "C channels" or "feed it into a NN" means so .
|
# ? May 21, 2018 07:30 |
|
Boris Galerkin posted:"feed it into a NN" Neural network maybe ?
|
# ? May 21, 2018 09:47 |
|
Linear Zoetrope posted:In numpy (or some interoperable library like pandas) is there a way to create basic shapes like triangles or circles in an ndarray/tensor/table/whatever? I'm featurizing a space of entities represented like {pos = (x,y) , shape = Triangle{base_len = 1.5}, ...} or whatever and while I'm entirely capable of writing this myself* it would be nice to be able to just have python do it. Numpy isn't an image library so it doesn't have functions like this. Your best bet I think is to use Pillow to draw a greyscale image for each channel and then convert those to numpy arrays and stack them. Pillow has a polygon function that should suit your needs, and you can convert a Pillow image to a numpy array with np.asarray(my_image).
|
# ? May 21, 2018 10:07 |
|
Thanks, I actually just found skimage.draw which provides almost exactly what I need. It's nice because it generates indices so I can do things likecode:
|
# ? May 21, 2018 11:43 |
|
Sad Panda posted:Anyone experienced with Selenium? It's been a while since I've used this stuff for work, but if you want to save some hassle, check out https://splinter.readthedocs.io/en/latest/ . It's a bit simpler and the documentation is good. Makes for cleaner looking source, too.
|
# ? May 23, 2018 23:05 |
|
I've got an array of 3 strings, that are only different by a single character (the child number).Python code:
Python code:
Feral Integral posted:It's been a while since I've used this stuff for work, but if you want to save some hassle, check out https://splinter.readthedocs.io/en/latest/ . It's a bit simpler and the documentation is good. Makes for cleaner looking source, too. Thanks for the suggestion. Selenium seems to be going OK at the moment. Does Splinter click on elements that are not visible? That's one of the things that frustrates me with Selenium. I've got to insert some Javascript, to aria-hidden to false, or some other things. Sad Panda fucked around with this message at 00:22 on May 24, 2018 |
# ? May 24, 2018 00:14 |
|
I haven't tried it, but on the next project where I need Selenium, I'm going to look into using pyppeteer instead.Sad Panda posted:I've got an array of 3 strings, that are only different by a single character (the child number). Using f-strings in py3.6. Or you can use a format string in lower python versions. Python code:
|
# ? May 24, 2018 00:36 |
|
Just do f"shared.string.{i}" edit almost
|
# ? May 24, 2018 00:37 |
|
On the off chance you'd rather have the string declared elsewhere in your code rather than in the for-loop, or are using an older version of Python, you can also do this:Python code:
|
# ? May 24, 2018 01:02 |
|
Sad Panda posted:Thanks for the suggestion. Selenium seems to be going OK at the moment. Does Splinter click on elements that are not visible? That's one of the things that frustrates me with Selenium. I've got to insert some Javascript, to aria-hidden to false, or some other things. Selenium/Webdriver's purpose is to emulate interacting with the browser as a user would, so you shouldn't need to click an element that isn't visible, because the user would actually be interacting with some other element instead of the hidden one. YMMV for sites that are trying to deliberately break browser automation or otherwise doing something insane, but most of the time in this situation, you've either got the wrong element, or you've got the right element but aren't waiting for it to get into an interactable state. If you want to provide some more information about the error, site, or whatever else is relevant, I'll be glad to have a look and see if anything pops out. A fair part of my job is dealing with this poo poo. breaks fucked around with this message at 07:20 on May 24, 2018 |
# ? May 24, 2018 06:26 |
|
Of course. f-strings. Thanks for that! I love those things dearly so can't think why I didn't consider them. I'll say it's cos it was past midnight...breaks posted:Selenium/Webdriver's purpose is to emulate interacting with the browser as a user would, so you shouldn't need to click an element that isn't visible, because the user would actually be interacting with some other element instead of the hidden one. I think it's people trying to deliberately break browser automation. My current project is basically writing something to work out how much money I have across all my accounts. Those accounts include bank accounts, but also a bunch of betting websites (I'm in the UK where it's nice and legal). Three examples of sites that refuse to work (claiming not visible/cannot focus) without some form of skullduggery include 1) https://spinson.com Here I can click on the login button as needed, but then if I try to interact with the username form it throws an error. The following gets around it through. I've no idea why using that Javascript, which uncovers a 3rd input box on the login field works, but it does. Python code:
The same issue. I can click on login, but then can't interact with the username field. I've tried HTML code:
HTML code:
3) Nationwide Online Banking. (https://onlinebanking.nationwide.co.uk/AccessManagement/Login). If you type in a random 10 digit customer number, click on Log in using memorable data, type a random thing into memorable data then the passnumber dropdowns become interactable. However, the dropdown values are hidden with aria-hidden = true JavaScript code:
Sad Panda fucked around with this message at 10:36 on May 24, 2018 |
# ? May 24, 2018 10:28 |
|
YMMV but I've always found it easier to reverse engineer the backend API by checking out the AJAX calls in Chrome Developer Tools. Extra bonus is that you sometimes uncover extra internal data that they don't bother to expose in the front-end.
|
# ? May 24, 2018 13:36 |
|
I'm having trouble getting matplotlib to overlay two plots on top of each other instead of creating two different plots and I'm not sure what I'm doing wrong in my example when I'm googling the issue:code:
bamhand fucked around with this message at 16:54 on May 24, 2018 |
# ? May 24, 2018 15:48 |
|
Is "a" an Axes object?
|
# ? May 24, 2018 16:31 |
|
I think it's a dataframe? I'm modifying some old code where I'm just changing the source of the data but trying to keep the plots the same. This is what was working before (also I'm pretty new to python in general):code:
a = self.sas.sd2df("reg_out", "work").loc[:,["date","heloc2","pred"]] a=a.loc[a["heloc2"].notnull(),] bamhand fucked around with this message at 16:54 on May 24, 2018 |
# ? May 24, 2018 16:37 |
|
bamhand posted:I think it's a dataframe? I'm modifying some old code where I'm just changing the source of the data but trying to keep the plots the same. This is what was working before (also I'm pretty new to python in general): Use the code tags. It makes it infinitely easier to read your code. code:
|
# ? May 24, 2018 16:52 |
|
bamhand posted:I think it's a dataframe? I'm modifying some old code where I'm just changing the source of the data but trying to keep the plots the same. This is what was working before (also I'm pretty new to python in general): axes = pt.axes() df_1.plot(..., ax=ax) df_2.plot(..., ax=ax) and both calls will show up in the same axes. Better: use seaborn instead. If you want to plot predicted vs. actual, consider using seaborn.jointplot.
|
# ? May 24, 2018 17:13 |
|
Awesome, thank you.
|
# ? May 24, 2018 18:08 |
|
Cingulate posted:pd.DataFrame's plot method takes an ax argument. I.e., do Yep. If they are DataFrames this is the right fix. I generally do f, ax = plt.subplots(1, 1), so I go ahead and get the figure object too, since I make a couple of tweaks and normally save to file instead of an interactive view.
|
# ? May 24, 2018 18:23 |
|
New question, if I'm working in Jupyter Notebook and I make a change to one of my imported files, what do I need to do to get the notebook to acknowledge that? Example: I have some program prog.py and I import prog2.py into it. In Spyder I can update prog2.py, hit, save, and then when I run prog.py it uses the new version. This doesn't appear to be the case in Jupyter Notebook.
|
# ? May 24, 2018 19:34 |
|
There are ways to reload, but the easiest is just to restart the kernel. Kernel > Restart / Restart and run all.
|
# ? May 24, 2018 19:38 |
|
SurgicalOntologist posted:There are ways to reload, but the easiest is just to restart the kernel. Kernel > Restart / Restart and run all. 00 (two zeros) is the default shortcut to do that.
|
# ? May 24, 2018 19:41 |
|
Is there a way to restart the kernel and re-run everything up to this point?
|
# ? May 27, 2018 03:23 |
|
KernelSlanders posted:Is there a way to restart the kernel and re-run everything up to this point? Do you mean like rerun all cells up to the current one selected (vs rerunning all cells period)? I’d like to know that too. What I do now is just insert a cell underneath where I’d like to stop and just type asdf so it fails and execution stops
|
# ? May 27, 2018 08:19 |
|
Boris Galerkin posted:Do you mean like rerun all cells up to the current one selected (vs rerunning all cells period)? I’d like to know that too. What I do now is just insert a cell underneath where I’d like to stop and just type asdf so it fails and execution stops
|
# ? May 27, 2018 08:31 |
|
Isn’t that an option in one of the drop down menus of the notebook? I believe it’s in the menu that you restart kernel. It was added in a recent-ish update I believe.
|
# ? May 27, 2018 13:39 |
I vaguely remember something about cells having “run above” option, but I’m not a frequent Jupyter user.
|
|
# ? May 27, 2018 15:56 |
|
Yeah, in the notebook there's a "run above." That doesn't help in the console though.
|
# ? May 27, 2018 17:19 |
|
I didn't realize you meant in the console... that's a fantastic idea, if it existed I would use it all the time. That's worth writing a %magic command for or at least a feature proposal to the developers.
|
# ? May 27, 2018 18:09 |
|
|
# ? Jun 3, 2024 23:58 |
|
Oh oops. Missed the console bit.
|
# ? May 27, 2018 20:31 |