|
Hughmoris posted:How is python+selenium for filling out lots of repetitive forms? I noticed that some people on my project team are manually entering in the data for 2000+ users in to a web portal. They've asked for help but my eyes will fall out of my head if I have to manually type in crap. It is basically automating someone sitting at the computer and doing all that stuff though, probably take a while. Is it possible to use something like Requests and just POST the form data that's being sent, without having to load their web pages?
|
#
?
Dec 5, 2017 03:53
|
|
|
|
| # ? May 16, 2024 03:45 |
|
|
baka kaba posted:It is basically automating someone sitting at the computer and doing all that stuff though, probably take a while. Is it possible to use something like Requests and just POST the form data that's being sent, without having to load their web pages? Hmmmmm... My web knowledge is pretty sparse. To see if this is feasible, should I try to record the network traffic while I submit a form and examine the parameters of the POST?
|
|
#
?
Dec 5, 2017 04:06
|
|
|
Hughmoris posted:How is python+selenium for filling out lots of repetitive forms? I noticed that some people on my project team are manually entering in the data for 2000+ users in to a web portal. They've asked for help but my eyes will fall out of my head if I have to manually type in crap. Do you think you could simplify this even more by just making a bunch of POST requests? See if you can spy on the transaction in Chrome Dev Tools. [Edit] Oh, another page...
|
|
#
?
Dec 5, 2017 04:23
|
|
|
Hughmoris posted:How is python+selenium for filling out lots of repetitive forms? I noticed that some people on my project team are manually entering in the data for 2000+ users in to a web portal. They've asked for help but my eyes will fall out of my head if I have to manually type in crap. Hi there, I'm an automation engineer. This is entirely my bread and butter. The solution to this depends what you're trying to solve. Is this a testing thing? Or is it literally just people doing manual data entry? If testing, are you testing that some service / webpage that takes data and crunches it? If that's the case, as people before have said, you're better off just issuing posts requests to the page, or if it's a service, interacting directly with the service. If you're testing some UI components, then selenium is the jam. If it's data entry, then yeah, see above minus the selenium suggestion.
|
|
#
?
Dec 5, 2017 05:10
|
|
|
Sockser posted:Hi there, I'm an automation engineer. This is entirely my bread and butter. Automation engineer sounds pretty cool. No testing involved in this, just plain 'ole data entry. With my skill set, using pure POST requests seems pretty risky. I'll likely play it safe and use selenium to navigate the page while I surf the web. Thanks for the ideas.
|
|
#
?
Dec 5, 2017 06:27
|
|
|
Hughmoris posted:Automation engineer sounds pretty cool. Honestly it's easy if it's possible - a lot easier than writing Selenium code (and all the edge case stuff to make sure the page is fully loaded, etc) Here's how you set up a Session so all your requests can use the same login deets automatically Here's how you POST some keys and values (spoiler: you just hand it a dictionary) Here's how you can check the response status and make sure it worked You can either look at the webpage to see what url the form is POSTing to, and all the key names for each of the inputs, or like someone said you can use Chrome's DevTools to spy on what the browser's POST request actually contains (just fill out your form, open the Network tab and empty it if necessary, then send your form and check out the request that pops up) The main issue I can see is that your form might be sending a user ID or some other identifying token, and maybe the only way you can get that is to search for their name? You might be able to do that with requests too... but yeah, if you can just do your own HTTP it'll be way faster and simpler. But if you also want to learn Selenium, this is a good way to do it!
|
|
#
?
Dec 5, 2017 08:56
|
|
|
Hughmoris posted:Automation engineer sounds pretty cool. The post requests are easier...if you can figure out how to do it. Sometimes it's kinda difficult on crazy Javascript heavy pages. Chrome dev tools is badass, so with some effort anything is possible.
|
|
#
?
Dec 5, 2017 16:25
|
|
|
Also try postman: https://www.getpostman.com/
|
|
#
?
Dec 5, 2017 17:14
|
|
|
Thanks for the advice. After attempting to (badly) analyze the network traffic for direct POST requests, I ended up going the selenium route. It was a nice learning exercise, and I've discovered quite a few little tricks that I think will make the next time easier.
|
|
#
?
Dec 6, 2017 05:37
|
|
|
I found this very interesting: https://medium.com/dunder-data/python-for-data-analysis-a-critical-line-by-line-review-5d5678a4c203 A brutal review of Wes McKinney's book on pandas.
|
|
#
?
Dec 6, 2017 12:35
|
|
|
Cingulate posted:I found this very interesting: https://medium.com/dunder-data/python-for-data-analysis-a-critical-line-by-line-review-5d5678a4c203 I think he nit picks here and there but overall I tend to agree with a lot of his points. I’ve never read his cookbook on pandas but maybe I’ll get work to buy it and I’ll thumb through it. I’ve used pandas enough now that all I usually need is a quick scan of the docs to remember things. However one thing he mentions early in his review that drives me up the loving wall is how pandas devs use all of these random weird functions and methods that aren’t really documented anywhere and when you’re learning the library it’s really hard to figure out what the hell they are for.
|
|
#
?
Dec 6, 2017 15:29
|
|
|
Is this summary of Pandas accurate? It's how I look at it/assess when to use it: Wrapper for 2-D and 1-D arrays that includes labels, non-numerical indexing, different syntax, and many R-style statistical methods and filters. Orders-of-magnitude slower than the wrapped array, but if a problem, can convert to an array, perform bottlenecked-calcs, then convert back to a DF/Series. Dominoes fucked around with this message at 17:01 on Dec 6, 2017 |
|
#
?
Dec 6, 2017 16:58
|
|
|
Dominoes posted:Is this summary of Pandas accurate? It's how I look at it/assess when to use it: Or, for people who know R: "Dataframes for Python."
|
|
#
?
Dec 6, 2017 19:03
|
|
|
Cingulate posted:"The Python default for dealing with tabular data." Exactly. I like DataFrames for quickly working with data in properly formatted CSV files -- especially when I need to work on groups of the data.
|
|
#
?
Dec 6, 2017 19:12
|
|
|
So I have some data I want to explore, with lots data manipulation like joins, groupby/pivots etcs. The actual data is from several CVS and need to be combined in different ways, and especially one tables is very memory expensive. I've tried two options. 1) Using Pandas dataframes exclusively. There you have merge, pivot and stack/unstack commands I can gently caress around with. 2) I can dump the data into a SQLite database and do all data manipulation as SQL. The extract data asa Pandas dataframe when needed for graphics. I suspect option 1) interacts better with other Pandas-functions, but I find that plots interact weirdly with multiindex data when I do data manipulation. The SQL-option does make the data operations more readable and possibly more accessible to other users, as more or less everyone are good at SQL. Any thoughts?
|
|
#
?
Dec 7, 2017 00:56
|
|
|
Learned today that PyCharm's debugger can display huge arrays in a separate window as a table of cells. I always thought this was something that pycharm lacked but I was just totally wrong
|
|
#
?
Dec 7, 2017 07:01
|
|
|
QuarkJets posted:Learned today that PyCharm's debugger can display huge arrays in a separate window as a table of cells. I always thought this was something that pycharm lacked but I was just totally wrong Yeah, it was an embarrassingly long time before I stumbled across that.
|
|
#
?
Dec 7, 2017 13:07
|
|
|
So, I'd like to get into Python, I'm currently an infrastructure/Ops guy and I'd like to add Dev in front of that and get on the money train (and advanced with technology), I figured Python is fairly system agnostic and can be applied in a lot of places. Is a good starting place the https://docs.python.org/3/ (official docs) tutorial? I went to school for programming (did C/C++/C#, HTML/CSS, Java and a few other things), but did not graduate and it's been 10 years since I've touched any of that, but I do use powershell now as much as possible, just to give you a background and rough estimate of where I'm at.
|
|
#
?
Dec 7, 2017 21:07
|
|
|
A lot of people learn through books and whatnot but I learned through the official tutorial docs and I think it was great for me. If you already understand programming then it's probably a fine option for you as well. I would recommend coming up with a not-too-complicated project that you can do as a learning goal, and then whenever you get far enough into the tutorial that you think you can accomplish it, stop reading and try the project.
|
|
#
?
Dec 7, 2017 21:12
|
|
MF_James posted:So, I'd like to get into Python, I'm currently an infrastructure/Ops guy and I'd like to add Dev in front of that and get on the money train (and advanced with technology), I figured Python is fairly system agnostic and can be applied in a lot of places. Is a good starting place the https://docs.python.org/3/ (official docs) tutorial? I went to school for programming (did C/C++/C#, HTML/CSS, Java and a few other things), but did not graduate and it's been 10 years since I've touched any of that, but I do use powershell now as much as possible, just to give you a background and rough estimate of where I'm at. I think the Python 3 docs are as solid of a place as any to start.
|
|
|
#
?
Dec 7, 2017 21:21
|
|
|
SurgicalOntologist posted:A lot of people learn through books and whatnot but I learned through the official tutorial docs and I think it was great for me. If you already understand programming then it's probably a fine option for you as well. I would recommend coming up with a not-too-complicated project that you can do as a learning goal, and then whenever you get far enough into the tutorial that you think you can accomplish it, stop reading and try the project. Yeah, I still understand programming at a higher level, I just don't know all the different terms thrown around that specific to each language (or are just newer terms that have come about in the past 10 years), which I can research (if needed) as I run into them in any tutorial. Eela6 posted:I think the Python 3 docs are as solid of a place as any to start. Cool! I'll give this a shot, thanks!
|
|
#
?
Dec 7, 2017 22:27
|
|
|
Thermopyle posted:It's probably better to ask in the general programming thread as there's nothing specific to python. OK, I want one of these magic scanners. I know scanners with OCR with the right fonts can do whatever, but I'm dealing with some complicated forms. They're all roughly the same form with minor irritating variations. If there's a particularly good scanner that we should be using I'd love to hear about it? Not being snarky.
|
|
#
?
Dec 9, 2017 06:49
|
|
|
MF_James posted:So, I'd like to get into Python, I'm currently an infrastructure/Ops guy and I'd like to add Dev in front of that and get on the money train (and advanced with technology), I figured Python is fairly system agnostic and can be applied in a lot of places. Is a good starting place the https://docs.python.org/3/ (official docs) tutorial? I went to school for programming (did C/C++/C#, HTML/CSS, Java and a few other things), but did not graduate and it's been 10 years since I've touched any of that, but I do use powershell now as much as possible, just to give you a background and rough estimate of where I'm at. Google 'Data Structures & Algorithms In Python Goodrich'. Looks like a freely available book. I say looks like, because I used it, and it appears to be free. Anyway start with the first couple chapters. Or more if you like it. I liked it.
|
|
#
?
Dec 9, 2017 06:54
|
|
|
For the Pandas users out there, what type of things (if any) do you bounce back to Excel for?
|
|
#
?
Dec 10, 2017 01:20
|
|
|
I use excel to get data, then import it to pandas. A lot of my data sources are microsoft ecosystem excel add-in monstrocities.
|
|
#
?
Dec 10, 2017 01:40
|
|
|
FAGGY CLAUSE posted:OK, I want one of these magic scanners. I know scanners with OCR with the right fonts can do whatever, but I'm dealing with some complicated forms. They're all roughly the same form with minor irritating variations. If there's a particularly good scanner that we should be using I'd love to hear about it? Not being snarky. My HP 8600 and 8720 both do this.
|
|
#
?
Dec 10, 2017 05:10
|
|
|
Are there any tutorials out there that's mostly dealing with PyCharm and some of the more useful things it can do? I've been using it for a while now and it's pretty great, but I can't help but feel like I'm missing out on a lot of its functionality that I either don't know exists or don't really know how to use to full advantage.
Boris Galerkin fucked around with this message at 15:15 on Dec 10, 2017 |
|
#
?
Dec 10, 2017 15:13
|
|
|
Hughmoris posted:For the Pandas users out there, what type of things (if any) do you bounce back to Excel for? Nothing other than writing out tabular summary views of the data I’m working with (which I may clean up the formatting here and there) that are usually sent to coworkers, or opening up Excel files I’ve been sent to see how they’re formatted, so I know how to import them.
|
|
#
?
Dec 10, 2017 15:16
|
|
|
Boris Galerkin posted:Are there any tutorials out there that's mostly dealing with PyCharm and some of the more useful things it can do? I've been using it for a while now and it's pretty great, but I can't help but feel like I'm missing out on a lot of its functionality that I either don't know exists or don't really know how to use to full advantage. I'd say let's just share here the cool features we know. - Right Click on variable/filename/etc -> Refactor -> Rename - finds any instance of that variable and renames it. Even able to look inside strings - Navigate -> File / Class - Just learned this this week. Open any file within your project. So much quicker than navigating the file structure - Click on any part of the file path at the top of the screen and you can browse relative to the file you're currently viewing. - View -> Tool Window -> Todo - Write TODO in any comment and you can keep track of all the things you need to do. Useful when wiring up a project initially. - Debug - So glad I don't have to throw print statements everywhere now - Integration with GIT is pretty cool. Especially color changes for lines/files within your project.
|
|
#
?
Dec 10, 2017 21:33
|
|
|
I also like Ctrl+Click on anything (function, variable, class, etc) to jump to definition or, if you're at the definition, display a list of all places used. Edit: Also I use Vagrant for everything I work on since I develop on Windows and run production on Linux. Pycharm has Vagrant integration, and the remote python interpreter support is the bee's knees. Full run/debug support using a python interpreter installed on a different computer. porksmash fucked around with this message at 17:58 on Dec 11, 2017 |
|
#
?
Dec 11, 2017 17:52
|
|
|
These are important if you don't know 'em: https://www.youtube.com/watch?v=XOkNJxvNtPw Code completion (ctrl+space), using snippets and templates, and quick fixes (alt+enter) instead of typing all the dang code, and stuff like auto formatting https://www.youtube.com/watch?v=jmTo5xTRka8 These are huge: turn off your tabs, hide the tab bar because you don't need it, preferably put away your mouse for a bit, gotta go fast
In the help menu there should be a 1-page pdf of keyboard shortcuts - have a look at that, because it lists a bunch of cool features that make you go 'wait... is that...?' and then you find out it is. I forget if pycharm has it, but there should also be Productivity thing in help too, that shows you features you use a lot and how much typing they've saved you, but also lists stuff you might not know about. And there's the tips popup too of course I don't know exactly how much crossover there is with IntelliJ, but this is a good talk that's old but entertaining and most of it should carry over. Get you feeling extremely powerful. Covers cool things including double-tap ctrl to add extra carets with the arrow keys (what, you work on one line at a time?  ) and uhhh alt+backspace or something to jump to the last edit position, all kinds of stuffhttps://www.youtube.com/watch?v=h8wRC7Qkcb8
|
|
#
?
Dec 11, 2017 18:42
|
|
|
Hughmoris posted:How is python+selenium for filling out lots of repetitive forms? I noticed that some people on my project team are manually entering in the data for 2000+ users in to a web portal. They've asked for help but my eyes will fall out of my head if I have to manually type in crap. Assuming you dont have access to the forms source code, consider using something like TamperData to get a capture of the forms data, then build up a script using python requests or a similar library (use requests, its fantastic) and just pump them in that way. requests should be able to hand any cookying you'll need to do. Selenium can be kinda flakey at times, due to externalities like page load times and the like. Just do POSTs using the requests library. Its super easy. Unless its some sort of hosed up ASP thing with all sorts of nasty javascript and hosed up non-restful state. Those things can be bitches to scrape and post to. edit: I must be the 5th or 6th person to suggest this. My bad. duck monster fucked around with this message at 13:48 on Dec 12, 2017 |
|
#
?
Dec 12, 2017 13:43
|
|
|
Python-adjacent question: Recommendations for a rich ui / internet application frontend to a Python-backed webapp? No more constraints to give, because this is strictly in the early / idle speculation stage for a possible analysis / data management platform. But I'd like the keep the backend in Python and while there was a burst of RIA toolkits some year ago, it seems to have settled down.
|
|
#
?
Dec 12, 2017 15:04
|
|
|
outlier posted:Python-adjacent question: Recommendations for a rich ui / internet application frontend to a Python-backed webapp? Are you building it in Django or Flask or ... hopefully you're not creating your own? Django comes with templates (https://docs.djangoproject.com/en/2.0/topics/templates/) and Flask works well with Jinja 2(http://jinja.pocoo.org/docs/2.10/). If you're using Django Rest Framwork it pairs pretty nicely with React, and I'm guessing every other front end framework out there these days (I only know React).
|
|
#
?
Dec 12, 2017 16:20
|
|
|
huhu posted:Are you building it in Django or Flask or ... hopefully you're not creating your own? Django comes with templates (https://docs.djangoproject.com/en/2.0/topics/templates/) and Flask works well with Jinja 2(http://jinja.pocoo.org/docs/2.10/). If you're using Django Rest Framwork it pairs pretty nicely with React, and I'm guessing every other front end framework out there these days (I only know React). I don't have any thoughts about the backend at the moment other than I'd like it to be a Python-based one (which I am not going to build myself). I figured the choice of frontend might constraint the backend. Will chase up Django REST.
|
|
#
?
Dec 12, 2017 17:26
|
|
|
duck monster posted:Assuming you dont have access to the forms source code, consider using something like TamperData to get a capture of the forms data, then build up a script using python requests or a similar library (use requests, its fantastic) and just pump them in that way. requests should be able to hand any cookying you'll need to do. Thanks for the advice. I've been poking around this some more to see if I can figure out how to do direct POST actions. I'm seeing a lot of jQuery and ajax stuff occuring when I'm looking in developer tools. I believe I've found where the form post occurs but it appears to have some key/tokens associated with it so I'm not sure how to approach it in Python. My web knowledge is weak so I might not be using the right terms.
|
|
#
?
Dec 12, 2017 18:23
|
|
|
outlier posted:I don't have any thoughts about the backend at the moment other than I'd like it to be a Python-based one (which I am not going to build myself). I figured the choice of frontend might constraint the backend. Will chase up Django REST. Something like React + DRF is what I do for work and it's a pretty solid pairing. There's quite a lot you'll need to read up on though. At the very least probably React, Redux, Webpack, ES6, Axios, Django, and Django Rest Framework. If you don't need a fancy front end and your back end isn't complicated I'd suggest going the Flask + Jinja 2 route.
|
|
#
?
Dec 12, 2017 21:26
|
|
|
outlier posted:Python-adjacent question: Recommendations for a rich ui / internet application frontend to a Python-backed webapp? Vue.js if you want to go full blown twiddly moustache. Its a slightly more parsable/simple alternative to the React way of doing things, although be rest assured its trying its hardest to turn into a fullblown wanker framework. Ractive if you want to be a sane human who doesnt want to learn 50 different frameworks, build toolchains, etc. I seriously don't know why this tiny itty bitty framework doesnt get more love. Its been around for a while, and you can learn it in a sitting. Ignore Angular with supreme prejudice. All you need to know about angular is "Its a web framework with a dependency injection service provider system". Horrific Java inspired braindamage duck monster fucked around with this message at 02:38 on Dec 13, 2017 |
|
#
?
Dec 13, 2017 02:34
|
|
|
Hughmoris posted:Thanks for the advice. I've been poking around this some more to see if I can figure out how to do direct POST actions. Whats the web thing based on? duck monster fucked around with this message at 02:38 on Dec 13, 2017 |
|
#
?
Dec 13, 2017 02:35
|
|
|
|
| # ? May 16, 2024 03:45 |
|
|
duck monster posted:Whats the web thing based on? I'm not positive what your question is. Its an administrative web portal where a user can assign roles and permissions to other users. I'm not sure what stack it's running but the interface looks pretty old.
|
|
#
?
Dec 14, 2017 00:51
|
|