|
samcarsten posted:Ok, got that working, new error when I try to click the button. Can be caused by a variety of factors, one being if the page has updated itself. When I was using Selenium I often did it in Pycharms interactive debug mode to help. It can be very fiddly. Especially deciding on the right selector to get the element you need on poorly designed sites.
|
# ? Nov 14, 2022 20:00 |
|
|
# ? May 15, 2024 03:55 |
|
Sad Panda posted:Can be caused by a variety of factors, one being if the page has updated itself. I tried using the wait command after inputting the text, but that didn't work. any other ideas?
|
# ? Nov 14, 2022 21:30 |
|
12 rats tied together posted:Ruby guru + woman sounds like Sandi Metz, who I have posted about ITT, but she is pro-OOP in the form of Alan Kay's "OOP is mostly message passing", and against the improper use of inheritance. That's the one! Thank you, this is a great talk
|
# ? Nov 14, 2022 22:49 |
|
Josh Lyman posted:The columns always have the same names because they're converted from a numpy array in a previous statement with mydf_in = pd.DataFrame(myNumpyArray). When I print mydf_in, the columns are just "named" 0 1 2. Why is list_column an argument of the function? According to this explanation, isn't it always "2"? What specific key is mentioned in the KeyError exception? If it's something you don't expect (it sounds like this should only ever be "2") then there's a typo in your code. If it's something you do expect, then you encountered an input array with an unexpected shape and need to write code to handle that
|
# ? Nov 14, 2022 22:58 |
|
i wonder if someone went thru the history of this thread and counted up the number of python issues that wouldn't have existed in a language with static typing, what % of the posts would that be
|
# ? Nov 15, 2022 00:07 |
|
DELETE CASCADE posted:i wonder if someone went thru the history of this thread and counted up the number of python issues that wouldn't have existed in a language with static typing, what % of the posts would that be Index and key errors are the most common I'm pretty sure, static typing wouldn't help with those. For instance if Josh Lyman was using C they'd probably be asking for guidance on how to debug an infrequently occurring segmentation fault. Although there have been some newer programmers itt who have received advice on how to avoid typing issues before they can become a problem (e.g. Try not to change a label's type) A lot of people just want style tips or are asking for library recommendations
|
# ? Nov 15, 2022 05:11 |
|
QuarkJets posted:Why is list_column an argument of the function? According to this explanation, isn't it always "2"? Of the last 20,000 or so mydf_in's, the KeyError occurred about 25 times. edit: Modified the code to remove the function call and put the statements directly in the script and cleaned it up a bit, should help with debugging if the KeyErrors still happen. Josh Lyman fucked around with this message at 07:54 on Nov 15, 2022 |
# ? Nov 15, 2022 05:51 |
|
Josh Lyman posted:It says "KeyError: 2". I'll remove that as an argument and see if that helps. Reducing encapsulation is rarely going to help you and I really recommend you don’t do that. I’d say about 40% of the weird python errors I’ve had to help coworkers with were caused by a name collision in a huge monolithic script, where breaking it up into smaller functions would have either prevented it entirely or made the error much more obvious. It’s a bit inelegant, but have you tried catching the KeyError and setting a breakpoint inside the catch statement? That should let you inspect the variables at the point of the error and get a better idea of what’s going on. Also, what’s your source for this data? Your comment about how three notebooks simultaneously hit an error kind of makes me suspect that the issue might originate with your data source (eg a database connection dropped, or JWT token expired) but is not getting caught until later.
|
# ? Nov 15, 2022 08:55 |
|
Josh Lyman posted:It says "KeyError: 2". I'll remove that as an argument and see if that helps. Given that KeyError, we can conclude that the function received a dataframe that didn't have 3 columns of data. I think it's likely that you received an array with an unexpected shape, and that's probably what you'll see if you inspect one of these objects the next time that you encounter the exception
|
# ? Nov 15, 2022 09:06 |
|
If you can, run it in a debugger (I know PyCharm and VSCode both have good debugging tools, but you can also figure out how to use good old fashioned PDB), this gives you a much better ability to see wtf is going on, and is an invaluable skill to build. The number of people I know who are like 'oh I'll just add a ton of print statements instead of one breakpoint' drives me a little crazy sometimes.
|
# ? Nov 15, 2022 18:49 |
|
QuarkJets posted:Given that KeyError, we can conclude that the function received a dataframe that didn't have 3 columns of data. I think it's likely that you received an array with an unexpected shape, and that's probably what you'll see if you inspect one of these objects the next time that you encounter the exception Hopefully that fixes the issue! edit: It fixed the issue, thanks everyone for your help! Josh Lyman fucked around with this message at 21:45 on Nov 18, 2022 |
# ? Nov 15, 2022 23:20 |
|
I've been having a rough time figuring out how to get this to work - unfortunately since this is a Django thing with a bunch of interrelated files, this will have a lot of code snippets, sorry. Basically I have a function that dumps a bunch of data into a csv. There's a text field where the client types in their name and this function pulls the "name" tag from the html form and makes that part of the filename. So far so good. But I also want to make the type of file part of the file name as well and it's behaving badly. Here's the html that's being pulled: (full contents at pastebin) code:
code:
I tried d_type = request.POST.get("type") as well as d_type = request.GET["type"] to no avail. Sometimes I get a "MultiValueDictKeyError when trying to retrieve "type"" error, but usually it just doesn't put the type into the filename. I am admittedly taking a shotgun approach here. Is there maybe a simple way to test the results of request.GET["type"]? Thanks for any help. If you want to take a look at the application, it's here: https://csvcheck971.pythonanywhere.com/
|
# ? Nov 18, 2022 21:09 |
|
Finally getting a chance to keep working on this database. This is the code I have to set up and populate the database:Python code:
I then have the following code to obtain each time series subsequences I want for the list of (SID, date_time) pairs I need data for (on average there are between 10 to 100 pairs): Python code:
|
# ? Nov 19, 2022 02:16 |
|
Seventh Arrow posted:I've been having a rough time figuring out how to get this to work - unfortunately since this is a Django thing with a bunch of interrelated files, this will have a lot of code snippets, sorry. I would not use "type" as a variable name anywhere since its the name of a native Python function request.GET contains GET parameters (eg. query strings), request.POST contains form data. If a form field is not required, you should get the value with request.POST.get('field'), this will return None if the field is not filled out rather than causing an error. or, ideally, use a FormView (or at least a form) and avoid reinventing the wheel.
|
# ? Nov 19, 2022 03:10 |
|
Jose Cuervo posted:The problem is that searching for a time series subsequence from a single pair takes several seconds, where I thought it would be much faster (sub 1 second), so the overall for loop can take minutes to run if there are 100 pairs to iterate over. Am I doing something wrong? My first off the hip guess is that you want to add an index to whatever column(s) you're searching on, otherwise it's doing a linear scan. Try checking https://www.sqlite.org/lang_createindex.html and see if that improves your times.
|
# ? Nov 19, 2022 03:10 |
|
UraniumAnchor posted:My first off the hip guess is that you want to add an index to whatever column(s) you're searching on, otherwise it's doing a linear scan. Try checking https://www.sqlite.org/lang_createindex.html and see if that improves your times. That did the trick, thank you!
|
# ? Nov 19, 2022 04:17 |
|
fisting by many posted:I would not use "type" as a variable name anywhere since its the name of a native Python function I kind of figured as much, which is why I started using "d_type" as a variable. quote:request.GET contains GET parameters (eg. query strings), request.POST contains form data. If a form field is not required, you should get the value with request.POST.get('field'), this will return None if the field is not filled out rather than causing an error. Thanks, I'm hoping something like this will set me on the right track https://www.geeksforgeeks.org/formview-class-based-views-django/
|
# ? Nov 19, 2022 04:27 |
|
I looked up whether sqlite has a built-in datetime or timestamp type, but I guess it doesn't. That's pretty lame.
|
# ? Nov 19, 2022 05:38 |
|
Seventh Arrow posted:I remember seeing a python learning site once that had a page that was something like "Why We Don't Recommend Zed Shaw/Learn Python The Hard Way", or something like that. I thought it was Reddit, but apparently not. Their points, if true, seemed to be pretty good reasons not to use his book(s). I actually started with his book as my intro to programming and python since it seemed to be recommended everywhere. I've since used other stuff too. It's actually really good for the beginner as it starts you off doing stuff in the command line and text files so you get a basic appreciation of what python is and does. The repetitive typing in of stuff seems to pay off too as you get familiar with what stuff should flow like. I find it odd that it worked for me as normally I like the big concepts explaining first so I know why i'm trying to do something but this was more like, do this now figure out how it works. If I had someone I know wanting to learn I'd probably still point them to it or something like codecademy. But i'm still a beginner really myself to wth do I know!
|
# ? Nov 21, 2022 07:07 |
|
If anyone here prefers a book over reading a browser, I've learnt python with Introducing Python by Bill Lubanovic and found it good. And I haven't used the Python book from this series but I'm reading the SQL one and they approach teaching you the thing in a novel manner, that I am sure some of you will appreciate. I have a regex problem. code:
And yet "match_obj = import_regexer.findall(full_text)" returns None all the time instead of a match object. I've tried raw strings. I've read the documentation. I am at a loss. Thanks in advance.
|
# ? Nov 22, 2022 22:07 |
|
Dawncloack posted:I have a regex problem. You're missing the re.MULTILINE flag: Python code:
Obligatory "now you have two problems" and all that aside, I would rethink the way you've chosen to write your regex. Look into \s at the very least if you want to capture whitespace, using a .* or .+ when you actually want whitespace is guaranteed to wreck your day at some point in the future.
|
# ? Nov 22, 2022 22:55 |
|
Can anyone help me understand why this example ends in an error:code:
code:
|
# ? Nov 22, 2022 23:29 |
|
nullfunction posted:now you have two problems As a general rule if you want to interrogate Python scripts in any meaningful way, you should look into the ast module. It's part of the standard library, though I'd wager that someone's done the hard work and written parsers for php and bash as well and you could take a similar approach for other files that you may need to parse with a quick pip install. Python code:
Python code:
code:
|
# ? Nov 22, 2022 23:40 |
|
Those are awesome answers, I am certainly going to learn about \s and ast. Thank you!
|
# ? Nov 23, 2022 09:19 |
|
Deadite posted:Can anyone help me understand why this example ends in an error: In the first example you’re trying to call astype(float) on the string ‘Missing’
|
# ? Nov 24, 2022 01:43 |
|
I can tell that from the error, what I don't understand is why that string would have the method applied in the first place, since it should have been filtered out in the first part of the where function.
|
# ? Nov 24, 2022 02:39 |
|
Deadite posted:I can tell that from the error, what I don't understand is why that string would have the method applied in the first place, since it should have been filtered out in the first part of the where function. That's not the order of operations; the way you have this set up, astype occurs first, and that new dataset would become one of the inputs of the "where" function
|
# ? Nov 24, 2022 02:53 |
|
QuarkJets posted:That's not the order of operations; the way you have this set up, astype occurs first, and that new dataset would become one of the inputs of the "where" function So astype is the first part executed before the first "where" happens? I thought the first "where" statement would have filtered out the 'Missing' strings before the second where executes and applies the astypes
|
# ? Nov 24, 2022 03:04 |
|
Deadite posted:So astype is the first part executed before the first "where" happens? I thought the first "where" statement would have filtered out the 'Missing' strings before the second where executes and applies the astypes The order of operations might become clearer if you were to break out every subexpression and assign it to a variable, instead of packing it all into one line.
|
# ? Nov 24, 2022 05:05 |
|
Deadite posted:So astype is the first part executed before the first "where" happens? I thought the first "where" statement would have filtered out the 'Missing' strings before the second where executes and applies the astypes Let's go look at what `where` does https://numpy.org/doc/stable/reference/generated/numpy.where.html quote:numpy.where(condition, [x, y, ]/) Put another way, `where` creates a new array with elements selected from x or y, depending on some conditional. It does not actually transform any data; conditional, x, and y are not being modified. So you're really passing in 3 input arrays and getting 1 output array. So what are conditional, x, and y in your case? Here's the line you wrote: Python code:
Python code:
Python code:
Since y is one of the inputs to `where`, it actually needs to be formed first before `where` can really execute. It doesn't matter that you were doing this in-line, you were still creating that temporary array. So what can you do about this? You can use boolean arrays to actually return a filtered array, operate on it, then write it back. Python code:
Python code:
QuarkJets fucked around with this message at 05:37 on Nov 24, 2022 |
# ? Nov 24, 2022 05:34 |
|
The good news is that my manager liked that csv analyzer project that I've been talking about. The bad news is that I'm now the office "python guy" and he dumped a huge ETL project in my lap, full of all kinds of things I don't know how to do, including:
Of course, I have a trillion questions, but for the sake of sanity I'm going to limit myself to a few: i) He wants to pull (meta)data from a REST API and then put it into a table with columns and rows. This doesn't make sense to me - as far as I know, data is retrieved from APIs in JSON format and would more properly fit in a dictionary rather than columns and rows. Am I missing something, or is there a way to do this? (the metadata that I saw has a format like "{character limit: 16, decimal places: 2}", etc.) ii) Is there some sort of awesome tutorial that will aid me in my journey of learning how to make python communicate with relational databases like MySQL? iii) As far as I know, none of the metadata is actually tagged in the system as "metadata." Is there any strategy for simplifying this, or am I really just going to have to pull each and every single item by name and assign it to a variable?
|
# ? Nov 24, 2022 05:48 |
|
Oooooooh okay, I see now. Thank you for your very clear explanation. I didn't think about how all of the arguments in the "where" function needed to be resolved before the function executes, but that is how literally all functions work. I just couldn't see it in this case for whatever reason. For context I was trying to write a "where" statement to replace values in a column that already contained the "Missing" strings, so I needed to find a way to filter those out, then apply the criteria to the intended population, and return all the values back to the dataframe with every value in its original index position.
|
# ? Nov 24, 2022 05:53 |
|
Seventh Arrow posted:The good news is that my manager liked that csv analyzer project that I've been talking about. i) You could think of each field in the JSON as a column, and one row is whatever you get from 1 call to the API. Or you could be inserting several rows from 1 API call. It really depends on what the API is actually returning and how that relates to the tables in the database. It sounds like this database already exists, and your boss is asking you to automate a data entry task that people have to do by hand - if that's the case, then go talk to the people currently doing that and take notes. ii) At the end of the day you probably want something like https://github.com/PyMySQL/PyMySQL, it's pretty easy to use. There's also SQLAlchemy, which I don't have experience using I just know that it's got a more Python-y interface whereas pymysql is more about executing whatever SQL you specify. iii) First you should develop a model of the data you're receiving from the salesforce API and a model of the data you're handling with the database and its procedures. That's the hard but important part. Then you can build up your data structures around those details. Then you can write the functions that transform the data from one system to the other. Use a dictionary to store field name/value pairs if you want to do something basic, but a dataclass is the right approach for data structures with rigid schemas, such as MySQL tables and specific keys returned by API calls.
|
# ? Nov 24, 2022 06:56 |
|
QuarkJets posted:i) You could think of each field in the JSON as a column, and one row is whatever you get from 1 call to the API. Or you could be inserting several rows from 1 API call. It really depends on what the API is actually returning and how that relates to the tables in the database. It sounds like this database already exists, and your boss is asking you to automate a data entry task that people have to do by hand - if that's the case, then go talk to the people currently doing that and take notes. Good advice, I will mull over this thank you!
|
# ? Nov 24, 2022 14:43 |
|
ExcessBLarg! posted:That's a name I haven't seen in a long time. I'd say the vast majority either never make it to market, or remain small and profitable enough that it survives but not necessarily goes big. What I've seen over 20+ years in the industry is that time and time again startups get locked in overly long development cycles and burn out all their cash without ever getting to the launch because they get stuck trying to engineer it for a billion customers when they havent got around to getting one customer yet. I'm always advising "Get the minimal viable product, and if we need to rapidly scale, then thats a good problem to have and we can hire some guys. Don't go overboard on features yet, just make it the basic product you promised the investors and we can add the juicy stuff AFTER launch", and don't get distracted by shitful buzz ideas like blockchains or whatever. And conceptually if you cant explain the thing in the time it takes an elevator to get to the lunch bar, the whole idea desparately needs a rethink. And ultimately scaleable isn't THAT much harder than stick to the heroku 12 principles, stick pg_bouncer in front of yr database and MAYBE a NoSQL if a clear case can be made for it. Just jam the thing in a kubernetes autoscale group and blammo, you have something that'll scale up to the first million customers (which lets face it will be a miracle if it ever gets that big but its a nice thing to tell investors) Uh. ADHD what am I talking about. Oh right , yeah my point is SUPPOSED to be that ultimately overly long dev times kill companies dead, because the REAL cost of the operation is always gonna be actually developing the thing (and if you make it to the end marketing the thing)
|
# ? Nov 26, 2022 00:02 |
|
duck monster posted:I'd say the vast majority either never make it to market, or remain small and profitable enough that it survives but not necessarily goes big. Most businesses throughout human history have been impatient for profitability but patient for growth. Modern financial systems have made the inverse possible, with sometimes very weird results! I am new to programming and Python and having some fun with it. I'm connecting to a Cloudera Impala ODBC source through Python. I'm using pyodbc library and it's working fine, but throws an angry warning that I should be using SQLAlchemy instead. Should I?
|
# ? Nov 26, 2022 00:47 |
|
QuarkJets posted:iii) First you should develop a model of the data you're receiving from the salesforce API and a model of the data you're handling with the database and its procedures. That's the hard but important part. Then you can build up your data structures around those details. Then you can write the functions that transform the data from one system to the other. Use a dictionary to store field name/value pairs if you want to do something basic, but a dataclass is the right approach for data structures with rigid schemas, such as MySQL tables and specific keys returned by API calls. +1 to this, because it's a very easy step to something skip past (and if you're using to writing one-off scripts, is something you generally do skip in those cases). But sitting down and really understanding what the data is you're handling is going to make it a ton easier to iterate on moving forward, instead of getting into dictionary hell where you find yourself passing around a gigantic dictionary of values you have no idea about. FWIW: If the API you're using returns a massive pile of data for some reason, it's also okay to go 'alright it's going to give me back a dictionary with fifteen sub-entries, I'm going to discard 14 of those and only model the one I actually care about for this application. For example, I was working with a ticketing API, so call it and you get back all the data in the ticket, but I only really cared about one specific portion. No reason for me to write up proper object handling for all the bits I'm never going to interact with in this program, so I just started by something like this: Python code:
Falcon2001 fucked around with this message at 01:20 on Nov 26, 2022 |
# ? Nov 26, 2022 01:15 |
|
Yeah don't be afraid to drop data that you don't actually care about. For instanceSeventh Arrow posted:(the metadata that I saw has a format like "{character limit: 16, decimal places: 2}", etc.) Do you actually care about these fields? I'm guessing not; there's probably no need to insert these into your database. But you can't really know without first understanding what's in your database. Don't assume anything.
|
# ? Nov 26, 2022 01:51 |
|
I'm probably going to need all of the metadata, yes. The idea is to have a dynamic ruleset that gets applied whenever we need to validate the client's actual data. So when the, say, account number is being validated, there should be an up-to-date rule saying that it has a max of 16 characters or whatever. Fortunately the validation is going to be done by stored procedures, which I will not be in charge of writing. All that my initial script has to do is pull the metadata from Salesforce and update the database with it. Fortunately, there's seems to be a module that makes interacting with salesforce easier: https://pypi.org/project/simple-salesforce/ If I get in over my head, I can just see if I can hire help from fiverr I guess.
|
# ? Nov 26, 2022 03:13 |
|
|
# ? May 15, 2024 03:55 |
|
Does anyone have an idea why the colors in vscode don't work? Like if I put some piece of code as #FF5050 in settings.json, it realizes as #9f4557 on my screen. It is so much lower in brightness and muted. I don't understand how inputting 1 color results in another color being displayed. Any tips? I use bright colors like code:
Like this. Qt left, VSC right: Edit: --disable-gpu launch param seems to have helped. Ihmemies fucked around with this message at 11:11 on Nov 26, 2022 |
# ? Nov 26, 2022 10:45 |