Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
General_Failure
Apr 17, 2005

Thermopyle posted:

I swear I've posted warnings about this multiple times in this thread! THANKS FOR NOT READING MY WORDS!


You may well have. I've only dropped into this thread recently. Not new to programming but new to python.

Adbot
ADBOT LOVES YOU

FCKGW
May 21, 2006

I learned python in community college a few years back and I'm going to be going back to school soon and would like to get a refresher course.

Anyone have any online video series they could recommend? Something from Coursersa or Edx or Codeacademy, etc..?

M. Night Skymall
Mar 22, 2012

FCKGW posted:

I learned python in community college a few years back and I'm going to be going back to school soon and would like to get a refresher course.

Anyone have any online video series they could recommend? Something from Coursersa or Edx or Codeacademy, etc..?

I really liked the mit one on edx.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

FCKGW posted:

I learned python in community college a few years back and I'm going to be going back to school soon and would like to get a refresher course.

Anyone have any online video series they could recommend? Something from Coursersa or Edx or Codeacademy, etc..?

I like codecademy but even more than that I like picking something you want to make and making it. You'll pick it up quickly.

unpacked robinhood
Feb 18, 2013

by Fluffdaddy
Can you run a Flask app as a Celery worker ?
I'm working on a project with a base app file that goes like this (way oversimplified)

rest.py
Python code:
apf = Flask()
#init and config
#...
#
celery = Celery()
task1 = OddJob()
celery.tasks.register(task1) # this call seems useless
apf.run()
Oddjob.py:
Python code:
apc = celery.Celery(..)
class Oddjob(celery.Task):
    def __init__():
        pass
    def run(*args, **kwargs):
        #do something!
apc.tasks.register(Oddjob()) # it doesn't work without this call
If I run Oddjob as a celery worker and run the Flask app everything works correctly.
In that situation I can request a task to be run by sending a request to the correct url, and get the result.

I'm asking because according to the author launching the flask app as a celery worker should be enough. It looks like
code:
celery worker -A rest:celery --stuff
If I do this the Flask app runs but celery.control.inspect().registered_tasks() is empty (It's correctly populated if I run both apps separately). Calling a task just stalls
If I Ctrl+C in the console, Flask stops and a celery worker starts, a second Ctrl+C stops the worker.

An alternate question would be, can you programmatically create a Task and spin it as a celery worker ?

unpacked robinhood fucked around with this message at 16:43 on Jun 20, 2019

larper
Apr 9, 2019
I am building a scraper for a music thread on here. bs4 is parsing the output and populating a dict with the posts, and each post contains dictionaries of the various youtube links, each with a 'scraped' key that contains a boolean to indicate whether or not its been downloaded. I would like to use youtube-dl's python api but I can't get it to work.

Python code:
def download_youtubes(dict_in):
    for post in dict_in:
        poster = dict_in[post]['poster']
        youtubes = dict_in[post]['youtubes']

        for id in youtubes:
            print('starting process for video',id)
            process = subprocess.Popen(ytdlcmd('download',id),stderr=subprocess.PIPE)
            errors = process.communicate()
            print(errors)
            print(len(errors))
            print('process completed for video',id)

            print('updating flagged dict')
            dict_out_file = open('dict_with_scraped_flag.json','w')
            dict_out[post]['youtubes'][id]['scraped'] = True
            dict_out_file.write(json.dumps(dict_out))
            dict_out_file.close()
            print('dictionary updated with scrape status')
If the internet goes out, subprocess gives the status of the download, but if it fails the program stalls and won't exit. Putting the pipe in stdout shows me errors and exits the process to go to the next one, but I can't tell how to turn that into an error code.

I'm halfway through a workaround that checks if the output file for each scrape process has been created (if it fails, these are not created and it is essentially a status code) but I would like to figure out how to do it with subprocess alone.

larper fucked around with this message at 00:06 on Jun 21, 2019

Sylink
Apr 17, 2004

Is there a good practical guide on the logging module anywhere?

I'm working on an internal library/module, with a setup that I think is typical (I'm not a professional dev by any means(

code:
lib_dir/
- subpackage
- subpackage
- ...
setup.py
Then pip install etc

Anyway, my question is how do I get the logging module imported everywhere consistently? Regardless of which subpackage I need in a particular script, I want to have the same logging options and be able to control them with command line flags, that way I can litter the code with proper log levels to help with later debugging.

But I see even with the logging info in the top level __init__.py , it doesn't take affect down to the subpackages.

QuarkJets
Sep 8, 2008

The doc pages for the logging module are very good and should basically be able to explain everything that you need. You should be able to Google for them, they'll be a lot more helpful than the module's docstrings

The module uses named instances, so if your submodules create logging instances with the same string name it should all simply work

Dr Monkeysee
Oct 11, 2002

just a fox like a hundred thousand others
Nap Ghost
logging.basicConfig definitely takes effect on subpackages but it has to be called before any other logging calls happen in your code or basicConfig does nothing.

Once you’ve set up basicConfig the way you want all loggers created via logging.getLogger will inherit those settings.

Mycroft Holmes
Mar 26, 2010

by Azathoth
i'm doing some homework and could use some help.

i've got an array of strings and i need to display the first letter of each string. i think the best way to do this would be a for..in loop, but i'm not sure how.

also, my cvode to find the number of user input letters in an array isn't working

code:
letter = str(input("Please enter a letter"))
print('17' , print(a.count(letter)))

Mycroft Holmes fucked around with this message at 19:24 on Jun 25, 2019

The Fool
Oct 16, 2003


Mycroft Holmes posted:

i'm doing some homework and could use some help.

i've got an array of strings and i need to display the first letter of each string. i think the best way to do this would be a for..in loop, but i'm not sure how.

strings are arrays of characters

Mycroft Holmes
Mar 26, 2010

by Azathoth

The Fool posted:

strings are arrays of characters

I am aware of that. I need a section of code that works regardless of how many strings are in the array

Mycroft Holmes
Mar 26, 2010

by Azathoth
i also have another array that is a list of vowels. I need to count how many times the vowels in array 2 appear in the array 1. How can I do this?

Chakan
Mar 30, 2011

Mycroft Holmes posted:

I am aware of that. I need a section of code that works regardless of how many strings are in the array

are you just looking for

code:
array = []
array.append("Hello")
array.append("World")
for x in array:
    print(x[0])
Which gives the expected result:
code:
H
W
This works because it just iterates through each value in the array and prints the first character in the list.


If that's not what you need, you might want to re-state?

Mycroft Holmes posted:

i also have another array that is a list of vowels. I need to count how many times the vowels in array 2 appear in the array 1. How can I do this?

The easiest and probably conceptually simplest way is to take each vowel in 2 and make a for loop that says "for each character in 1, check if it's the same as the vowel I've got, if so, increment a counter. Once you're done with everything in array 1, print the counter, and reset it to zero."

The Fool
Oct 16, 2003


Mycroft Holmes posted:

I am aware of that. I need a section of code that works regardless of how many strings are in the array

Then maybe I didn't understand which part you needed help with.

In broad strokes:
You use your for loop to loop through your array of strings.
For each string in your array, you display the first character of the string.


Mycroft Holmes posted:

i also have another array that is a list of vowels. I need to count how many times the vowels in array 2 appear in the array 1. How can I do this?

there are better ways to do this, but the naive way would be to use two loops:
use a for loop each vowel, then loop through your string comparing the current letter to current vowel
if you get a match, increment a counter

Mycroft Holmes
Mar 26, 2010

by Azathoth

The Fool posted:

Then maybe I didn't understand which part you needed help with.

In broad strokes:
You use your for loop to loop through your array of strings.
For each string in your array, you display the first character of the string.


there are better ways to do this, but the naive way would be to use two loops:
use a for loop each vowel, then loop through your string comparing the current letter to current vowel
if you get a match, increment a counter

ok, how do i select each entry? just a[0] and increment by one each time?\

the actual questions are:
4. Display a string that consists of the first letter from each of 11 names in the list a
Output should be the string “EANDFTEEBFN”
See 9 in the output and consider using the + operator Must use a loop
5. Display a string that consists of the last letter from each of 11 names in the list a
Output should be the string “dsnstgrneih”

Mycroft Holmes
Mar 26, 2010

by Azathoth
ok, so i've got

code:
for b in a:
    string1.append(a[0][0])
    a[0][0] + 1
so i should have it select a[0][0] and then increment to a[1][0]. how do i do that?

Mycroft Holmes fucked around with this message at 20:01 on Jun 25, 2019

Hollow Talk
Feb 2, 2014
On top of an annoying day, I've banged my head against this a fair bit today:

How do I go about mocking calls to functions from the requests library (post, get) etc in my unit tests? More specifically, I know how to make it return values (using unittest.mock.patch and its side_effect argument), but requests still tries to connect to the underlying TCP socket etc., which I very much want to avoid. I tried overriding bits and pieces of urllib3, socket, and requests itself, to no avail. I would prefer to this with just pytest/mock, avoiding an external library like requests-mock.

Any ideas?

mbt
Aug 13, 2012

I've never programmed in an enterprise environment but probably want to do that now. Are ETL and Airflow something I can learn shallowly and easily enough to slap on my resume or would someones bullshit meter immediately be flipped

The Fool
Oct 16, 2003


Mycroft Holmes posted:

ok, so i've got

code:
for b in a:
    string1.append(a[0][0])
    a[0][0] + 1
so i should have it select a[0][0] and then increment to a[1][0]. how do i do that?


Python code:

for x in range(0,len(a)):

Then x will be a number that automatically increments for the length of a.

Then you can access the first element of your second dimension by a[x][0]

The Fool fucked around with this message at 20:11 on Jun 25, 2019

Chakan
Mar 30, 2011

Mycroft Holmes posted:

ok, how do i select each entry? just a[0] and increment by one each time?\
5. Display a string that consists of the last letter from each of 11 names in the list a
Output should be the string “dsnstgrneih”

The right way to do this is to basically do the same as question 4, but instead of zero, you use -1, because that will make it take the last letter. so that it looks like this:
code:
array = ["Patty", "Selma", "Marge"]
for x in array:
    print(x[-1])
or, more accurately, it should be result.append(x[-1]) then print(str(result))

e: The Fool I think they're trying to go about it in a backwards kinda way.

baka kaba
Jul 19, 2003

PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

A for loop basically iterates over a bunch of items, handing you each item in turn. So in the body of the loop, you do something with the current item, and when you run the loop it will basically do that thing with all the items.

so you want something like
Python code:
# start with an 'empty' version of the thing that's gonna hold your result
final_string = ""
# loop over all the names - each item you're handed is gonna be called 'name' so you can refer to it in your loop
for name in names:
  # grab the first letter from the name string (item 0 in the string sequence) and add it to your result variable
  final_string += name[0]

print(final_string)
you have a bunch of names, and for each one you just want to look at the first letter and stick it on the end of what you've got so far. That's the only index you need to mess with, referring to that first letter. It's a pretty simple procedure, so make sure it makes sense to you before you tackle the rest of it, or you're gonna overcomplicate things where Python should be doing the work for you

e- lookit all them replies

Hollow Talk
Feb 2, 2014

Meyers-Briggs Testicle posted:

I've never programmed in an enterprise environment but probably want to do that now. Are ETL and Airflow something I can learn shallowly and easily enough to slap on my resume or would someones bullshit meter immediately be flipped

ETL can be many things, depending on source systems, target systems, architectures, etc., so this strongly depends on the place, what they do, and what they understand as such. ETL can be anything from downloading a CSV file and directly copying it into a database to building multi-tiered, highly complicated systems with myriads of transformations, rules, systems, orchestration layers etc.

Airflow is just a tool (and one I'm not too fond of), though at its core, it's basically just an abstraction to write sequences of tasks, express dependencies, and schedule those tasks.

If I see somebody with "ETL" on their resume without accompanying mentions of SQL, dealing with databases, and general programming, I'm going to be very inquisitive, indeed. Ditto if you only list graphical ETL tools (Alteryx, Talend, etc.).

edit: Questions that come to mind rather immediately for me concern database structures, schemas, normalisation, when to prefer ELT, how to orchestrate ETL/ELT jobs etc.

Hollow Talk fucked around with this message at 20:14 on Jun 25, 2019

Mycroft Holmes
Mar 26, 2010

by Azathoth

The Fool posted:

Python code:
for x in range(0,len(a)):
Then x will be a number that automatically increments for the length of a.

Then you can access the first element of your second dimension by a[x][0]

cool, that works. Now, the strings are of variable length. how do i print only the last letter of the string? is there a function for that?

The Fool
Oct 16, 2003


Mycroft Holmes posted:

cool, that works. Now, the strings are of variable length. how do i print only the last letter of the string? is there a function for that?

Someone else answered that already further up.

Mycroft Holmes
Mar 26, 2010

by Azathoth
alright, those two questions are working. Now, the next question is :
6. Ask the user for a letter. Display the number of times the letter appears in the list See the count function from the prior labs.

my code is :
code:
letter = str(input("Please enter a letter"))
print('17' , print(a.count(letter)))
however, it doesn't work. what am i doing wrong?

necrotic
Aug 2, 2005
I owe my brother big time for this!
You're just counting how many strings are literally just the letter entered.

Mycroft Holmes
Mar 26, 2010

by Azathoth

necrotic posted:

You're just counting how many strings are literally just the letter entered.

oh. how do id o this correctly, then? do i do a[x][x] and just increment?
except the strings are of variable length, so that won;t work....

baka kaba
Jul 19, 2003

PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

Use a for loop, look at each name, build up a total

The Fool
Oct 16, 2003


Assuming a is a list of words

Make a counter that starts at 0
Loop through a for each word
Check the letter count of each word
Increment your counter by the number of letters
When the loop is over, print the final count

Mycroft Holmes
Mar 26, 2010

by Azathoth
okay it might help to post the arrays

a = [ "Euclid", "Archimedes", "Newton", "Descartes", "Fermat", "Turing", "Euler", "Einstein", "Boole", "Fibonacci", "Nash"]
vowels = ['A', 'a', 'E', 'e', 'I', 'i', 'O', 'o', 'U','u']

so i need to find each instance of the values in list vowels in list a and then increment a counter.

so if i go
code:
for x in range(0,len(a)):
goddamn it i know what i need to do, i just don't know how to do it.

Mycroft Holmes
Mar 26, 2010

by Azathoth
wait poo poo thats the next question, i'm stuck on the find a number of letters instance

Mycroft Holmes
Mar 26, 2010

by Azathoth
i hate online courses

baka kaba
Jul 19, 2003

PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

Mycroft Holmes posted:

wait poo poo thats the next question, i'm stuck on the find a number of letters instance

nobody just wants to give you the answer but

Python code:
letter = input("Please enter a letter")
total = 0
for name in names:
  total += name.count(letter)

print(total)
you need to understand what's happening there. This is fundamental stuff, which isn't to say "duh it's simple" but it uses some basic ideas, and the later questions build on that. Like you're gonna struggle to do the vowels one if you haven't grasped how to do it for a single letter yet

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Hollow Talk posted:

On top of an annoying day, I've banged my head against this a fair bit today:

How do I go about mocking calls to functions from the requests library (post, get) etc in my unit tests? More specifically, I know how to make it return values (using unittest.mock.patch and its side_effect argument), but requests still tries to connect to the underlying TCP socket etc., which I very much want to avoid. I tried overriding bits and pieces of urllib3, socket, and requests itself, to no avail. I would prefer to this with just pytest/mock, avoiding an external library like requests-mock.

Any ideas?

I'm on my phone so not a lot of details here, but there's nothing special about the requests library. You just have make sure you're patching it in the right place.

If you're importing requests in foo.bar, you do mock.patch("foo.bar.requests").

If that's not where you're having trouble then it's likely with doing the side effects thing. Have you used side effects on anything before? I can't remember the exact syntax right now so I'll have to give you more help on it later.

Mycroft Holmes
Mar 26, 2010

by Azathoth

baka kaba posted:

nobody just wants to give you the answer but

Python code:
letter = input("Please enter a letter")
total = 0
for name in names:
  total += name.count(letter)

print(total)
you need to understand what's happening there. This is fundamental stuff, which isn't to say "duh it's simple" but it uses some basic ideas, and the later questions build on that. Like you're gonna struggle to do the vowels one if you haven't grasped how to do it for a single letter yet

except that answer fails

n/m i'll wait until my teacher gets back to me

baka kaba
Jul 19, 2003

PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

Mycroft Holmes posted:

except that answer fails

n/m i'll wait until my teacher gets back to me

You probably need to convert each name and your input letter to lowercase (or whatever, just so there's no mismatch) because 'e' won't match 'E'

Hollow Talk
Feb 2, 2014

Thermopyle posted:

I'm on my phone so not a lot of details here, but there's nothing special about the requests library. You just have make sure you're patching it in the right place.

If you're importing requests in foo.bar, you do mock.patch("foo.bar.requests").

If that's not where you're having trouble then it's likely with doing the side effects thing. Have you used side effects on anything before? I can't remember the exact syntax right now so I'll have to give you more help on it later.

I managed to figure it out. The "where to patch"-thing was a good hint to go and look at it again, thanks for that. Side effects were working, but I didn't quite patch the right thing, or rather, I didn't patch enough.

This was the solution (this is obviously simplified):

module/submodule.py
Python code:
import requests

def do_the_thing():
	r = requests.post(...)
	r.raise_for_status()
	
	yield from r.json().get('content', [])
tests/test_submodule.py
Python code:
from unittest.mock import patch

import module.submodule

def test_do_the_thing():
	with patch('module.submodule.requests') as requests_mocked:
		requests_mocked.Response.raise_for_status.return_value = None
		requests_mocked.Response.json.side_effect = [{'a': 1}, {'b': 2}]
		requests_mocked.post.return_value = requests_mocked.Response

		result = module.submodule.do_the_thing(...)
		result = list(result)
:suicide:

edit: syntax highlighting :saddowns:

Hollow Talk fucked around with this message at 23:25 on Jun 25, 2019

Wallet
Jun 19, 2006

baka kaba posted:

You probably need to convert each name and your input letter to lowercase (or whatever, just so there's no mismatch) because 'e' won't match 'E'

The list of vowels posted has both capital and lowercase (for whatever reason) so I assume that's not the issue. I'm also not sure I really follow what the question is, exactly, since Mycroft Holmes seems to be talking about two different questions at the same time.

Mycroft Holmes posted:

so i need to find each instance of the values in list vowels in list a and then increment a counter.

so if i go
code:
for x in range(0,len(a)):
goddamn it i know what i need to do, i just don't know how to do it.

There's no reason to use a range here, because you want to iterate through all items in a list and for x in list will do that right out of the box.

Based on the questions you posted before it, this seems to be a lesson about loops, and so I assume they want you to use nested loops to do something like this:

Python code:
names = [ "Euclid", "Archimedes", "Newton", "Descartes", "Fermat", "Turing", "Euler", "Einstein", "Boole", "Fibonacci", "Nash"]
vowels = ['A', 'a', 'E', 'e', 'I', 'i', 'O', 'o', 'U','u']

#loop through vowels
for vowel in vowels:
    # create a count that starts at 0
    count = 0
    # loop through names
    for name in names:
        # increase count by instances of vowel
        count += name.count(vowel)
    print(f'{vowel} = {count}')
Strings are also, themselves, arrays (as someone else mentioned upthread), so you could instead go nuts with something like this:
Python code:
#loop through vowels
for vowel in vowels:
    # create a count that starts at 0
    count = 0
    # loop through names
    for name in names:
        # loop through letters
        for letter in name:
            # add 0 if the current letter is not vowel and 1 if it is
            count += int(letter == vowel)
    print(f'{vowel} = {count}')
If you've decided that you don't like loops anymore you could also just use a single loop:
Python code:
join = ''.join(names)

for vowel in vowels:
    print(f'{vowel} = {join.count(vowel)}')
If you're finding loops inside of loops confusing, using an IDE to step through your code while it's running can help.

Wallet fucked around with this message at 23:22 on Jun 25, 2019

Adbot
ADBOT LOVES YOU

The Fool
Oct 16, 2003


Wallet posted:

The list of vowels posted has both capital and lowercase (for whatever reason) so I assume that's not the issue.

I believe he was counting just 'E' or just 'e', and I assume he needed the count of both.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply