Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

Plasmafountain: Jun 17, 2008

Crossposting here from SAL:

How do I get a function to iterate with while command to keep looping while there are any elements in an array that are NaN?

# ¿ Jun 28, 2015 20:39

Adbot: ADBOT LOVES YOU

# ¿ May 8, 2024 20:48

Plasmafountain: Jun 17, 2008

I've got issues with array slicing and iteration - inadvertently overwriting values that should (?) remain static, but I cant see exactly where I've gone wrong.

Pastebin: http://pastebin.com/7CREYYwX[1]

I'm having an issue with the functions on and after line 162, where I'm trying to fill in the arrays initial_theta (theta_copy for static values) and initial_nu (nu_copy). The values already there are boundary conditions that need to be satisfied; the NaN values are the ones that need to be replaced by calculation.

These two arrays are sliced columnwise (166-169) with two columns of interest - the a column and the b column. The b column is the column that has NaN values being replaced by calculated values by the function. The a column is the column that contains the values that the b column is calculated from.

These two columns get passed to different functions to calculate the values.

If you run the code and compare the output of the function nuandthetacheck with the initial array copies (nu_copy & theta_copy) a couple of things become apparent:

1) Values of theta simply get passed from column to column overwriting the initial boundary conditions.

2) The nu array is filled out entirely while the theta array has missing values in the final column.

I guess really my question is: What have I missed here? Is it array indexing of some kind? As far as I can tell my indexing avoids overwriting the initial given values in each array but obviously this isnt working quite like I intended.

# ¿ Jul 5, 2015 08:11

Plasmafountain: Jun 17, 2008

QuarkJets posted:

It looks like you're dereferencing theta with index i and then setting values using index j. Did you mean to do that?

I take it you're referring to this bit:

code:

def find_theta_and_nu2(theta_values,nu_values):
   
    for i in range(1,mesh_size-1):
        for j in range(1,mesh_size-1):
            theta_a = theta_values[:,i-1]
            theta_b = theta_values[:,i]
            nu_a = nu_values[:,j-1]
            nu_b = nu_values[:,j]

And then later:

code:

  
theta_values[:,j] = intermediate_step_theta
nu_values[:,j] = intermediate_step_nu

I thought in this case the second nested for j still carried the i indexing through - although I think youre right with the second part, theta_values[:,j] should probably be theta_values[:,i].

Hmm. Changed the above to:

code:

def find_theta_and_nu2(theta_values,nu_values):
   
    for i in range(1,mesh_size-1):
	 theta_a = theta_values[:,i-1]
         theta_b = theta_values[:,i]
        for j in range(1,mesh_size-1):
            theta_a = theta_values[:,i-1]
            theta_b = theta_values[:,i]
            nu_a = nu_values[:,j-1]
            nu_b = nu_values[:,j]

code:

  
theta_values[:,i] = intermediate_step_theta
nu_values[:,j] = intermediate_step_nu

And got the same result.

# ¿ Jul 5, 2015 10:22

Plasmafountain: Jun 17, 2008

baka kaba posted:

Tried stepping through it with a debugger? That way you can watch it fill out entries and work out why it's tripping up on some

Ive had a go at doing so with Spyder's inbuilt debug prompt and pdb but I get the feeling that I'm doing that wrong as well - stepping into the function and then trying to proceed from there leaps to the end of the script. :/

# ¿ Jul 5, 2015 15:36

Plasmafountain: Jun 17, 2008

Had a look at Pycharm and I think I'm going to echo Pmchem's opinion - Spyder seems to be better for my kind of thing which is essentially crunching a lot of numbers.

To that end, I discovered my problem by doing a lot of calculations by hand, although I'm not sure how to fix it and I'd really appreciate any pointers in that direction.

http://pastebin.com/rQ4EWyG0

As a brief recap, the two output arrays Theta and Nu have columns that are calculated from their own and the others previous columns as the range is stepped through from 1 - mesh_size from left to right.

For some reason and as far as I can tell, Theta[1,2] which is 0.11475 is the single wrong value, and it is wrong because it is the only time that it performs a calculation where the values for the previous Theta column are not used in the calculation of the current Theta column [:,2]- although they should be as the values for Theta[:,1] were calculated on the previous iteration.

I have a feeling that its something to do with the funky indexing - despite almost every other value in the array calculating fine (apart from those values that depend on the funky one above) the apparent indexing goes beyond the array index being calculated - the range is 1,mesh_size in the primary function and 1,mesh_size-1 in the subsequent functions that actually do most of the work. This should put the array index to five, but that cant be because the highest index in a length 5 array would 4.

Its puzzling and I can't quite figure it out.

# ¿ Jul 6, 2015 00:24

Plasmafountain: Jun 17, 2008

The equations I'm using mean that I can specify values at the boundaries of an array and fill the rest of it in later. The indices 0 and 4 are already specified, its meant to simply calculate the stuff in the middle. This is why the initial row of nu gets its own little function to calculate that first before the rest of it is filled in - the calculation cant propagate from left to right without the edge values.

Its also why Im puzzled about the error/s - the calculation is proceeding for only *some* elements as if the boundary values arent there.

# ¿ Jul 6, 2015 00:53

Plasmafountain: Jun 17, 2008

I found the error, it was in my nu_b function for calculating the top line of nu. It was wrong and threw everything else off simply because there was a positive instead of a negative. :argh:

# ¿ Jul 6, 2015 13:25

Plasmafountain: Jun 17, 2008

EDIT: embarrassingly bad error. :derp:

Plasmafountain fucked around with this message at 14:04 on Jul 8, 2015

# ¿ Jul 8, 2015 10:47

Plasmafountain: Jun 17, 2008

Edit: Derp. Ignore me.

Plasmafountain fucked around with this message at 11:06 on Jul 13, 2015

# ¿ Jul 13, 2015 10:52

Plasmafountain: Jun 17, 2008

Thanks for that, I realised only a couple of minutes before you posted.

Well now I've got an issue that isnt simply down to poor proofreading.

I have two large arrays (25x25) of coordinates - one for x, one for y. So a point is defined by x[i,j], y[i,j]. Matplotlib will gladly draw lines column by column, but I need point-to-point along the lines of x[i,j],y[i,j] - x[i+1,j+1],y[i+1,j+1].

Can this kind of thing simply be done with a for loop or two?

# ¿ Jul 13, 2015 17:56

Plasmafountain: Jun 17, 2008

SurgicalOntologist posted:

This isn't really clear. You can plot multiple lines in the same call, but they will connect by column. If you want them to connect by row, transpose the data. But based on what you posted it looks like you want the lines to connect on the diagonal, which makes little sense.

Thats exactly it. This script I've been writing solves for the method of characteristics, a way of collapsing complicated differential equations to linear ones, resulting in positive characteristic lines (diagonal to upper right) and negative characteristic lines (diagonal to lower right). The coordinate arrays I have essentially mark where these two families cross, but I would like to plot the actual +/- characteristics to display their paths of propagation.

# ¿ Jul 13, 2015 19:34

Plasmafountain: Jun 17, 2008

SurgicalOntologist posted:

Ah, okay. Is it just the main diagonals or all of them? In any case this can probably get you there without a loop: http://docs.scipy.org/doc/numpy/reference/generated/numpy.diagonal.html#numpy.diagonal

E: you might still need a loop if you're doing all the diagonals, but at least you won't need an inner loop to actually construct each diagonal.

I need all of them. I know that I can use the same method for the right to left diagonals by simply flipping the array with np.fliplr, but still not confident about finding all of the diagonals in a nice simple method. I've been dicking around a little bit with it, but Im not sure that I can plot them without creating a mass of new arrays to store each diagonal - I had hoped that I could avoid doing that by doing things inside a function but Im not sure thats the case.

# ¿ Jul 15, 2015 22:38

Plasmafountain: Jun 17, 2008

I've got an idea for a thing that involves a lot of things that I've never used before and thought I would ask for some help.

I'm working at a company that uses a CFD program that largely operates as a black box, not giving any information until the run is finished. After poking through the files that the program generates while it is running, it generates and stores a lot of data in a bunch of Notepad - readable files delimited by column, headers in text and values for various properties of the flow. I think this is ripe for a program/script that reads each file and plots each (or a selection of) parameters as they are generated using Matplotlib. This way we can see if the simulation is worth continuing early on instead of having to wait until the end.

1) The files are not csv files, but .in and .out . Will a module like csvwrite happily take it anyway if its in a clearly delimited format?

2) What else could I do to read this file?

3) I'm used to working with arrays of numbers in numpy but these files will have a couple of header rows (column number, Parameter name, units) and then a list of floats in scientific notation. Is it ok to use text and floats in an array or am I better off using a different way of storing these?

4) The CFD program updates these files continuously - how would I make my script/program check the files for updates? I think its a pretty simple thing to do to simply check the files every ten seconds or so and generate new images by wrapping the entire thing in a timed loop, but I think that computationally instead of reading and plotting values from (number of files) x 15 variables x 40000 iterations every ten seconds its probably easier to simply append the new values to the arrays (or other device) where they are stored.

Any pointers?

# ¿ Aug 28, 2015 11:23

Plasmafountain: Jun 17, 2008

What are people's recommendations for modules to scape information from and then interact with a website? I know beautifulsoup is very popular but theres a lot of others that get mentioned as well.

Application is creating a bot to first scrape tabulated info, then analyse & implement the best moves to make in an online management game.

# ¿ Oct 16, 2015 21:20

Plasmafountain: Jun 17, 2008

Hoping someone can help me out because I cant find anything applicable on stackexchange.

I have a list of numbers that is an imported txt file output in rows/columns from a tabulated data output. The numbers are currently in str form as elements in the array and is ~16000 elements long.

How can I split this list such that:

The values in each row are assigned to one array row;
A new row is created at every 16th element where the table row ends and a new one begins.

I can't quite figure out how to get this, but what I am really after is the ability to look up the data in it with a simple data[row][col] in a later script. I'm really struggling with turning this long list into an organised array in a nice manner.

Plasmafountain fucked around with this message at 13:56 on Feb 3, 2016

# ¿ Feb 3, 2016 13:50

Plasmafountain: Jun 17, 2008

Its a lot of numbers that are an output from a CFD program to generate some graphs and diagnose some issues with funky simulations I've been running at work.

Thats essentially:

Organised = longlist(index i to i +16) for i in range of (0 to the end of the list also in steps of 16)

It works perfectly - I thought I had an issue with trying to come up with a way to turn them from strings to floats, but I've put that in further up the script where the numbers are read from the original reader into the longlist.

I dip my toe in the water every once in a while so my python is very very rusty. Thanks for helping me out!

# ¿ Feb 3, 2016 14:33

Plasmafountain: Jun 17, 2008

Is there a way to use python to open a windows program using command line arguments?

# ¿ Feb 6, 2016 13:13

Plasmafountain: Jun 17, 2008

Trying to carry on with the command line thing I mentioned earlier.

I've got a part of a script where I have:

code:

import subprocess

subprocess.Popen(['D:\\File\\post_convert.exe','convert.in'])
subprocess.Popen(['C:\\Windows\\notepad.exe' , 'D:\\File\\convert.in'])

But for some reason, while the notepad line works fine and I get to check the format passed to post_convert, it doesnt actually execute post_convert.

If I use the windows command line:

code:

D:\File\>post_convert.exe convert.in

Then it runs quite happily. Changing post_convert to postconvert doesnt make a difference.

Does anyone have any idea whats up?

# ¿ Feb 8, 2016 16:13

Plasmafountain: Jun 17, 2008

Im trying to kludge together a queuing system. I have a bunch of dicts that hold various parameters, but what I'd really like to do in this first instance is to sort by priority before checking for the presence of files to see if the task has been run or not.

If Ive got dicts structured like:

code:

simQ['Folders']['Folder1']['Priority']=3
simQ['Folders']['Folder2']['Priority']=1
simQ['Folders']['Folder3']['Priority']=2
simQ['Folders']['Folder4']['Priority']=4

How can I sort by the priority of each folder in folders?

# ¿ Mar 4, 2016 12:15

Plasmafountain: Jun 17, 2008

I know its bad form, but I urgently need a debug since I cant make this equation match the example in the book.

I have an equation:

X4 = ((L2 - b) - SQRT((L2-b)^2 - 4*c*(a-Y2 + L2*X2))) / (2*c)

Its bugging the gently caress out of me since its a fairly straightforward, slightly-modified quadratic equation formula, but my answer doesnt match the result in the book.

code:




implemented in python as:

X2 = 60.48

Y2 = 59.625

a = 22.1852

b = 0.71568

c = 0.0010787

L2 = 1.2187

Xwall = ((lambda2-b)-(np.sqrt((lambda2-b)**2 - (4*c*(a-Y2+lambda2*X2))))/(2*c))

I should be getting Xwall = 63.46, but instead I'm getting anything but.

If anyone could tell me where I'm going wrong here I'd greatly appreciate it.

(PS I know lambda is a reserved term in python programming but its handy to keep track of equation terms in comparison to book formulae).

Plasmafountain fucked around with this message at 23:35 on Jun 27, 2016

# ¿ Jun 27, 2016 23:27

Plasmafountain: Jun 17, 2008

Yes, L2 = lambda2. = 1.2187

# ¿ Jun 27, 2016 23:54

Plasmafountain: Jun 17, 2008

Thanks for the attempts, gents - I was getting values of ~89, ~134 and some others depending on the exact position of parentheses, but nothing approaching the book value.

Ok, I've found a copy of the book on Scribd.

I'm trying to find X4 and Y4 in part E, lines k & l :

Theres a snippet of equivalent Fortran code a couple of pages earlier which form the same thing, here in the line marked 30:

Its this line that I thought I had transcribed into python.

If theres an alternate function for finding X4 and Y4 I'm all ears - I've previously used np.linalg.solve for a system of equations in terms of x and y, but im not sure how to formulate a solution array consisting of y, x and x squared terms.

Plasmafountain fucked around with this message at 09:16 on Jun 28, 2016

# ¿ Jun 28, 2016 08:39

Plasmafountain: Jun 17, 2008

Jose Cuervo posted:

I believe that c = -0.0010787 from what you posted, and this should give you the right answer.

It is, and it does, thankyou.:facepalm:

Is there a way of solving that kind of system to find x and y given x and x^2 terms?

Or will I need to repeat this process with the derivation and rearranging to isolate my term of interest?

# ¿ Jun 28, 2016 17:09

Plasmafountain: Jun 17, 2008

I need some advice on how to structure some arrays of data.

I'm essentially trying to code something akin to a CFD problem. I have a grid of 2D points in X and Y. Each point has a bunch of properties associated with it - velocity in x and y, temperature, pressure, density.

In a lovely attempt from my undergrad days, I've got parallel arrays for each property - one 2D array for x coordinates, another for y coordinates, another for Vx, and so on.

I tried a quick mashup just now where I had a 1D array of these values assigned to one element of a 2D array representing my 2D XY grid, but I couldnt assign this correctly in numpy.

What are my options here for getting this kind of indexing to work instead of having a lot of arrays making GBS threads up the place?

# ¿ Jul 6, 2016 23:21

Plasmafountain: Jun 17, 2008

Nippashish posted:

This sounds like a good format for the type of data you have, especially if you want to do the same computation at every point in space.

I don't understand what

means though. Can you maybe explain what you're trying to do in detail?

OK.

My issue is that in my prototype script from last year is that I have a fuckton of duplicated arrays for each property of fluid in the flow. They're all 2D arrays. They're structured like this:

I've got a nozzle thats separated into three different regions, so three different grids, each grid has something like 10 properties. So I have 3x10x(whatever my grid size is). Error decreases with increasing grid density, so if I have a 100x100 sized grid, I have 300,000 points to calculate for. Most of these calculations have got a lot of steps and iteration to reduce residuals and thats before I start adding even more complex stuff in the latest version thats got lots of integration.

What I'd really like to do is clean these 30 arrays up so I dont have a ton of these separate arrays to try and keep track of, something nice like this:

If I try and create a np array of zeros, then set individual elements to be numbers, thats fine.

If I try and create an np array of zeros, and try and append a 1D np array or normal array, I get an error.

If I try and create a normal array and insert another 1D array into one of its elements, its fine.

Its looking like I can do what I need to do with ordinary arrays - I'm just concerned about the speed of operations and I'm sure there must be a better way.

SurgicalOntologist posted:

Use xarray.

I've had a look at that but I cannot get how it works. Might be because I've got the worst headache known to man, but it doesnt appear all that intuitive. I'll have another look in the morning.

# ¿ Jul 7, 2016 22:33

Plasmafountain: Jun 17, 2008

Ah! I didnt realise that! That looks like it could be pretty useful.

It being ugly and the rememberance problem isnt so much of an issue because its still a marked improvement over what I've got already - and remembering the indexing of what array is which is rather trivial in comparison to the rest of the operations I need to do.

That said, theres not a lot of vectorised operations going on. The calculations propagate from left to right ( [:,0] are generally given as boundary conditions) so really they're mainly there as an ease to plotting the result with matplotlib later for comparison with CFD data from Fluent or Openfoam.

Havent done all that much object oriented stuff - am I right in thinking I do something like:

code:


class Grid:

	 def __init__(self, mesh_size):
        	self.XCoord = np.zeros((mesh_size,mesh_size))
        	self.YCoord = np.zeros((mesh_size,mesh_size))
        	self.Vx = np.zeros((mesh_size,mesh_size))
		self.Vy = np.zeros((mesh_size,mesh_size))
		self.Theta = np.zeros((mesh_size,mesh_size))

And can then do poo poo like:

code:


Grid.XCoord[:,0] = 1

....actually, just trying this out is like a breath of fresh air and my worries about this whole thing have evaporated.

Thanks for putting me on the right track.

# ¿ Jul 7, 2016 23:58

Plasmafountain: Jun 17, 2008

Got a question for you fine folks.

I've got a function where I'm dealing with mach numbers and angles. I need to get the angle of the sum of these mach angles.

Mu (mach angle) = sin^-1 (1/M)

I'm having a hassle as you might expect where M is exactly equal to 1. Or at least, very very close to it.

If I use:

code:


Angle  = np.tan( 0 + np.arcsin(1/1))

I get a stonkingly large number for a gradient angle. This is fine, and to be expected.

However, if I use actual numbers where 0 is a float 0.0, I get a NaN error being returned.

If I use a really long number of 0.9999999999999999999999 then I get a float being returned, but this length is in excess of the normal number of decimal places normally being passed:

code:


np.arcsin(1.0/0.99999999999)
S:/Python/MSC/sauers.py:1: RuntimeWarning: invalid value encountered in arcsin
  # -*- coding: utf-8 -*-
Out[10]: nan
np.arcsin(1.0/0.99999999999)
S:/Python/MSC/sauers.py:1: RuntimeWarning: invalid value encountered in arcsin
  # -*- coding: utf-8 -*-
Out[10]: nan

np.arcsin(1.0/0.999999999999999999999999999999)
Out[11]: 1.5707963267948966

np.tan(0 + np.arcsin(1.0/0.999999999999999999999999999999))
Out[12]: 16331239353195370.0

How can I work around this issue if the number the pre-conditioning function gives is only 0.9999999999999999 instead of its much longer brother?

Additionally, if I add in a line so that:

code:

def lambdaPosCalc(Theta,M):
    
    if M <= 1.0:
        M = 1.0
        
        L = np.tan(Theta + np.arcsin(1/M))
    
        print('L = TAN({0} + Sin^-1 (1/{1})) = {2}'.format(Theta,M,L))
    
        return L

I get some weird thing where the value of the M variable is being changed outside the function when it should be static.

Plasmafountain fucked around with this message at 23:42 on Aug 3, 2016

# ¿ Aug 3, 2016 23:17

Plasmafountain: Jun 17, 2008

Dominoes posted:

Are you sure your formula's right? I don't think an angle can be infinite!

Its a function to create a gradient, in this case a vertical line for arcsin(1), so an infinite gradient is exactly what I'd expect.

Some weird stuff is going on, but im going to call it a night and look at it tomorrow. I get the feeling theres some radian/degrees fuckery going on somewhere in the remainder of the program.

# ¿ Aug 3, 2016 23:56

Adbot: ADBOT LOVES YOU

# ¿ May 8, 2024 20:48

Plasmafountain: Jun 17, 2008

Anyone got any recommendations for a method of creating animations (avi/mpeg/mp4) from a bunch of .png files? I've inherited a collection of code that does this with cv2 (I think) but its awfully put together and throws lots of errors with the l/w/h parameters.

# ¿ Mar 5, 2017 00:30

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.