Ask General Programming Questions Not Worth Their Own Thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »

Literal Hamster: Mar 11, 2012; YOSPOS

Is there any way to asynchronously fetch an arbitrary number of pages of paged data at once when the number of pages may change at any time during the asynchronous operation and therefore cannot be known in advance? Am I required to serially fetch a single page at a time, checking for a next page, in this situation?

# ? Aug 16, 2017 01:37

Adbot: ADBOT LOVES YOU

# ? May 31, 2024 23:29

KernelSlanders: May 27, 2013; Rogue operating systems on occasion spread lies and rumors about me.

Literal Hamster posted:

Is there any way to asynchronously fetch an arbitrary number of pages of paged data at once when the number of pages may change at any time during the asynchronous operation and therefore cannot be known in advance? Am I required to serially fetch a single page at a time, checking for a next page, in this situation?

Any reason not to have a number of workers (say 5) to fetch pages concurrently -- on start up workers would fetch pages 1-5, the first one to come back grabs 6, etc. Then when one finds the last (or rather first missing) page it sets a flag for the others to stop?

# ? Aug 16, 2017 01:47

Literal Hamster: Mar 11, 2012; YOSPOS

KernelSlanders posted:

Any reason not to have a number of workers (say 5) to fetch pages concurrently -- on start up workers would fetch pages 1-5, the first one to come back grabs 6, etc. Then when one finds the last (or rather first missing) page it sets a flag for the others to stop?

That's exactly the solution I was looking for, thanks!

# ? Aug 16, 2017 15:39

KernelSlanders: May 27, 2013; Rogue operating systems on occasion spread lies and rumors about me.

This is possibly more of a software architecture / data engineering than "programming" question, but has anyone dealt with passing data between batch jobs and production systems? Like when your Hadoop job needs to get its final output into your web servers' MySQL or Redis, how did you actually set that up? I've seen some ok approaches and some terrible ones, but nothing I've thought was great. Anyone have something they think worked well?

# ? Aug 17, 2017 03:52

JawnV6: Jul 4, 2004; So hot ...

KernelSlanders posted:

This is possibly more of a software architecture / data engineering than "programming" question, but has anyone dealt with passing data between batch jobs and production systems? Like when your Hadoop job needs to get its final output into your web servers' MySQL or Redis, how did you actually set that up? I've seen some ok approaches and some terrible ones, but nothing I've thought was great. Anyone have something they think worked well?

Why's the data have to move. Make the compute the thing that moves.

# ? Aug 17, 2017 04:23

KernelSlanders: May 27, 2013; Rogue operating systems on occasion spread lies and rumors about me.

JawnV6 posted:

Why's the data have to move. Make the compute the thing that moves.

Yes, I agree Spark is better than Hadoop if that's what you're getting at, but it doesn't really address my question. There are batch processes (like Spark/Hadoop jobs that run once a day) and there are online jobs (like serving web pages when requested by a browser). How do you get data from one process to the other?

e: generally, also these processes are developed and maintained by different teams

KernelSlanders fucked around with this message at 06:00 on Aug 17, 2017

# ? Aug 17, 2017 05:58

Steve French: Sep 8, 2003

The general pattern we tend to follow is instead of having the batch process write back to some other component's data store, it writes to its own data store and provides some way for the other component to read from it as necessary. In our stack, this is usually a Spark job writing to Cassandra, with a Thrift service providing read access to the data as appropriate to the production serving front end component. Not the only way to do it by any means, but it has worked alright for us.

# ? Aug 17, 2017 06:11

dougdrums: Feb 25, 2005; CLIENT REQUESTED ELECTRONIC FUNDING RECEIPT (FUNDS NOW)

JawnV6 posted:

Why's the data have to move. Make the compute the thing that moves.

KernelSlanders posted:

Yes, I agree Spark is better than Hadoop if that's what you're getting at, ...

I only know about 1/4 of these proper nouns but I think the point is to do everything lazily.

It's hard to say with out knowing things like how long a usual job takes, or what sort of reductions need to be done for the view. But, I think a good solution is to give the batch system a little status database that references the jobs (if one of these frameworks does not already provide one), and an API for queries on the job status and result. Wrap your batch system up tight in this API.

So, the batch data and results stays in the batch system, and you query that system from the app. If the app must wait for the job to complete (or rather, the job takes < min or two), allow the batch system to push the result back by sending continuation code/query for when the job is finished.

I built a system like this but it used GNU parallel and jobs took hours, as the task at hand involved collecting sensor data online. I'm not sure what the differences might be. GNU parallel is pretty good though.

I guess the more general point is to allow each system/team to be their own little king of data, and make sure that the teams design and document good interfaces to query that data.

dougdrums fucked around with this message at 13:13 on Aug 17, 2017

# ? Aug 17, 2017 13:03

KernelSlanders: May 27, 2013; Rogue operating systems on occasion spread lies and rumors about me.

dougdrums posted:

I only know about 1/4 of these proper nouns but I think the point is to do everything lazily.

It's hard to say with out knowing things like how long a usual job takes, or what sort of reductions need to be done for the view. But, I think a good solution is to give the batch system a little status database that references the jobs (if one of these frameworks does not already provide one), and an API for queries on the job status and result. Wrap your batch system up tight in this API.

So, the batch data and results stays in the batch system, and you query that system from the app. If the app must wait for the job to complete (or rather, the job takes < min or two), allow the batch system to push the result back by sending continuation code/query for when the job is finished.

I built a system like this but it used GNU parallel and jobs took hours, as the task at hand involved collecting sensor data online. I'm not sure what the differences might be. GNU parallel is pretty good though.

I guess the more general point is to allow each system/team to be their own little king of data, and make sure that the teams design and document good interfaces to query that data.

Interesting. I'm thinking mostly about use cases with very long running batch jobs. For example, you have 10M customers and 100k SKUs. Your recommendations job takes around an hour to run, but when a user loads the home page it needs to display the output fast enough that it's the user sees no delay. Typically I've done this by having the recommendations team (who typically have no interest in maintaining a server with any uptime requirements, on call schedules, etc.) write their output to S3 and then hit an internal web endpoint that triggers something maintained by the platform team to load all those files into either redis or the primary db. There's a number of problems with this, but it also gives a very clear separation of ownership between the two teams.

# ? Aug 18, 2017 02:39

White Light: Dec 19, 2012

Ran into an issue when working on this game of mine, this time withmultiple enemies!

Trying to get more than one enemy to show up on my screen browser, but despite typing in the proper array code data, only one enemy shows up.

Here is my code:

code:

<html>
<head>
	<title></title>
</head>
<body>
	<style type="text/css">
		#ocean {
			background-image: url("img/ocean.jpg");
			width: 900px;
			height: 700px;
		}
		.player {
			position: absolute;
			background-image: url("img/player.png");
			width: 70px;
			height: 75px;
		}
		.enemy {
			position: absolute;
			background-image: url("img/enemy.png");
			width: 70px;
			height: 75px;
			left: 300px;
			top: 300px;
		}
	</style>
	<div id="ocean">
		<div id="players"></div>
		<div id="enemies"></div>
	</div>

	<script type="text/javascript">
		
		var player = {
			left: 450,
			top: 620
		}

		var enemies = [
			{left: 350, top: 200},
			{left: 450, top: 250},
			{left: 550, top: 150},
			{left: 250, top: 350}
		]

		function drawPlayer(){
			content = "<div class='player' style='left:"+player.left+"px; top:"+player.top+"px'></div>"
			document.getElementById("players").innerHTML = content;
		}
		drawPlayer();

		function drawEnemies(){
			content = "";
			for(var idx=0; idx<enemies.length; idx++){
				content = "<div class='enemy' style='left:"+enemies[idx].left+"px; top:"+enemies[idx].top+"px'></div>"
			}
			document.getElementById("enemies").innerHTML = content;
		}
		drawEnemies();

		document.onkeydown = function(e) {
			console.log(e);
			if(e.keyCode == 37){ // LEFT
				player.left = player.left - 10;
			}
			if(e.keyCode == 39){ // RIGHT
				player.left = player.left + 10;
			}
			if(e.keyCode == 38 && player.top > 500){ // UP
				player.top = player.top - 10;
			}
			if(e.keyCode == 40 && player.top < 625){ // DOWN
				player.top = player.top + 10;
			}
			drawPlayer();
		}

	</script>
</body>
</html>

I'm sure it's something rather simple to fix, but the goofball in the tutorial never post any reference code so it always feels like i'm shooting in the dark over here. Can anyone take a crack at this to see what i'm missing?

# ? Aug 20, 2017 16:43

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

What do you think this block is doing?

code:

			content = "";
			for(var idx=0; idx<enemies.length; idx++){
				content = "<div class='enemy' style='left:"+enemies[idx].left+"px; top:"+enemies[idx].top+"px'></div>"
			}
			document.getElementById("enemies").innerHTML = content;

What happens on each iteration of the for loop?

# ? Aug 20, 2017 16:58

carry on then: Jul 10, 2010; by VideoGames
(and can't post for 10 years!)

Parrotine posted:

code:

<html>
<head>
	<title></title>
</head>
<body>
	<style type="text/css">
		#ocean {
			background-image: url("img/ocean.jpg");
			width: 900px;
			height: 700px;
		}
		.player {
			position: absolute;
			background-image: url("img/player.png");
			width: 70px;
			height: 75px;
		}
		.enemy {
			position: absolute;
			background-image: url("img/enemy.png");
			width: 70px;
			height: 75px;
			left: 300px;
			top: 300px;
		}
	</style>
	<div id="ocean">
		<div id="players"></div>
		<div id="enemies"></div>
	</div>

	<script type="text/javascript">
		
		var player = {
			left: 450,
			top: 620
		}

		var enemies = [
			{left: 350, top: 200},
			{left: 450, top: 250},
			{left: 550, top: 150},
			{left: 250, top: 350}
		]

		function drawPlayer(){
			content = "<div class='player' style='left:"+player.left+"px; top:"+player.top+"px'></div>"
			document.getElementById("players").innerHTML = content;
		}
		drawPlayer();

		function drawEnemies(){
			content = "";
			for(var idx=0; idx<enemies.length; idx++){
				content = "<div class='enemy' style='left:"+enemies[idx].left+"px; top:"+enemies[idx].top+"px'></div>"
			}
			document.getElementById("enemies").innerHTML = content;
		}
		drawEnemies();

		document.onkeydown = function(e) {
			console.log(e);
			if(e.keyCode == 37){ // LEFT
				player.left = player.left - 10;
			}
			if(e.keyCode == 39){ // RIGHT
				player.left = player.left + 10;
			}
			if(e.keyCode == 38 && player.top > 500){ // UP
				player.top = player.top - 10;
			}
			if(e.keyCode == 40 && player.top < 625){ // DOWN
				player.top = player.top + 10;
			}
			drawPlayer();
		}

	</script>
</body>
</html>

It looks like here:

code:

for(var idx=0; idx<enemies.length; idx++){
				content = "<div class='enemy' style='left:"+enemies[idx].left+"px; top:"+enemies[idx].top+"px'></div>"
			}
			document.getElementById("enemies").innerHTML = content;

You're overwriting content on each iteration, so when the loop is finished, it only holds the div for the last enemy in the array. I think you mean to concatenate onto content each time, using +=, instead of assignment, =.

# ? Aug 20, 2017 16:58

White Light: Dec 19, 2012

EDIT: Wait nevermind, it turns out it was option B! Big thanks fellow goons :cheers:

It's crazy how a single character like a plus sign can be the cause of hours of frustration, but i'm sure this is preaching to the choir with y'all

White Light fucked around with this message at 17:56 on Aug 20, 2017

# ? Aug 20, 2017 17:46

Morroque: Mar 6, 2013

I have a web design client that wants to run a customized online store merged into the web design itself. It is my first time making a website capable of point-of-sale -- it's something I've always had on the backburner as a thing to learn how to do, but there was never quite an opportunity for it. I've had to program various inventory-management-systems in the past, but the actual component where you can charge money for something was never part of the spec until now.

Thus far I've studied a bit with the PayPal API and had a cursory look at various third-party things like Shopify and such, but I still feel a little out of my element here. I know I'm going to need an SSL-secured private server setup, but I don't know anything beyond that. My Google-fu for finding a resource on best practices for e-commerce web programming hasn't really given me anything particularly revelatory or trustworthy-looking beyond an overbroad PCI Compliance Guide, which does sound useful but might be overkill for the stage I'm currently at. Does anyone have any resources they can recommend on e-commerce programming/web-design best practices?

I'm asking because even if I go with a third-party solution like Shopify for now, I can't shake the feeling that this is something I'm going to need to expand on.

# ? Aug 23, 2017 01:18

reversefungi: Nov 27, 2003; Master of the high hat!

Morroque posted:

I have a web design client that wants to run a customized online store merged into the web design itself. It is my first time making a website capable of point-of-sale -- it's something I've always had on the backburner as a thing to learn how to do, but there was never quite an opportunity for it. I've had to program various inventory-management-systems in the past, but the actual component where you can charge money for something was never part of the spec until now.

Thus far I've studied a bit with the PayPal API and had a cursory look at various third-party things like Shopify and such, but I still feel a little out of my element here. I know I'm going to need an SSL-secured private server setup, but I don't know anything beyond that. My Google-fu for finding a resource on best practices for e-commerce web programming hasn't really given me anything particularly revelatory or trustworthy-looking beyond an overbroad PCI Compliance Guide, which does sound useful but might be overkill for the stage I'm currently at. Does anyone have any resources they can recommend on e-commerce programming/web-design best practices?

I'm asking because even if I go with a third-party solution like Shopify for now, I can't shake the feeling that this is something I'm going to need to expand on.

In terms of actually charging people, Stripe's API is incredibly straightforward to use. You hash a users payment information with a one-way public key, use the secret on the back end, and then, depending on the framework of your choice, there's usually a library that handles actually creating a charge and then returns the details of the transaction to save to your DB. Might not be a bad idea to start with Stripe and their documentation and see if that's helpful.

On the front end they have widgets and scripts for everything to make things super easy. If you're using React, there are some library components that are very easy to use as well.

reversefungi fucked around with this message at 01:34 on Aug 23, 2017

# ? Aug 23, 2017 01:25

The Fool: Oct 16, 2003

Morroque posted:

I have a web design client that wants to run a customized online store merged into the web design itself. It is my first time making a website capable of point-of-sale -- it's something I've always had on the backburner as a thing to learn how to do, but there was never quite an opportunity for it. I've had to program various inventory-management-systems in the past, but the actual component where you can charge money for something was never part of the spec until now.

Thus far I've studied a bit with the PayPal API and had a cursory look at various third-party things like Shopify and such, but I still feel a little out of my element here. I know I'm going to need an SSL-secured private server setup, but I don't know anything beyond that. My Google-fu for finding a resource on best practices for e-commerce web programming hasn't really given me anything particularly revelatory or trustworthy-looking beyond an overbroad PCI Compliance Guide, which does sound useful but might be overkill for the stage I'm currently at. Does anyone have any resources they can recommend on e-commerce programming/web-design best practices?

I'm asking because even if I go with a third-party solution like Shopify for now, I can't shake the feeling that this is something I'm going to need to expand on.

Check out stripe.com

I found their documentation to be really easy to follow, and their Checkout component lets you just embed a form, pass it the total, and they take care of the rest.

# ? Aug 23, 2017 01:25

pokeyman: Nov 26, 2006; That elephant ate my entire platoon.

Another +1 for Stripe.

# ? Aug 23, 2017 03:02

OneEightHundred: Feb 28, 2008; Soon, we will be unstoppable!

Wondering if anyone has any thoughts on a compiler problem: Is it possible to statically determine if a .NET generic requires infinite expansion?

i.e.

code:

public class SomeClass<T>
{
   public static void DoSomething()
   {
       SomeClass<SomeClass<T>>.DoSomething()
   }
}

That's valid, but can't be compiled offline because it requires an infinite number of classes.

I feel like there's either a simple solution to this, or it's unsolvable, but can't quite figure it out.

# ? Aug 23, 2017 04:19

sarehu: Apr 20, 2007; (call/cc call/cc)

C++ handles it by having a recursion limit. In general whether you can solve this problem depends on the type system. (In C++ you can't because template expansion is Turing-complete. But with something simpler, you could solve it.)

# ? Aug 23, 2017 07:12

sarehu: Apr 20, 2007; (call/cc call/cc)

If my understanding is correct, in C# a generic type constraint using 'where' can't be used to pick alternate types -- so I guess it won't have an effect on the problem.

It is doable. Basically if Foo<T> invokes Bar<BiggerExpressionContaining<T>>, then declare the ordering "Foo < Bar". If doing this transitively leads to "Foo < Foo", then bingo. (In other words look for a cycle in the directed graph you create.)

Edit: Also it's cleaner if you think about the problem not in terms of SomeClass<T> needing to use SomeClass<SomeClass<T>>, but instead needing to use SomeClass<Succ<T>>.

Edit: Instead of "Foo < Bar" it's "(Foo, 1) < (Bar, 1)" because you can have multiple generic parameters.

sarehu fucked around with this message at 11:38 on Aug 23, 2017

# ? Aug 23, 2017 09:48

UWBW: Aug 3, 2013; Permanently banned from the Alamo

I used to code some Java back in college and working with Game Maker again has got me in the mood for real code once again. Unfortunately, the computers at work (where I would have plenty of time to hash out code) don't allow the installation of the JDK. In fact, I can't install any .exe files at all. Any workaround do this?

# ? Aug 23, 2017 21:06

reversefungi: Nov 27, 2003; Master of the high hat!

UWBW posted:

I used to code some Java back in college and working with Game Maker again has got me in the mood for real code once again. Unfortunately, the computers at work (where I would have plenty of time to hash out code) don't allow the installation of the JDK. In fact, I can't install any .exe files at all. Any workaround do this?

Have no experience with Java, but would a cloud editor like Cloud9 or Codenvy be helpful at all? You can set up an entire dev environment on these tools and access everything from the browser, as far as I know.

# ? Aug 23, 2017 21:34

UWBW: Aug 3, 2013; Permanently banned from the Alamo

The Dark Wind posted:

Have no experience with Java, but would a cloud editor like Cloud9 or Codenvy be helpful at all? You can set up an entire dev environment on these tools and access everything from the browser, as far as I know.

Codenvy looks to be just what I was hoping for! Cloud9 asks me for my credit card, so I'll take a hard pass on that.

# ? Aug 23, 2017 22:53

OneEightHundred: Feb 28, 2008; Soon, we will be unstoppable!

sarehu posted:

If my understanding is correct, in C# a generic type constraint using 'where' can't be used to pick alternate types -- so I guess it won't have an effect on the problem.

It is doable. Basically if Foo<T> invokes Bar<BiggerExpressionContaining<T>>, then declare the ordering "Foo < Bar". If doing this transitively leads to "Foo < Foo", then bingo. (In other words look for a cycle in the directed graph you create.)

Edit: Also it's cleaner if you think about the problem not in terms of SomeClass<T> needing to use SomeClass<SomeClass<T>>, but instead needing to use SomeClass<Succ<T>>.

Edit: Instead of "Foo < Bar" it's "(Foo, 1) < (Bar, 1)" because you can have multiple generic parameters.

Yeah, .NET generics are designed so that the class is complete before substitution, and anything that would prevent that (like inheriting from a generic parameter) is illegal.

The problem I'm having is just figuring out what the actual cycle detection would look. There are cases where nesting isn't cyclic (i.e. SomeClass<T> creating SomeClass<SomeClass<int>>), and I think permutations are always terminal if they don't nest (i.e. SomeClass<A, B> creating SomeClass<B, A>).

I think the rule would be something like, if Foo<T> leads to the creation of any pre-substitution type which contains a nested Foo, and the inner Foo uses a generic parameter, then it expands infinitely, and if it doesn't do that, it will eventually terminate, but not really sure if that's true or how to prove it.

OneEightHundred fucked around with this message at 04:28 on Aug 24, 2017

# ? Aug 24, 2017 04:16

Jabor: Jul 16, 2010; #1 Loser at SpaceChem

What about cases like Foo<T> creating Bar<Foo<T>>?

Or Bar<Bar<Foo<Foo<T>>>>?

# ? Aug 24, 2017 05:17

sarehu: Apr 20, 2007; (call/cc call/cc)

I don't know what those words mean. What you want afaict is, if Foo<A_1, A_2, ..., A_n> uses the type Bar<E_1, E_2, ..., E_m>, where A_1 through A_n are type variables and E_1 through E_m are type expressions, and if the expression E_j uses the type variable A_i but the entire expression is not solely A_i, we construct the directed graph edge (or less-than relation) (Foo, i) < (Bar, j).

If the program's graph edges form a cycle we have a problem.

quote:

SomeClass<T> creating SomeClass<SomeClass<int>>

In that case, the expression SomeClass<int> does not contain T, which means the edge (SomeClass, 1) < (SomeClass, 1) is not present.

quote:

I think permutations are always terminal if they don't nest (i.e. SomeClass<A, B> creating SomeClass<B, A>).

In that case, we have (A_1, A_2) = (A, B), and (E_1, E_2) = (B, A). Because the expression E_1 solely consists of A_2, the edge (SomeClass, 2) < (SomeClass, 1) is not present. And because E_1 does not contain A_1 at all, the edge (SomeClass, 1) < (SomeClass, 1) is also not present. Likewise the other two possible edges aren't present.

On the other hand, suppose that Foo<A, B> created Foo<Succ<B>, A>. Then we have (Foo, 2) < (Foo, 1). Hmm. We need more, because that'll expand into an infinite chain of type definitions.

What we also need is: If the expression E_j consists solely of the type variable A_i, then we have an equality relation. We want to combine the graph nodes (Foo, i) and (Bar, j). Or we can create two directed edges of weight zero (the less-than edges being weight 1) and make the graph problem being that of finding a cycle containing a non-zero edge.

Edit:

Jabor posted:

What about cases like Foo<T> creating Bar<Foo<T>>?

That's perfectly fine -- (Foo, 1) < (Bar, 1) and that's that. (If Bar<T> creates Foo<Succ<T>> then we might be in trouble. But if Bar<T> is just a List<T> it's very reasonable.)

quote:

Or Bar<Bar<Foo<Foo<T>>>>?

Then Foo<T> creates a Foo<Foo<T>> which means trouble. We've got the edge (Foo, 1) < (Foo, 1), which is a cycle.

Also we've got the edge (Foo, 1) < (Bar, 1).

sarehu fucked around with this message at 05:27 on Aug 24, 2017

# ? Aug 24, 2017 05:19

Argue: Sep 29, 2005; I represent the Philippines

Anyone ever done any work trying to reverse engineer the formats for Photoshop's tool presets or Manga Studio's brushes? I want to write a converter but neither of them seem to be documented, either officially or unofficially.

# ? Aug 24, 2017 07:16

ufarn: May 30, 2009

My CPU is ancient, so I can't do a lot of rendering and compute.

If I just want to run a shell script to generate a graphic (probably just CPU, not GPU), what's the most straightforward way to just set up a virtual instance with decent specs somewhere where you can be in and out in ten minutes without having to worry about some crap that's still running and charging you money like AWS EC2 does?

I don't have any experience with VM and SSH tunnelling, just need something as simple as possible. Whether it costs $1 or $3 an hour I couldn't care less about. I wish I could just use a Windows service, but I really like my Ubuntu Bash on 10, and most Windows images don't have it.

# ? Aug 25, 2017 11:13

leper khan: Dec 28, 2010; Honest to god thinks Half Life 2 is a bad game. But at least he likes Monster Hunter.

ufarn posted:

My CPU is ancient, so I can't do a lot of rendering and compute.

If I just want to run a shell script to generate a graphic (probably just CPU, not GPU), what's the most straightforward way to just set up a virtual instance with decent specs somewhere where you can be in and out in ten minutes without having to worry about some crap that's still running and charging you money like AWS EC2 does?

I don't have any experience with VM and SSH tunnelling, just need something as simple as possible. Whether it costs $1 or $3 an hour I couldn't care less about. I wish I could just use a Windows service, but I really like my Ubuntu Bash on 10, and most Windows images don't have it.

Have you looked at azure?

# ? Aug 25, 2017 13:21

ufarn: May 30, 2009

leper khan posted:

Have you looked at azure?

There are (fortunately) a billion cloud services out there, but between Google, MS, Amazon, and all the microservices out there, I thought I'd just hit up someone who's sampled them all and maybe use a similar workflow.

# ? Aug 25, 2017 13:35

Hughmoris: Apr 21, 2007; Let's go to the abyss!

Could someone recommend me some books/blogs/videos that cover Data Visualization 101? I'm a clinician that is in an analyst role right now and I'm starting to realize that my charts and presentation tools look like garbage. I would really like clean/neat views of my data to present to others for decision making.

I have a baseline working knowledge of Python + Pandas, and Excel.

# ? Aug 26, 2017 03:43

lifg: Dec 4, 2000; <this tag left blank>; Muldoon

Hughmoris posted:

Could someone recommend me some books/blogs/videos that cover Data Visualization 101? I'm a clinician that is in an analyst role right now and I'm starting to realize that my charts and presentation tools look like garbage. I would really like clean/neat views of my data to present to others for decision making.

I have a baseline working knowledge of Python + Pandas, and Excel.

Read this and do the opposite: https://forums.somethingawful.com/showthread.php?threadid=3745614

# ? Aug 26, 2017 03:53

ufarn: May 30, 2009

Hughmoris posted:

Could someone recommend me some books/blogs/videos that cover Data Visualization 101? I'm a clinician that is in an analyst role right now and I'm starting to realize that my charts and presentation tools look like garbage. I would really like clean/neat views of my data to present to others for decision making.

I have a baseline working knowledge of Python + Pandas, and Excel.

You're going to end up using R with ggplot(2) sooner or later, so whichever covers those for you the best.

At the end of the day, R is the go-to, and nothing comes close to ggplot. Not for lack of trying.

I've generally avoided R because I can't stand the editors and interface and prefer to just build it in Sublime Text, but I'll have to do it eventually, even though the workflow feels a little clunky still.

e: Looks like the creator published a somewhat recent book about it:

- https://www.amazon.com/Data-Science-Transform-Visualize-Model/dp/1491910399/ref=asap_bc?ie=UTF8
- And another: https://www.amazon.com/ggplot2-Elegant-Graphics-Data-Analysis/dp/331924275X/ref=asap_bc?ie=UTF8

I'd be careful reading something older than 2016 - maybe even 2017.

e: Looks like it's available for free online: http://r4ds.had.co.nz/

ufarn fucked around with this message at 11:07 on Aug 26, 2017

# ? Aug 26, 2017 10:58

du -hast: Mar 12, 2003; BEHEAD THOSE WHO INSULT GENTOO

Hello:

I'm very bad at Python and can't figure out the proper way to do something even though I am sure that it's obvious.

Let me know if I should ask this in the python thread instead.

I have written a script that connects to a router and feeds ips into it via telnet.

I've set up a contraption that takes a regular IP series, IE 2.3.4.128/29, splits the /29 off and turns it into a subnet, and then increments the last octet to .129. But it's rigged in an ugly and definitely wrong way:

code:

 
# example ip: 2.3.4.16/29

    block = ip[:-3] #block with the subnet missing
    subnet = ip[-3:] #the subnet
    ender = block[-1:] #the trailing integer on the block to be increased
    

.....

    ip = router_ip_output = 'ip address ' + block + ender + ' ' + subnet

Then an ugly if loop that changes /29 to 255.255.255.248

so that when I input say 2.3.4.16/29, it spits out 2.3.4.17 255.255.255.248 .

This works fine. However, now I am trying to set it up so that I can parse several lines all at once. From the following:

1.1.1.0/29 (goatse) 707 8 n/a Nov-30-15-1 Edit Del
2.2.2.0/29 (goatse) 707 8 n/a Nov-30-15-2 Edit Del
69.42.0.0/28 (goatse) 707 16 n/a Nov-30-15-3 Edit Del
80.08.5.0/28 (goatse) 707 16 n/a Nov-30-15-4 Edit Del
14.88.69.16/28 (goatse) 707 16 n/a Nov-30-15-5 Edit Del
2.3.4.60/28 (goatse) 707 16 n/a Nov-30-15-6 Edit Del

I need it to spit out the same idea as aboe:

1.1.1.1 255.255.255.248
2.2.2.1 255.255.255.248
69.42.0.1 255.255.255.240
80.08.5.1 255.255.255.240
14.88.69.17 255.255.255.240
2.3.4.61 255.255.255.240

I need to remove all the trailing crap - (goatse) 707 ... ...

I am sure there is a proper way to do this but I am really bad at this and could use some help. As it is currently it is fine - I have it set to output an IP block in router format, but I now need to do it for a ton of IPs rather than a single raw_input for each IP.

I'm pretty sure I need to strip these in an array, then print it into the output incrementally with a loop, but I don't know how to break the trailing stuff off of the block to do this safely.

Thanks for looking. Also this router is not live so I am not destroying anything with my retardation. I am using Python 2.7.6

# ? Aug 26, 2017 11:40

ufarn: May 30, 2009

You're probably better off using a regex to match it with. I've created a simple one (not Python-specific) for the whole IP; you can then either slice the match (use split("/")[1] to get the last number) or use a regex capture "( ) around the part you want" and work on that.

I haven't use Python and regex in a bit, but I think you can just use .group() or .groups() to easily access your captures. I've updated the example so it now captures each group of numbers for you to just go through depending on which group you want to change. Just remember to change the string to an int, ofc.

ufarn fucked around with this message at 12:18 on Aug 26, 2017

# ? Aug 26, 2017 12:15

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

You don't need a regex for this; you can accomplish it using the string.split function. For example, "1.1.1.1/29".split("/") yields ["1.1.1.1", "29"].

I mean, you can also use regexes, but if you're having trouble with basic string manipulation, you're probably better off avoiding regexes. They have a tendency to make your life more complicated rather than less unless you know what you're doing, and they're irritating to debug.

# ? Aug 26, 2017 14:57

du -hast: Mar 12, 2003; BEHEAD THOSE WHO INSULT GENTOO

TooMuchAbstraction posted:

You don't need a regex for this; you can accomplish it using the string.split function. For example, "1.1.1.1/29".split("/") yields ["1.1.1.1", "29"].

I mean, you can also use regexes, but if you're having trouble with basic string manipulation, you're probably better off avoiding regexes. They have a tendency to make your life more complicated rather than less unless you know what you're doing, and they're irritating to debug.

Thank god, a way to do it without regex :cheers:

. I've used this method to break up these addresses before, and it's pretty easy to turn 2.2.2.2/29 into 2.2.2.3 255.255.255.248 with it (though kinda clunky, works like a charm).

My issue is that I need to be able to parse all of these multiple lines at once. The overall goal is to be able to copy and paste the above bullshit into a terminal and have it put out all 5 at once - while .split() works individually I don't understand well enough how to take the full input (all the lines) then break off the ip/subnet mask, and stick that into an array or something.

The end goal is to be able to print something them all out at once, fully formatted, something like this poo poo pseudo-code:

x = 0
print ip_addr[x]
x++

and be able to print them, in order, formatted if that makes sense. I know there is some easier way to do this buy my mind deserts me. The reason for all of this by the was is because sometimes I have hundreds of /29s that have to be put into a router, which leads to a lot of human error

Thank you very much for all your help so far

# ? Aug 26, 2017 15:33

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

You could use something like str.splitlines() to... you know... and then just iterate over those with your transforming function

[hack_ip(line) for line in input.splitlines()] would give you a list of your new versions, or you could do a for loop with your formatting/printing code in there, whichever you're into

# ? Aug 26, 2017 16:02

mystes: May 31, 2006

You want to do something like this:

(using netaddr)

code:

#!/usr/bin/python3
import netaddr
from sys import stdin

for input_line in stdin:
    cidr = input_line.split()[0]
    network = netaddr.IPNetwork(cidr)
    first_addr = netaddr.IPAddress(network.first+1) # adding 1 here to end in  X.X.X.1 as in the example output
    mask = netaddr.IPAddress(network.netmask)
    print(str(first_addr) + " " + str(mask))

Then you can do:

code:

user@computer:~$ cat << EOF | ./cidr.py
1.1.1.0/29 (goatse) 7078n/aNov-30-15-1Edit Del
2.2.2.0/29 (goatse) 7078n/aNov-30-15-2Edit Del
69.42.0.0/28 (goatse) 70716n/aNov-30-15-3Edit Del
80.08.5.0/28 (goatse) 70716n/aNov-30-15-4Edit Del
14.88.69.16/28 (goatse) 70716n/aNov-30-15-5Edit Del
2.3.4.60/28 (goatse) 70716n/aNov-30-15-6Edit Del
EOF

output:

code:

1.1.1.1 255.255.255.248
2.2.2.1 255.255.255.248
69.42.0.1 255.255.255.240
80.8.5.1 255.255.255.240
14.88.69.17 255.255.255.240
2.3.4.49 255.255.255.240

mystes fucked around with this message at 16:42 on Aug 26, 2017

# ? Aug 26, 2017 16:38

Adbot: ADBOT LOVES YOU

# ? May 31, 2024 23:29

nielsm: Jun 1, 2009

mystes posted:

You want to do something like this:

(using netaddr)

Yes this. Use a library someone else wrote for you if you can. It generally means less risk of mistakes on your end and gets you clearer and more robust code.
In this case it handles all the subnet calculation for you in the correct way with bitmasks etc., instead of the not-really-proper stuff you were doing. E.g. your original method wouldn't be able to handle subnets larger than /24, and would also produce wrong results if the input for some reason specified any other address than the first one in the subnet.

# ? Aug 26, 2017 16:50

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »