Go: Google's Plan9 refugees are staging a breakout

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Go: Google's Plan9 refugees are staging a breakout

ErIog: Jul 11, 2001

an skeleton posted:

Why do I have to use CombinedOutput() instead of just Output() to get the output of a script I'm executing from go?

Because the thing you're running outputs to stderr instead of stdout, and so getting the combined stderr/stdout gets the output you expect?

os/exec is pretty solid. I've built a number of research projects around it. I did wrap it to be more convenient for my uses, but I've never found a bug in it despite some code being just numerous wrappers around things called by os/exec.

ErIog fucked around with this message at 13:24 on Aug 22, 2018

# ¿ Aug 22, 2018 13:22

Adbot: ADBOT LOVES YOU

# ¿ May 12, 2024 06:20

ErIog: Jul 11, 2001

I tried to Google this question, but nothing useful comes up.

I'm working on a web server that has a REST-style API. You give it a newline-delimited list of things in a POST body and it vomits back JSON objects separated by newlines.

I thought I could do something like below because hey, it's an io.ReadCloser so why not.

code:

scanner := bufio.NewScanner(request.Body)
for scanner.Scan() {
  line := strings.TrimSpace(scanner.Text())
  .. the rest of my function
}

This led to not all JSON objects requested being returned. When I check the ContentLength of the request it is as expected. The full Body does exist inside the request.

So to see what's happening I moved to bufio.Reader and reader.ReadString('\n'). It turns out the response.Body is being closed somewhere along the way. I'm not closing it anywhere. I grepped all the code and there's not a single Close() anywhere.

My workaround for this was to ioutil.ReadAll the request body to pull the data before the request gets closed by whatever is closing it. The amount of data in the request Body is only like 5 KB. So it's not like there's a memory issue happening and the process is getting killed. However I would like to develop this into more of a streaming style because there's potential in the future for a lot of data to be running through this server.

I'm using Gorilla Mux for URL routing, and then just http.ListenAndServe for the server. Is there some timeout for reading the request.Body or for processing the request that I'm not aware of?

ErIog fucked around with this message at 09:04 on Aug 30, 2018

# ¿ Aug 30, 2018 09:02

ErIog: Jul 11, 2001

After spending way too much time investigating it turns out that net/http will close the Request body if you write to the Response writer, but not immediately.

Vanadium posted:

Do you really care about the newlines or can you just put the body into a json.Decoder directly and keep asking it to decode until .More() or w/e is false?

I'm somewhat wary of doing that because this is going to be a public API used by academic researchers, not professional programmers. I'm worried that people who using random-off-the-shelf-slow JSON parser in their language of choice might not be able to parse the JSON as a stream like that. I don't want to get e-mails saying my API is slow when the problem is they're using a parser that requires there to be a complete root object in memory before it can parse anything.

The number of objects being serialized could be anywhere between 1 and a million. So taking input/output as a stream of newline-separated objects seemed like the better way to go. I'm also supporting XML output, and so I have the same concerns about those parsers too. I'm trying to support the lowest common denominator of parser/encoder.

If someone tells me I'm solving a problem that doesn't exist then I'll knock it off. I just chose this approach based on my limited experience with serialization libraries in different languages. I'll do some testing and see which approach works best.

ErIog fucked around with this message at 02:13 on Sep 1, 2018

# ¿ Aug 31, 2018 04:31

ErIog: Jul 11, 2001

I have a dumb question, but why would you care about the specific pointer to the function? You have tests. Go has idioms for testing you can use to test your stuff. So if those tests are good and they passed, what does testing the pointer get you?

I also have an unrelated super smarty genius question. I want to generate pretty docs for my Go libraries like the standard library has. When I Google I find lots of stuff about running an HTTP server with godoc, and then I find other stuff that's like give us the path to the GitHub and we'll generate docs on the fly. The libraries in question aren't publicly available on Github. They're internal to my workplace, and they will never in a million years be publicly available. Is there something I can do other than launching the godoc HTTP server and spidering with wget?

# ¿ Dec 14, 2018 12:20

ErIog: Jul 11, 2001

InAndOutBrennan posted:

Is there a better or more idiomatic way of accomplishing what I'm after?

I don't know, but it sounds like you're trying to implement something similar to what SQLx does. SQLx allows you to scan SQL query results into structs the same way encoding/json and other things in the standard library allow you to do.

I believe it's handled there using reflection rather than relying solely on empty interfaces, but I haven't source-dived it in a while.

# ¿ Jan 31, 2019 00:33

ErIog: Jul 11, 2001

Hadlock posted:

Best sound library? I see beep, and a few others but nothing is really jumping out at me

I'm trying to build a specific waveform given a digital input

Are you trying to output it to a file or directly to speakers? Encoding LPCM manually isn't that hard, and it looks like beep would be able to handle that.

You might look into building your own Beep.Streamer implementation that plays your own generated tones instead of things from files.

# ¿ Feb 2, 2019 00:49

ErIog: Jul 11, 2001

30.5 Days posted:

go is fuckin good

It is OP, it is. I just doubled my salary (not hard, I was making dog poo poo before, lol) based primarily on me being proficient in Go. I was in academia before doing scaled web services for outside researchers, and I've now gone to the private sector.

The thing that will jokerfy me, though, is that they were real excited to have someone like me on their team during early interviews. Then in final round they dropped the bomb that they don't even use Go, really. They just let the engineers freehand it.

Thankfully my time in academia has prepared me to walk into a mess so I'm kind of excited either way. I like Go because I specifically have a lot of experience with other languages/stacks where things are quite a lot more broken. So it's actually really weird how good their job posting was because it was like exactly suited to someone like me who can come in to clean up messes.

ErIog fucked around with this message at 02:49 on Apr 15, 2022

# ¿ Apr 15, 2022 02:44

ErIog: Jul 11, 2001

Hughmoris posted:

For you Go'ers (or whatever you call yourselves...): do you use Go for any sort of data analytics / data analyst jobs? I keep seeing it pop up on those type of job postings, and can't tell if I'm missing a big chunk of the data sector or if they are just spitballing keywords for the job description.

I wrote many scientific research pipelines in Go at a previous job, a lot of which hinged on analytics for making sure everything was working how it was intended to work. Go was lacking a lot of high level statistical functions at the time, but the other language features made it a very good fit for the job of reliably gathering the information (or erroring properly when the data was busted) to do statistics on. There were some points at the very end of finalizing a paper we'd pass small pieces of the data to R, but if we had ever exposed R to all the data it would have fallen over instantly or never finished the analysis we wanted to do.

So in some ways Go is not the sexiest choice for doing that kind of work, but it is frequently one of the best tools for doing 99% of the job.

cruft posted:

The built in fully capable performant multi-threaded HTTP/2 server is a real selling point for me and the work I do.

I guess everybody already knows Go is good at this, but I feel like I need to jump on the bandwagon here.

I also made a lot of use of this, and I was always expecting to find a point where it fell over or it was somehow worse than the more common stacks. I never did. Every bottleneck I found in stress-testing my endpoints was either in the firewall, Linux itself, or the database/app. It always kept up and worked reliably.

The other benefit I found over other stacks is that layers of the stack just disappear when you use Go because those things are built-in. It becomes so much easier to debug problems.

ErIog fucked around with this message at 03:34 on Apr 17, 2022

# ¿ Apr 17, 2022 03:27

Adbot: ADBOT LOVES YOU

# ¿ May 12, 2024 06:20

ErIog: Jul 11, 2001

Pimblor posted:

I've been writing Go now professionally since 2019. Mostly for boring automation type stuff, but I deeply appreciate being able to write code, have it hermetically sealed and just loving work every where on a fairly large fleet. Rust is there, but it's weird man. I hated Go with a passion when I first came to it, but I'm like a pod person now. I've been writing utility UI's in react served by gin and gorilla and stuffed into lambdas and chuffed is a word I'd use to describe deployment.

The thing I think about constantly is whether or not Go got to where it is because other people were off bike-shedding Rust.

That said, I don't like the changes they made to modules and stuff past a certain point. They're not that hard to work around, though.

But yes, it does do the thing you're saying without the overhead of something like Java. It'll give you a set of swole binaries that never give you dependency issues anywhere.

# ¿ Jun 2, 2023 19:55

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Go: Google's Plan9 refugees are staging a breakout