Go: Google's Plan9 refugees are staging a breakout

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Go: Google's Plan9 refugees are staging a breakout

Blame Pyrrhus: May 6, 2003; Me reaping: Well this fucking sucks. What the fuck.; Pillbug

Dumb question about dealing with json in GO.

In powershell, let's pretend the json body I am working with is:

code:

{
    "value": "thing",
    "number": 3
}

In powershell I can store that json body as say, $data and then simply:

code:

$array = $data | convertFrom-Json

and BAM, I have an object where I can say, express the value of number via $array.number

Similarly in python I can store that json statement as data and then:

code:

array = json.loads(data)

and BAM, I have an object I can do stuff with.

How can I do this same thing in Go?

I've googled and sure, I can declare a struct, but I'm writing this to service API calls to dozens of different endpoints, all with different json responses, I shouldn't need to define dozens of structs as well. Googling how to do this more dynamically leads me to use map[string]interface{} which just pollutes the data. with "map" prefixes and brackets all over the place.

This seems like such a basic thing, I'm shocked that I can't more easily do this.

Blame Pyrrhus fucked around with this message at 08:52 on Jul 9, 2020

# ¿ Jul 9, 2020 04:50

Adbot: ADBOT LOVES YOU

# ¿ May 11, 2024 06:55

Blame Pyrrhus: May 6, 2003; Me reaping: Well this fucking sucks. What the fuck.; Pillbug

geonetix posted:

There's typicall two schools of thought about the json stuff in go as far as I've seen: declare your expected responses explicitely and unmarshal into that, or use the map method.

I'm a strong believer of the first method with predefined expected structs/types. Not only do you need to know what you're getting anyway, so you can actually do something meaningful with it, but it also helps you find changes, malformed input, reject nonsense, etc etc etc. All the code that follows from the json unmarshalling ends up a lot easier afterwards.

Using map[string]interface{} (note, not all json starts with an object, it can be an array, or just a string, or many other valid json types, too. so that would break immediately as well), leads you down a road straight into reflection hell, and personally I think you should avoid that at all cost, or at least isolate it in some tailored libraries so it gives you... structured data that you can predictably work on.

In neither python nor PowerShell have I ever had to pre-define a response structure and then populate the data. I've had to declare classes, which expect a structure and use the resulting REST response object to help populate a class properties. But how do people deal with REST APIs using Go if they have to pre-write structs to deal with the dozens, sometimes hundreds of different responses? By literally writing dozens, sometimes hundreds of structs? Really?

I want to write a helper function to invoke a rest call and leave parsing of the response to the functions that invoke that helper function. JSON in -> object out.

For instance, in PowerShell, I'll write a bespoke Invoke-ProductRestCall function that does all of the heavy lifting (authentication, URI construction and validation, response error handling, etc)

Then I'll write a cmdlet as a function for like, Get-ProductObject that will use the Invoke-ProductRestCall to populate object data.

For a very basic example in PowerShell, look at lines 32 and 33 here. Line 32 invokes a custom rest call to the API and populates a (generic PSObject) $response, and then the cmdlet function returns that response as-is.

At no point do I ever need to pre-define a struct. I get that powershell is often regarded as a weaker language, and I'm obviously no developer so idk, I'm sure it is. But I can't believe that a modern language like Go is this much more cumbersome for basic JSON management.

Blame Pyrrhus fucked around with this message at 17:05 on Jul 9, 2020

# ¿ Jul 9, 2020 16:07

Blame Pyrrhus: May 6, 2003; Me reaping: Well this fucking sucks. What the fuck.; Pillbug

gariig posted:

Go is very opinionated language. One of those opinions is that it's strongly typed and statically checked, like 99%. Like geonetix said you can use map[string]interface{} but Go is going to fight you every step of the way. Forcing you to cast and check when assigning to something not an interface{}. Go is going to push you to define a struct and say how to marshal the JSON to that struct. You can choose not to do this but since you are going against Go's opinion it's going to make it VERY difficult.

The one up side to Go's opinion is that an "object.foo" was meant to be there because there's a struct that says "foo string" exists and a compiler checked it. With Powershell when you do "object.foo" does it work? Beats me, run the script and see. You can't know that "object.foo" is correct because the check is dynamic and has to checked at run time.

How my job gets around making lots of structs is using Protocol Buffers and letting the protobuf library handle the marshal/unmarshal of data (both JSON and protobuf). I'm sure there's a Swagger->Go struct tool out there which will help. Ultimately, Go has very little support for dynamic programming unlike Python and Powershell which is strongly typed but dynamically checked)

Honestly, pulling JSON into Go is probably the weaker side of the language. It can do it but it's not a great experience. It's better at receiving and returning JSON as a server, which is where the static nature of the JSON marshalling is great. Much easier to see where things changed IMO.

As far as Powershell is a "weaker language" it's a language that is trying to solve a different problem. Doesn't make it better or worse and it's probably the "better" language for talking to random JSON APIs.

This makes some sense, and honestly when I re-read my last post it comes across as more angry than I intended. It's an approach that - as an operations person who writes code in service of automating "things" like (IaaS deployments) - seems counter-intuitive, but that's mostly because I am just dumb.

Ultimately my goal with learning Go was to author a basic terraform provider for an API-driven storage platform, and at a high level everything about how a terraform provider is written is pretty straight forward. provider() and main() functions largely seem to serve as schemas that map the resource -> resourceFunction relationship, and downstream from that you write the CRUD functions to do all of the heavy lifting. This is going to involve modeling a lot of REST calls to our platform, so I've been looking more closely at the parts involved with doing that specifically.

You're right about Go fighting you though, it's pretty easy to create an inferred "struct" of sorts by declaring a map[string]interface{} but the moment I try to do something like, access a pointer by name within that map, or range loop through over an 'interface{}' type, lint just yells at me and calls me an idiot. Which apparently I am.

As for the $object.foo in PowerShell, not sure if this is what you mean, but you can enforce foo as string when you create the class. It's flexible, either let the object infer it's own type for foo via convert when it doesn't matter, or if it does matter just build a class and enforce the type like:

code:

class testClass {
    [string] $foo
}
$object = [testClass]::new()
$object.foo = "string"

Trying to set $object.foo to an invalid value will get you red ink. One thing I really have grown to like about PowerShell is how flexable the class declarations are as well as being able to natively and super-easily enforce classes and types for parameters.

Blame Pyrrhus fucked around with this message at 22:14 on Jul 9, 2020

# ¿ Jul 9, 2020 22:08

Blame Pyrrhus: May 6, 2003; Me reaping: Well this fucking sucks. What the fuck.; Pillbug

Yeah I'm just maintaining a types.go in the project and appending it as-needed.

Worked close with an old friend to build out the bones of a SDK and learned a few good practices around dealing with API calls, as well as using mapstructure for dealing with json responses a little more easily. Now I'm having a blast writing functions in a small Go SDK for our platform and using those when I build out resource functions in our terraform provider.

Learning a lot, and can really appreciate how well documented a lot of things in Go (and terraform) are, since I'm not a developer at all but can make sense of what I am reading. Now when I see lint I immediately know where I hosed up.

# ¿ Aug 20, 2020 01:07

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Go: Google's Plan9 refugees are staging a breakout