|
Ignoring the "compile struct to parsing code" alternatives, some dynamic stuff: 1. https://github.com/jmoiron/jsonq 2. https://github.com/valyala/fastjson 3. https://github.com/stretchr/objx 4. https://github.com/yalp/jsonpath Ranked in order of interestingness.
|
![]() |
|
![]()
|
# ? May 25, 2024 17:25 |
|
geonetix posted:There's typicall two schools of thought about the json stuff in go as far as I've seen: declare your expected responses explicitely and unmarshal into that, or use the map method. In neither python nor PowerShell have I ever had to pre-define a response structure and then populate the data. I've had to declare classes, which expect a structure and use the resulting REST response object to help populate a class properties. But how do people deal with REST APIs using Go if they have to pre-write structs to deal with the dozens, sometimes hundreds of different responses? By literally writing dozens, sometimes hundreds of structs? Really? I want to write a helper function to invoke a rest call and leave parsing of the response to the functions that invoke that helper function. JSON in -> object out. For instance, in PowerShell, I'll write a bespoke Invoke-ProductRestCall function that does all of the heavy lifting (authentication, URI construction and validation, response error handling, etc) Then I'll write a cmdlet as a function for like, Get-ProductObject that will use the Invoke-ProductRestCall to populate object data. For a very basic example in PowerShell, look at lines 32 and 33 here. Line 32 invokes a custom rest call to the API and populates a (generic PSObject) $response, and then the cmdlet function returns that response as-is. At no point do I ever need to pre-define a struct. I get that powershell is often regarded as a weaker language, and I'm obviously no developer so idk, I'm sure it is. But I can't believe that a modern language like Go is this much more cumbersome for basic JSON management. Blame Pyrrhus fucked around with this message at 17:05 on Jul 9, 2020 |
![]() |
|
Linux Nazi posted:At no point do I ever need to pre-define a struct. I get that powershell is often regarded as a weaker language, and I'm obviously no developer so idk, I'm sure it is. But I can't believe that a modern language like Go is this much more cumbersome for basic JSON management. Go is very opinionated language. One of those opinions is that it's strongly typed and statically checked, like 99%. Like geonetix said you can use map[string]interface{} but Go is going to fight you every step of the way. Forcing you to cast and check when assigning to something not an interface{}. Go is going to push you to define a struct and say how to marshal the JSON to that struct. You can choose not to do this but since you are going against Go's opinion it's going to make it VERY difficult. The one up side to Go's opinion is that an "object.foo" was meant to be there because there's a struct that says "foo string" exists and a compiler checked it. With Powershell when you do "object.foo" does it work? Beats me, run the script and see. You can't know that "object.foo" is correct because the check is dynamic and has to checked at run time. How my job gets around making lots of structs is using Protocol Buffers and letting the protobuf library handle the marshal/unmarshal of data (both JSON and protobuf). I'm sure there's a Swagger->Go struct tool out there which will help. Ultimately, Go has very little support for dynamic programming unlike Python and Powershell which is strongly typed but dynamically checked) Honestly, pulling JSON into Go is probably the weaker side of the language. It can do it but it's not a great experience. It's better at receiving and returning JSON as a server, which is where the static nature of the JSON marshalling is great. Much easier to see where things changed IMO. As far as Powershell is a "weaker language" it's a language that is trying to solve a different problem. Doesn't make it better or worse and it's probably the "better" language for talking to random JSON APIs.
|
![]() |
|
gariig posted:Go is very opinionated language. One of those opinions is that it's strongly typed and statically checked, like 99%. Like geonetix said you can use map[string]interface{} but Go is going to fight you every step of the way. Forcing you to cast and check when assigning to something not an interface{}. Go is going to push you to define a struct and say how to marshal the JSON to that struct. You can choose not to do this but since you are going against Go's opinion it's going to make it VERY difficult. This makes some sense, and honestly when I re-read my last post it comes across as more angry than I intended. It's an approach that - as an operations person who writes code in service of automating "things" like (IaaS deployments) - seems counter-intuitive, but that's mostly because I am just dumb. Ultimately my goal with learning Go was to author a basic terraform provider for an API-driven storage platform, and at a high level everything about how a terraform provider is written is pretty straight forward. provider() and main() functions largely seem to serve as schemas that map the resource -> resourceFunction relationship, and downstream from that you write the CRUD functions to do all of the heavy lifting. This is going to involve modeling a lot of REST calls to our platform, so I've been looking more closely at the parts involved with doing that specifically. You're right about Go fighting you though, it's pretty easy to create an inferred "struct" of sorts by declaring a map[string]interface{} but the moment I try to do something like, access a pointer by name within that map, or range loop through over an 'interface{}' type, lint just yells at me and calls me an idiot. Which apparently I am. As for the $object.foo in PowerShell, not sure if this is what you mean, but you can enforce foo as string when you create the class. It's flexible, either let the object infer it's own type for foo via convert when it doesn't matter, or if it does matter just build a class and enforce the type like: code:
Blame Pyrrhus fucked around with this message at 22:14 on Jul 9, 2020 |
![]() |
|
Yeah, if you use interface{} you have to cast to some other type (probably a nested map[string]interface{}) all the time. Writing down struct definitions is annoying if you have many/complicated json responses and we also mostly deal with that by having both service and caller share autogenerated protobuf structs. But I'd definitely prefer writing a struct definition over a bunch of casting, particularly because it also documents the overall shape of the expected response locally.
|
![]() |
|
Linux Nazi posted:pre-write structs to deal with the dozens, sometimes hundreds of different responses? By literally writing dozens, sometimes hundreds of structs? Really? Yeah this is what is done in statically typed languages in my experience. Tooling usually makes it easier, like in C#/VS you can just import a huge json structure and it'll create all the classes for you. A couple of refactoring commands later they are all in seperate files with correct formatting and naming conventions applied. This probably comes down to personal preference but I would for sure prefer a stack of structs over a map of interface{}'s.
|
![]() |
|
I use JSON-to-Go for these sorts of tasks.
|
![]() |
|
When I was learning Go, I've written a utility that would parse the json (what is json but a string in a specific format) into separate files with defined structs. Every object was a file, and then properties converted to estabilished types; if there was a nested object - another file. While I could import those files after the program finished running I couldn't import them dynamically - I didn't simply want to generate JSON structs that would be usable next time the program would run - I wanted them now. Then I started playing with plugin - figuring, hey, let's import those structs as plugin, found out plugins don't allow that and you still end up with interface{} because something something and then got fed up with it all. Was a couple of years ago, so if I'm getting something wrong, I blame it on my broke brain. I guess, my problem with Go was that even with knowledge of different programming languages, looking at the documentation and after going through tutorials and book about programming in Go I still didn't know what was possible and what wasn't, effectively fighting with the language, because yes, same as you I found it weird that modern language can't import a string in a defined format into a usable struct. canis minor fucked around with this message at 00:58 on Aug 1, 2020 |
![]() |
|
But if you imported a random json into a dynamic type, it wouldn't be typesafe in other functions and you're playing defense with execution-time debugging. That's the whole point of strong typing. I do a lot of json with Go (rest integrations) and don't find it to be annoying. I'm defining the types I'm using anyway, it's not much extra to tack on a json tag. I also use couchDB as a back end instead of doing whatever silly file storage thing that is above.
|
![]() |
|
Yeah I'm just maintaining a types.go in the project and appending it as-needed. Worked close with an old friend to build out the bones of a SDK and learned a few good practices around dealing with API calls, as well as using mapstructure for dealing with json responses a little more easily. Now I'm having a blast writing functions in a small Go SDK for our platform and using those when I build out resource functions in our terraform provider. Learning a lot, and can really appreciate how well documented a lot of things in Go (and terraform) are, since I'm not a developer at all but can make sense of what I am reading. Now when I see lint I immediately know where I hosed up.
|
![]() |
|
Hello Go-thread. I'm planning my first Go-project with a couple of people and am having a hard time figuring out whether my design choices are sane or not, please help if you have the time. The plan is to make a backend consisting of several microservices and run it off Google Cloud Run. A Kubernetes Engine would probably be a better choice, but it doesn't have a free tier and we don't want to spend money, so we decided that Cloud Run should be fine for a proof-of-concept / minimum viable product. My background is primarily in C#, so I'm used to singletons and the like. So my plan was to have my main.go file instantiate a single copy of my other classes (logger, db, etc.) and then pass pointers around. For example, my db class does some logging, so after creating an instance of the logger in main, I would then create an instance of the db class and pass a pointer of the logger to the db class, instead of creating a new instance specifically for the db class. To me, this also feels like it would mesh well with Cloud Run, because there's generally no need to create multiple instances of service classes. Does this seem sensible? It's technically not a strict singleton, since there's nothing limiting me from creating 10,000 loggers in main. Because of this, I think trying to keep a singleton pattern going on might be a bad idea, and if someone else keeps working on the project it'll eventually lead to a situation where you have a mixed bag of a single instance being passed around as a pointer and some classes creating their own instances. Is there a way of forcing a strict singleton pattern with go? Am I overthinking this too much and should I just go gently caress it and implement what feels good to me?
|
![]() |
|
What advantage does singleton provide over passing in the logger and DB connections (or storage providers) as args? In my opinion, it's always better for a web server to instantiate things at startup time instead of attempting to do it in a handler. This way you can write tests and benchmarks with full confidence that all of the needed pieces go in as args instead of having to remember to set up other state beforehand.
|
![]() |
|
rt4 posted:What advantage does singleton provide over passing in the logger and DB connections (or storage providers) as args? In my opinion, it's always better for a web server to instantiate things at startup time instead of attempting to do it in a handler. This way you can write tests and benchmarks with full confidence that all of the needed pieces go in as args instead of having to remember to set up other state beforehand.
|
![]() |
|
Kuule hain nussivan posted:Hello Go-thread. I'm planning my first Go-project with a couple of people and am having a hard time figuring out whether my design choices are sane or not, please help if you have the time. Cloud Run's free tier has been great for me. I did move a service to GKE that had some larger memory requirements, but Cloud Run seems to work for everything else. For loggers, I honestly use package level variables, as they tend to be safe for concurrent use, and don't impact my overall application design very much. For other things, like DB connections, or other service connections, I typically instantiate a Server struct, and provide those connections or configuration to it in the main function. This helps you structure your handlers in a way where it's easier to plug in a fake database for testing, or a fake/in-memory database or service for local development. I find it helps me keep a separation of concerns between different objects in my app. In general, instantiating things in Go is extremely cheap, so I take advantage of that unless I want persistent connections, which is frequently the case for databases and other services. If you're new to Go, one of the first things that helped me a lot when learning how to work with the language was that in general, functions should return pointers to structs, and accept interfaces as arguments. Interface definitions should go by where they're consumed, not where they're produced. This will help you keep a good contract at the call-site, and make it easier to change implementations without breaking consumers. As a bonus, it makes things really easy to fake out for unit testing, helping you avoid integration tests except where necessary. Finally, always run tests with the race detector on, and use go vet. Have fun.
|
![]() |
|
Does this thread have any recommendations for Go books? I'm looking for a book that ideally :
I'm coming from a Java & Ruby background and am trying to learn Go, and some parts are just confusing the hell out of me (modules & local packages, whatever "vendor" and "vgo" are, some bits of syntax, etc.). Is there a good book that covers all of this, or is Go too new/changing too quickly for this? If it helps, the "Programming Ruby - The Pragmatic Programmers' Guide" is a good example of the sort of book I'm after.
|
![]() |
|
For the basics / syntax either of the following options are good, iirc I made my way through the tour site referencing sections of the book when I wanted to dive deeper into something: The Go Programming Language - Alan Donovan (https://www.amazon.com/dp/0134190440) https://tour.golang.org/welcome/1 My current read as I’m attempting to shift from .NET to go: Concurrency in Go - Katherine Cox-Buday (https://www.amazon.com/dp/1491941197)
|
![]() |
|
The modules system is the hardest-to-understand part of Go imo. It's strict, it's picky, and it can be unintuitive. It's also better than the two previous alternatives (dep, or "just grab latest off github and run with it") so I put up with it, but on our large corporate project we have to fight it from time to time. On a small project, it should be pretty reasonable to deal with, just follow the wiki.
|
![]() |
|
Iverron posted:The Go Programming Language - Alan Donovan (https://www.amazon.com/dp/0134190440) It says this was published in 2015. Is this still "current"? I've been getting the impression that Go is changing & developing as a language quite quickly, and I don't want to accidentally learn out-of-date syntax & conventions as that can lead to me internalising bad practices as "the way you do things in Go". Which would be bad!
|
![]() |
|
FnF posted:It says this was published in 2015. Is this still "current"? I've been getting the impression that Go is changing & developing as a language quite quickly, and I don't want to accidentally learn out-of-date syntax & conventions as that can lead to me internalising bad practices as "the way you do things in Go". Which would be bad! 2015 was well into Go 1.x territory, which means nothing you learn there should be invalidated by later updates in the 1.x line. Syntax will not change, but syntax is minor anyway. The standard library calls should all work, although the best practices may have shifted on a few things. Kernighan is the co-author on that book, and I'll always recommend his work. I haven't read it, but I'm sure it will help you get your head around the concept of goroutines and channels, which seems to be the biggest challenge for a lot of new Go programmers. Then, use the online documentation (https://github.com/golang/go/wiki/Modules helps) to learn how to use the modules system, and you'll be in good stead.
|
![]() |
|
Alright then. Thank you both for your help!
|
![]() |
|
I'm new to go, and having trouble with math/big. I'm creating lots of equations, but math/big uses all methods, no functions, so my code is looking hellish. Like, I want to do this: code:
code:
code:
Am I doing this right?
|
![]() |
|
Taking a look at the docs (https://pkg.go.dev/math/big), it seems like those methods all return the result in addition to setting the receiver. So you should be able to use it as you wish; please forgive my slight re-writing:code:
|
![]() |
|
Thanks. I’ll play around with using a receiver like that, but maybe you’re right and I’ll stick with what I have.
|
![]() |
|
Trying to understand an idiomatic approach to a set-once lazy value for a struct. I have a simple struct: code:
Should I: 1. Pass in an instance of a seeded *rand.Rand() to IsShiny and only calculate it if o.shiny == nil 2. Create a constructor that accepts *rand.Rand() and do the dice roll during construction My goal is to be able to test the IsSiny function and make sure the `shiny` field can't be changed after it's set. edit: also, the dice roll does not need to be perfectly random Fluue fucked around with this message at 03:56 on Sep 5, 2021 |
![]() |
|
Fluue posted:Should I: If you got with option 1 every time someone calls IsShiny a *rand.Rand() has to be passed in even if shiny has been set. I would go with option 2 and I would have two constructors. One is internal that’s used by the tests where a rand.Rand can be passed in. The exported constructor would not take a rand.Rand, make one, and use the internal constructor. This keeps it so your test (if in the same package) can set a predetermined rand.Rand but the caller doesn’t get to set anything. There’s no way to guarantee shiny isn’t changed in some code path internally. Go doesn’t have the concept of const or readonly. So basically you just have to make sure nothing sets it outside the constructor
|
![]() |
|
gariig posted:If you got with option 1 every time someone calls IsShiny a *rand.Rand() has to be passed in even if shiny has been set. Agree. For bonus points, don't take rand.Rand but rather func() int. This will make the unit tests you are writing a lot more bearable.
|
![]() |
|
Thank you both! Good suggestion to use an internal constructor.
|
![]() |
|
Be careful about the logic of setting shiny if it's nil. If you have two threads enter that function at the same time (e.g., if the struct is being read and therefore shiny is being initialized once by two different go-routines) you can end up with two different values of shiny, as if both check if shiny is nil at the same time they'll both go on to randomly generate a value. A constructor will help to prevent this as you're less likely to accidentally use a constructor in this way, but another solution (if the constructor ends up being un-ergonomic and you want to stay lazy) is the neat sync.Once mutex, which I think is fairly idiomatic. Just set it inside the sync.Once.Do, and then read it directly in the getter. No comparing to nil needed so it works for non-pointer values. This is safe "Because no call to Do returns until the one call to f returns". Either way, n-thing what the other posters have said, make sure your source of randomness is injectable otherwise testing is going to be annoying.
|
![]() |
|
I want to marshal json into a struct. Given the following json, I've come up with the following struct. But the keys set for tags could actually be anything, not restricted to name or aggregatedBy. How do I represent this in my struct?code:
code:
code:
Methanar fucked around with this message at 18:40 on Sep 6, 2021 |
![]() |
|
Methanar posted:I want to marshal json into a struct. Given the following json, I've come up with the following struct. But the keys set for tags could actually be anything, not restricted to name or aggregatedBy. How do I represent this in my struct? This will handle arbitrary json fields if you marshal the extra data in tags to that member. Go code:
|
![]() |
|
So the existence of tags at all is optional. But if it does exist, it's a map of strings with values of also strings.code:
|
![]() |
|
Methanar posted:So the existence of tags at all is optional. But if it does exist, it's a map of strings with values of also strings. Yeah that should work. Edit: Actually I think using anonymous structs would work better if Tags is omitted, the other way if Tags always exists but can be empty. Go code:
Da Mott Man fucked around with this message at 19:19 on Sep 6, 2021 |
![]() |
|
go is fuckin good
|
![]() |
|
30.5 Days posted:go is fuckin good ![]()
|
![]() |
|
30.5 Days posted:go is fuckin good It is OP, it is. I just doubled my salary (not hard, I was making dog poo poo before, lol) based primarily on me being proficient in Go. I was in academia before doing scaled web services for outside researchers, and I've now gone to the private sector. The thing that will jokerfy me, though, is that they were real excited to have someone like me on their team during early interviews. Then in final round they dropped the bomb that they don't even use Go, really. They just let the engineers freehand it. Thankfully my time in academia has prepared me to walk into a mess so I'm kind of excited either way. I like Go because I specifically have a lot of experience with other languages/stacks where things are quite a lot more broken. So it's actually really weird how good their job posting was because it was like exactly suited to someone like me who can come in to clean up messes. ErIog fucked around with this message at 02:49 on Apr 15, 2022 |
![]() |
|
For you Go'ers (or whatever you call yourselves...): do you use Go for any sort of data analytics / data analyst jobs? I keep seeing it pop up on those type of job postings, and can't tell if I'm missing a big chunk of the data sector or if they are just spitballing keywords for the job description.
|
![]() |
|
Hughmoris posted:For you Go'ers (or whatever you call yourselves...): do you use Go for any sort of data analytics / data analyst jobs? We use it for logs and event analysis of pretty egregiously large streaming datasets (petabytes). I think Segment does too for data, as well as Uber for lots of different things. It's pretty well suited for getting most of the c++ performance with fewer footguns. pprof is your friend.
|
![]() |
|
Breaking Glass posted:We use it for logs and event analysis of pretty egregiously large streaming datasets (petabytes). I think Segment does too for data, as well as Uber for lots of different things. It's pretty well suited for getting most of the c++ performance with fewer footguns. Funny, I also use it for logs & event analysis. Do you work on an analysis product, and if so, do you mind sharing which one? If not, that's obviously fine.
|
![]() |
|
Breaking Glass posted:We use it for logs and event analysis of pretty egregiously large streaming datasets (petabytes). I think Segment does too for data, as well as Uber for lots of different things. It's pretty well suited for getting most of the c++ performance with fewer footguns. That's good to know, thank you. My limited experience has been in healthcare with Python and SQL but I'm starting to look in a new sector and I was puzzled by Go showing up.
|
![]() |
|
![]()
|
# ? May 25, 2024 17:25 |
|
Pham Nuwen posted:Funny, I also use it for logs & event analysis. Do you work on an analysis product, and if so, do you mind sharing which one? If not, that's obviously fine. Can't post it here but happy to nerd out over discord if you wanna DM
|
![]() |