Ask General Programming Questions Not Worth Their Own Thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »

The Fool: Oct 16, 2003

Not the popular choice, but VSTS ( https://www.visualstudio.com/team-services/ ) is pretty nice these days.

# ? Jan 11, 2018 18:15

Adbot: ADBOT LOVES YOU

# ? Jun 1, 2024 21:55

Capri Sun Tzu: Oct 24, 2017; by Reene

Peristalsis posted:

I work in a Ruby on Rails shop, and we use a local GitLab instance for issue tracking, merge requests, etc. We've been doing code reviews by assigning merge requests to other developers, who look over the changes, comment on them, assign them back, etc. It's not a great system, for a few reasons, and I'm wondering what recommendations people have for better code review tools. This could be either a GitLab add-in, or something completely different that we integrate with our git repository in some way. It's even possible we could abandon GitLab entirely if some other suite works as well for issue tracking, and better for code reviews. It doesn't have to be free, though that would be nice (and would need at least a free trial period to check out).

GitHub if you don't mind paying, Bitbucket if you do.

# ? Jan 11, 2018 23:27

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

I'm debugging a graph connectivity issue. The tl;dr is that I have a directed graph, with certain nodes marked as exit nodes, and I need to know that when I remove edges from a specific node in the graph, it remains possible to get from any node in the graph to any exit node, and from any exit node to any node (functionally, that the graph remains fully-connected). I want to do this efficiently, because I'm going to be trying a lot of different combinations of edge removals; doing a graph traversal for every possible modification is too expensive. However, I'm always modifying the same node N in the graph, so what I've been doing up to now is saying "okay, assuming the rest of the graph is fixed, pre-calculate connectivity without relying on N, then see what different sets of edges involving N do to connectivity."

The link above highlights an issue I'm not currently handling, which is that I'm not recognizing transitive connectivity. If node A connects to B connects to C, and N connects to C, then A connects to N. I'm pretty sure I could handle this if I had a many-to-many connectivity map, detailing for every tile what tiles can reach it / be reached from it. I would only have to generate this many-to-many mapping once for each N I touch. However I don't know of any many-to-many graph traversal functions. In the past I'd just been doing two breadth-first searches for each exit node (one for each edge direction). I could extend that to doing two breadth-first searches for every node in the graph, which would have a runtime of (number of nodes) * (number of nodes + number of edges), which isn't great. Anyone know of a better option?

# ? Jan 12, 2018 06:01

Linear Zoetrope: Nov 28, 2011; A hero must cook

TooMuchAbstraction posted:

I'm debugging a graph connectivity issue. The tl;dr is that I have a directed graph, with certain nodes marked as exit nodes, and I need to know that when I remove edges from a specific node in the graph, it remains possible to get from any node in the graph to any exit node, and from any exit node to any node (functionally, that the graph remains fully-connected). I want to do this efficiently, because I'm going to be trying a lot of different combinations of edge removals; doing a graph traversal for every possible modification is too expensive. However, I'm always modifying the same node N in the graph, so what I've been doing up to now is saying "okay, assuming the rest of the graph is fixed, pre-calculate connectivity without relying on N, then see what different sets of edges involving N do to connectivity."

The link above highlights an issue I'm not currently handling, which is that I'm not recognizing transitive connectivity. If node A connects to B connects to C, and N connects to C, then A connects to N. I'm pretty sure I could handle this if I had a many-to-many connectivity map, detailing for every tile what tiles can reach it / be reached from it. I would only have to generate this many-to-many mapping once for each N I touch. However I don't know of any many-to-many graph traversal functions. In the past I'd just been doing two breadth-first searches for each exit node (one for each edge direction). I could extend that to doing two breadth-first searches for every node in the graph, which would have a runtime of (number of nodes) * (number of nodes + number of edges), which isn't great. Anyone know of a better option?

You may want to try so-called "lifelong planning" algorithms as your starting point, like D* Lite, which is essentially A*, but with an optimization that allows nodes to change. (Note that D*-lite has a small flaw in that the closer the change is to the goal, the longer it takes to recompute, but it's an example of a starting point).

E: Your general approach here is going to be caching, compute from the exits back if something is reachable, and maintain a list of nodes known to be reachable, once you hit a reachable node in your traversal, you can mark all speculative nodes as reachable. The fact that this isn't a DAG makes the removal case a bit harder tho, I don't think you can simply backtrack from the removal and see if there's an alternate route because those other nodes may have been marked reachable. You can probably do something with the matching Strongly Connected Component graph.

E2: Thinking more, SCC is what you want here, when you create the graph, find the strongly connected components, and annotate the edges with what edges in the source graph they correspond to. When you remove the edge:

1. If the edge also exists between two SCC towards the exit, you're fine if that edge disappears.
2. If not, you're going to have to recompute the SCC for the subgraph containing the nodes of that SCC, and make sure those still lead to the exit.

Linear Zoetrope fucked around with this message at 06:40 on Jan 12, 2018

# ? Jan 12, 2018 06:25

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

Linear Zoetrope posted:

You may want to try so-called "lifelong planning" algorithms as your starting point, like D* Lite, which is essentially A*, but with an optimization that allows nodes to change. (Note that D*-lite has a small flaw in that the closer the change is to the goal, the longer it takes to recompute, but it's an example of a starting point).

E: Your general approach here is going to be caching, compute from the exits back if something is reachable, and maintain a list of nodes known to be reachable, once you hit a reachable node in your traversal, you can mark all speculative nodes as reachable. The fact that this isn't a DAG makes the removal case a bit harder tho, I don't think you can simply backtrack from the removal and see if there's an alternate route because those other nodes may have been marked reachable. You can probably do something with the matching Strongly Connected Component graph.

E2: Thinking more, SCC is what you want here, when you create the graph, find the strongly connected components, and annotate the edges with what edges in the source graph they correspond to. When you remove the edge:

1. If the edge also exists between two SCC towards the exit, you're fine if that edge disappears.
2. If not, you're going to have to recompute the SCC for the subgraph containing the nodes of that SCC, and make sure those still lead to the exit.

Thanks for the tips, I'll definitely check them out!

# ? Jan 13, 2018 05:34

Linear Zoetrope: Nov 28, 2011; A hero must cook

TooMuchAbstraction posted:

Thanks for the tips, I'll definitely check them out!

Note that if you have multiple exits, you may want to compound them into a single "sink" exit that represents all of them just for ease of use.

# ? Jan 13, 2018 06:19

Pollyanna: Mar 5, 2005; Milk's on them.

I'm trying to get Homebrew to install and launch Postgres and it doesn't work and I don't know what the gently caress. Googling has failed me. The post-install instructions ask me to run `brew services start postgresql` but I just get this:

code:

$ brew services start postgresql
Usage: launchctl enable <service-target>
Error: Failure while executing: /bin/launchctl enable gui/501/homebrew.mxcl.postgresql

Nothing like this shows up online. Does Homebrew somehow not know how to use launchctl, or did something get hosed up?

EDIT: Turns out that trying to `brew services restart postgres` put me in a bad state and stuck me in as user 501. I don't know what the gently caress, but forcibly uninstalling all Postgres installations in Homebrew and deleting the `/usr/local/var/postgres/`, then reinstalling Postgres thru Homebrew fixed it. Still had to make the postgres superuser myself, though.

Pollyanna fucked around with this message at 18:55 on Jan 14, 2018

# ? Jan 14, 2018 17:54

Jewel: May 2, 2009

Hey how the hell do you get Haskell working properly on Windows. I hate windows package managers so bad.

I primarily use C++ at work and C# at home so I'm spoiled when it comes to pleasant IDE's (though I wouldn't say C++ is always pleasant, it at least immediately works in an IDE). I was looking to get into Haskell because friends use it and I wanted to learn a functional lang, but because I've been spoiled and I've always used Windows, I'm extremely bad at command-line style installation/config.

So I install full Haskell Platform as it recommends, and that all works

Then, I want to set up Atom because it seemed to have the best tooling available with debuggers and etc. So I go to install dependencies, and every guide I can find says stuff like this

So I'm like, okay, I'll take it slow, never used this before and don't know how it looks if there's a problem; let's just try ghc-mod

Already I'm in hell :negative:

So, "stack solver" just wants me to make a package and do it local to the project (I don't want a project right now, these should be global hooks; and also if I do make a project it still has the same errors), so the only lead I can find at all is adding the extra-deps. So.. I do!

Every one I add, it fixes the warning in the list. Great! Just one last one, and, uh,

It just refuses to detect base. No matter what I do. And setting allow-newer also does nothing. So I'm at a complete standstill after even the first package. How the heck do I get this to work and why are all windows package managers so bad :sigh:

# ? Jan 17, 2018 01:03

b0lt: Apr 29, 2005

Jewel posted:

Hey how the hell do you get anything that isn't from microsoft working properly on Windows.

don't

# ? Jan 17, 2018 01:08

-Anders: Feb 1, 2007; Denmark. Wait, what?

Maybe try F# if you want to try your hand with functional programming? I really dig the syntax, and I suppose it's got great support on Windows/VS

# ? Jan 17, 2018 06:42

Jewel: May 2, 2009

Specifically chose Haskell because some friends use it very often and I want to join them in things.

# ? Jan 17, 2018 11:39

Ghost of Reagan Past: Oct 7, 2003; rock and roll fun

Try stack upgrade.

Otherwise reinstall the Haskell Platform. I expect it's some PATH fuckery.

All else fails, I recommend looking into Windows Subsystem for Linux.

# ? Jan 17, 2018 13:46

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

b0lt posted:

don't count on open source people to make a working installer/platform for any environment they don't personally use

ftfy

# ? Jan 17, 2018 14:53

Jewel: May 2, 2009

I got an answer on reddit, in case anyone wanted to know the solution (not at home yet but I'm sure it'll work judging by multiple responses)

quote:

The problem here is that ghc-mod hasn't caught up with the newest GHC version yet. You'll probably have to either wait a bit for a new release or install GHC 8.0 instead of 8.2.

quote:

changing to resolver: lts-9.21 in the global stack.yaml file (and removing all extra-deps) allows stack install ghc-mod to start compiling.

Just my luck that I keep getting things when they happen to be too new and not supported!

# ? Jan 17, 2018 15:41

Roundboy: Oct 21, 2008

Having a bad time dealing with python loggin and I keep missing the google-fuo to figure out the exact issue.
Python 3.5+

I have a custom formatter which adds two extra variables to all my output logs :

code:

logger = logging.LoggerAdapter(logging.getLogger(__name__), {'jobid': job_id,  'source': source})

which works fine. The problem is other modules (like google or pysftp, etc) that also use python standard logging throwing Key Error because they do not have access to these 2 extra variables.

I realize the custom format is the issue but i am having trouble either:

Selectively applying the custom format to ONLY the log messages called by my script explicitly (not the info/debug from imported modules)
or
extending logging to add the extra to every message, regardless of module.

I do want to avoid adding each exta explicitly like :

code:

 log.info('This is a message',extra={'jobid': job_id, 'source': source})

each and every time.

Can anyone help me wrap my head around this I'm just not getting it

# ? Jan 17, 2018 17:06

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Roundboy posted:

logging

I don't have the wherewithal to dig my hard-won python logging knowledge out of my brain right now, but I wanted to point out that there's a Python thread.

# ? Jan 17, 2018 17:42

Ranzear: Jul 25, 2013

Let's not talk about what version of gcc the Windows Haskell installer will drop on you either...

# ? Jan 17, 2018 19:11

Looten Plunder: Jul 11, 2006; Grimey Drawer

Apologies that this isn't programming related, but is there a Microsoft Excel questions/help thread anywhere on these forums?

# ? Jan 19, 2018 03:58

downout: Jul 6, 2009

Looten Plunder posted:

Apologies that this isn't programming related, but is there a Microsoft Excel questions/help thread anywhere on these forums?

Ctrl + F -> excel

https://forums.somethingawful.com/showthread.php?threadid=3132163

# ? Jan 19, 2018 04:08

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

Linear Zoetrope posted:

E2: Thinking more, SCC is what you want here, when you create the graph, find the strongly connected components, and annotate the edges with what edges in the source graph they correspond to. When you remove the edge:

1. If the edge also exists between two SCC towards the exit, you're fine if that edge disappears.
2. If not, you're going to have to recompute the SCC for the subgraph containing the nodes of that SCC, and make sure those still lead to the exit.

Thanks again for this. I'm still working on my rewrite, but it's pretty clear that strongly-connected components are what I wanted from the get-go. I'd been doing all of my analysis at the individual-node-in-the-graph level, which made it extraordinarily hard to efficiently reason about large-scale graph structure. Boiling everything down into components effectively gives me a smaller, simpler graph where each node is a strongly-connected component of the original graph. It's pretty easy from there to determine which components can reach / be reached from other components, and make certain that when I modify the underlying graph, I maintain the constraint of "everyone can reach and be reached by everyone else".

# ? Jan 19, 2018 17:04

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

I wanna try and set up my own simple web stack, but I'm not really sure where to start. I feel competent enough in my job working front-end and back-end, but have next to no idea how either is set up, or how to glue them together in a reasonable way. Are there any good tutorials that cover everything you need to know to start making your own efficient and secure web stack?

Specifically, I don't want one that wants to teach me how to program, I can already do that, just one that teaches the patterns of web development and how to set it all up.

# ? Jan 19, 2018 17:56

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Joda posted:

I wanna try and set up my own simple web stack, but I'm not really sure where to start. I feel competent enough in my job working front-end and back-end, but have next to no idea how either is set up, or how to glue them together in a reasonable way. Are there any good tutorials that cover everything you need to know to start making your own efficient and secure web stack?

Specifically, I don't want one that wants to teach me how to program, I can already do that, just one that teaches the patterns of web development and how to set it all up.

This is kind of a broad question.

Do you need a database? What kind of server do you have or want to get? What language do you want to use on the backend?

# ? Jan 19, 2018 18:23

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

Thermopyle posted:

This is kind of a broad question.

Do you need a database? What kind of server do you have or want to get? What language do you want to use on the backend?

Yes need a database and an authentication system. A rest service should suffice. I'm not fuzzy about back-end language, I'd just prefer it to be typed (i.e. no Node.js)

My big problem is, I'm not the sort of person who can just start learning these things one by one. I need some level of guidance, and was hoping there'd be a tutorial that gave me the big picture I need to at least get a road map of the things I need to learn.

Joda fucked around with this message at 18:33 on Jan 19, 2018

# ? Jan 19, 2018 18:30

The Fool: Oct 16, 2003

Use DRF with Postgres, and React for your dynamic content. Use nginx to serve up the front end.

There, decision made, go out into the world.

# ? Jan 19, 2018 19:09

downout: Jul 6, 2009

Joda posted:

I wanna try and set up my own simple web stack, but I'm not really sure where to start. I feel competent enough in my job working front-end and back-end, but have next to no idea how either is set up, or how to glue them together in a reasonable way. Are there any good tutorials that cover everything you need to know to start making your own efficient and secure web stack?

Specifically, I don't want one that wants to teach me how to program, I can already do that, just one that teaches the patterns of web development and how to set it all up.

What fe/be languages and frameworks are you currently comfortable with?

# ? Jan 19, 2018 21:22

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

downout posted:

What fe/be languages and frameworks are you currently comfortable with?

Front-end: JavaScript (ES5 for now) with jQuery, Back-end: C# with ServiceStack and MS-SQL (although mostly through an ORM.) I've been looking into ServiceStack for my own projects but it seems like it does a whole lot for you, and I wanna get my hands dirty with as much of the stack as possible. Mostly out of curiosity, but also because I kinda feel like I need to know how a stack operates to some extend at all levels, if I'm to call myself a full-stack dev.

# ? Jan 19, 2018 23:54

downout: Jul 6, 2009

Joda posted:

Front-end: JavaScript (ES5 for now) with jQuery, Back-end: C# with ServiceStack and MS-SQL (although mostly through an ORM.) I've been looking into ServiceStack for my own projects but it seems like it does a whole lot for you, and I wanna get my hands dirty with as much of the stack as possible. Mostly out of curiosity, but also because I kinda feel like I need to know how a stack operates to some extend at all levels, if I'm to call myself a full-stack dev.

Id just use the MS stack. Asp.net -> iis with whatever DB you want. I did it that way because it can mostly be done for free. If you really want free then php on apache with whatever DB you want. Is this a personal site or to get full stack experience?

# ? Jan 20, 2018 02:21

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

For experience and porfolio.

# ? Jan 20, 2018 02:36

The Fool: Oct 16, 2003

Comedy option: go serverless with Amazon lambda or azure functions

# ? Jan 20, 2018 02:48

downout: Jul 6, 2009

Joda posted:

For experience and porfolio.

That gets into what do you want to dev? If you think you'd like to focus on FE, then non-microsoft products seem best (I'm not FE, so others could give better details). If you want to focus on BE type stuff then C#, asp.net mvc, sql server (or others dbs), etc are an option. Other devs I work with build sites on other setups, but I have limited experience with anything beyond the basic apache.

Like fool said above -> serverless with amazon. There are lots of choices, but it's hard to describe what is easier without more specifics and/or use cases.

edit: just saw your comment on 'getting your hands dirty'. For C#, it's pretty easy. The problems are getting your hands on a microsoft server (maybe just a non-server OS, I can't remember) to deploy your web application to. If you can find that, then it's just Internet Information Services. IIS standard install is pretty much ready to go with default settings. Once that is up and running you can create a VS publish profile to publish the web application straight to the site folder. There's other stuff, but that's mostly starting to get into the dirty hands stuff.

downout fucked around with this message at 03:19 on Jan 20, 2018

# ? Jan 20, 2018 03:07

Joda: Apr 24, 2010; When I'm off, I just like to really let go and have fun, y'know?; Fun Shoe

downout posted:

That gets into what do you want to dev? If you think you'd like to focus on FE, then non-microsoft products seem best (I'm not FE, so others could give better details). If you want to focus on BE type stuff then C#, asp.net mvc, sql server (or others dbs), etc are an option. Other devs I work with build sites on other setups, but I have limited experience with anything beyond the basic apache.

Like fool said above -> serverless with amazon. There are lots of choices, but it's hard to describe what is easier without more specifics and/or use cases.

I've been playing with a couple of ideas that would be nice to put in a portfolio. Like a spoiler-free entertainment wiki that lets users say "I'm on season X of show Y," or "Page/chapter Z of book W" so it'd only show lines/paragraphs/infoboxes tagged safe for that season and earlier. Comes with some interesting challenges both front-end and back-end, and would also allow me to get into the server side of things with generating content pages on-the-fly. At least that's the idea I keep returning to that I also think potential employers might be interested in seeing.

I'm definitely more of a BE guy, although I think that's mostly an aversion to the languages used on the FE. WIth ES6 and TS I'd definitely enjoy it more.

The Fool posted:

Comedy option: go serverless with Amazon lambda or azure functions

Speaking of comedy options, all this searching for web stack tutorials has Google trying to advertise drag'n'drop personal/store website solutions at me.

# ? Jan 20, 2018 03:16

downout: Jul 6, 2009

Joda posted:

... spoiler-free entertainment wiki that lets users say "I'm on season X of show Y," or "Page/chapter Z of book W" so it'd only show lines/paragraphs/infoboxes tagged safe for that season and earlier. Comes with some interesting challenges both front-end and back-end, and would also allow me to get into the server side of things with generating content pages on-the-fly...

I think it's pretty dependent on how you want to implement the web application for this. If it's written in net mvc, then that really points to figuring out IIS, either hosted or created by you in some fashion.

# ? Jan 20, 2018 03:21

Volguus: Mar 3, 2009

Google's Protobuf, how I love thee.

Let's assume that I have a message like this (proto3):

code:

 message Foo {
 int32 a=1;
string b=2;
}

Life is good, everything works out nicely, and the API is being used by a few developers. Now, I would like to change the API (and the messages) to accommodate future expansions. What I want right now is something like this:

code:

message Bar {
  repeated Foo foos=1;
  //and later one will be
 repeated OtherFoos otherFoos=2;
....
}

Can I, somehow, determine at runtime if I am being sent a Foo directly or a Bar that contains Foos? According to everything that I have read out there it's not possible. What happens right now is that I successfully read a Bar, that simply doesn't have any Foos in it, and then I know that I can try to read a Foo.
Is there a way to validate the type of message that's being sent over the wire? Does protobuf have any kind of metadata around the messages that it sends and receives? How does it protect against the client lying, something like: "the repeated message will contain 100 Foos, but I'm only sending 10"?

Also, how come that the C++ API does not have "ParseDelimited..." and "WriteDelimited...". They can be added, but is quite a hassle.

# ? Jan 20, 2018 17:16

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

I'm not at all familiar with the C++ protobuf library, but generally I would expect that your API would declare "I take a Bar as input". Then your code would get handed a binary blob, you would tell the library "here, try to turn this blob into a Bar", and it would either say "here you go" or "nope, that's not a Bar", and in the latter case you return an error to the client.

If your method is declared to take a Foo and you decide you want to change it to take a Bar, then you're dealing with a backwards-incompatible change. Generally, the nice way to handle that is to make a new method that takes Bars, mark the old method as deprecated, and a few versions down the road (at minimum, a year from the introduction of the new method), remove the old version from the library.

So far as I'm aware, you can't have a single method that takes either a Foo or a Bar. Certainly even if it's possible I don't think it's a thing you should do. There should be one correct way to use your library.

# ? Jan 20, 2018 17:39

Volguus: Mar 3, 2009

TooMuchAbstraction posted:

Then your code would get handed a binary blob, you would tell the library "here, try to turn this blob into a Bar", and it would either say "here you go" or "nope, that's not a Bar", and in the latter case you return an error to the client.

That's exactly what I expected as well. But the protobuf library, when told to "make a Bar out of this blob of bytes" it made a Bar even if it had a Foo. It did fail when I sent a blob of bytes all 0x01, or completely random. That was invalid blob. But if I sent a Foo or even half a Foo (I just cut the blob of bytes in half) it worked . And then I had to step in and say: In order for Bar to be valid it has to have at least one Foo.

TooMuchAbstraction posted:

So far as I'm aware, you can't have a single method that takes either a Foo or a Bar. Certainly even if it's possible I don't think it's a thing you should do. There should be one correct way to use your library.

I am not using gRPC at the moment, it is just simply reading the stream of bytes from the network. I agree that there should only be one correct way to use it, I just want to change that way (deprecate the old way) and not inconvenience the 3 people that are still using the old way, not for a while at least. I should have had the Bar message from the start, but is too late now.

# ? Jan 20, 2018 17:47

TooMuchAbstraction: Oct 14, 2012; I spent four years making
Waves of Steel
Hell yes I'm going to turn my avatar into an ad for it.; Fun Shoe

Volguus posted:

That's exactly what I expected as well. But the protobuf library, when told to "make a Bar out of this blob of bytes" it made a Bar even if it had a Foo. It did fail when I sent a blob of bytes all 0x01, or completely random. That was invalid blob. But if I sent a Foo or even half a Foo (I just cut the blob of bytes in half) it worked . And then I had to step in and say: In order for Bar to be valid it has to have at least one Foo.

I'd guess it's just dumb luck that it happens to work out, then. Especially if this is proto3 (which I assume it is since I don't see required/optional keywords being used) I wouldn't be surprised to get a "valid" unpacking where the proto just has zeros for fields. AIUI the binary packing for protos strives for efficiency since wasted bytes can have a huge impact on bandwidth use at scale; that means the binary blob has very little (preferably, zero) data about what shape it is, and it's up to the code at the end to interpret things correctly.

quote:

I agree that there should only be one correct way to use it, I just want to change that way (deprecate the old way) and not inconvenience the 3 people that are still using the old way, not for a while at least. I should have had the Bar message from the start, but is too late now.

Yep, it is too late. Look at it this way: you're getting good practice for how to handle evolving an API. Even if you'd planned things correctly from the start, eventually you'd need to make some backwards-incompatible change.

# ? Jan 20, 2018 18:07

nielsm: Jun 1, 2009

What if you add an int32 version field to the start of Bar, would it then still eat a single Foo buffer?

# ? Jan 20, 2018 18:32

Volguus: Mar 3, 2009

nielsm posted:

What if you add an int32 version field to the start of Bar, would it then still eat a single Foo buffer?

If I do that then it fails as expected. But, looking at the exception thrown, it fails when it reads the string of the Foo, since it is no longer a valid UTF8 starting from that byte. Hmm, it seems that the library (at least the Java one) it does its best to provide an object from whatever it has been given. If two messages are close enough in structure, on the receiving end you may not be any wiser of what you have been sent.

I guess it does make sense, but I am surprised to be honest. I would have thought there must have been some metadata around the message, something that if I send a Foo when I expect a Bar (even if Foo and Bar have an identical structure) the library would yell at me. But yes, at scale, all those extra bytes would add up. Oh well, lesson learned.

# ? Jan 20, 2018 18:58

Mr Shiny Pants: Nov 12, 2012

Well, at least its flexible.

# ? Jan 20, 2018 20:41

Adbot: ADBOT LOVES YOU

# ? Jun 1, 2024 21:55

Jabor: Jul 16, 2010; #1 Loser at SpaceChem

Generally what I'd do in that situation is something like:

code:


message Bar {
  // filled in only by legacy clients that only send a single Foo
  int32 a = 1;
  string b = 2;

  repeated Foo foo = 3;
}

Then in your server the first thing you can do is check if there are any foos, and if not, create one by converting it from the legacy field before passing it to the rest of your code. (If you were using the proto2 libraries, you could also explicitly check whether the legacy fields were set.)

In general, reusing field numbers for semantically different content is a bad idea. You might get away with it in this instance, because raw ints use an identifiably different encoding from embedded messages (though this depends on exactly what library you use - some will error out on seeing this because the proto is clearly malformed, others will skip the mismatched data and carry on), but if you tried to reuse a string field as a different message, you'd start getting junk data when it interprets the content of that string as a serialised message.

# ? Jan 21, 2018 00:28

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Ask General Programming Questions Not Worth Their Own Thread

«‹›671 »