|
cinci zoo sniper posted:but why mongo, not postgres or some poo poo https://www.youtube.com/watch?v=b2F-DItXtZs
|
#
?
Mar 20, 2018 14:17
|
|
|
|
| # ? Jun 4, 2024 23:43 |
|
|
MononcQc posted:message queues will not improve reliability of communication with other services, just make failures more obscure we've had this conversation before and you're not wrong but backpressure is real and not something you can always throw more api servers at
|
|
#
?
Mar 20, 2018 14:28
|
|
|
Sagacity posted:want to use message queues to reliably communicate with other (micro)services? well you're already hosed then
|
|
#
?
Mar 20, 2018 15:59
|
|
|
tef posted:well you're already hosed then i am living this hell
|
|
#
?
Mar 20, 2018 16:03
|
|
|
he’ll yeah queuechatBlinkz0rz posted:we've had this conversation before and you're not wrong but backpressure is real and not something you can always throw more api servers at what do you mean? backpressure is reducing the rate of production on the producer side and if you have it, you shouldn’t need a queue, yeah?
|
|
#
?
Mar 20, 2018 16:19
|
|
|
the closest i ever got to proper message queue infrastructure was working with tibco rendevouz, and that was simultaneously crude and awesome. awesome since the tooling was neat and you could just to listen in on flows of fairly structured data and figure out what was what, often as a result snapping existing infrastructure together into new systems in a pretty ad-hoc but understandable manner. the main advantage, however, was that it was loosely coupled enough that it was a sys/network admin task, so all routing, provisioning, monitoring, setting up repeaters and replay nodes, and a lot more such stuff was the work of the same people who configured switches and ran cables, which was a hugely valuable separation of concerns (mostly let us be idiots in peace, throwing data on made-up topics out into the swirling whirlpool) i am rather unsure whether that was just one of those rare encounters with competence rather than a lesson about how to build things however
|
|
#
?
Mar 20, 2018 16:23
|
|
|
We're currently switching a synchronous rest api to instead communicate asynchronously over kafka and it's seeming fine so far.
|
|
#
?
Mar 20, 2018 17:07
|
|
|
Blinkz0rz posted:we've had this conversation before and you're not wrong but backpressure is real and not something you can always throw more api servers at a queue is the exact opposite of backpressure
|
|
#
?
Mar 20, 2018 17:19
|
|
|
gonadic io posted:We're currently switching a synchronous rest api to instead communicate asynchronously over kafka and it's seeming fine so far. you’re hosed
|
|
#
?
Mar 20, 2018 17:59
|
|
|
MALE SHOEGAZE posted:you’re eventually consistently hosed
|
|
#
?
Mar 20, 2018 18:02
|
|
|
MALE SHOEGAZE posted:you’re hosed
|
|
#
?
Mar 20, 2018 18:03
|
|
|
gonadic io posted:We're currently switching a synchronous rest api to instead communicate asynchronously over kafka and it's seeming fine so far. let me know how the rest of 'reimplementing a database' goes
|
|
#
?
Mar 20, 2018 18:12
|
|
|
MALE SHOEGAZE posted:then we migrated all of our enterprise oracle database data onto it. what the christ why would you even think of this whyyyyyyyyyy I mean migrating away from Oracle: not unexpected migrating to mongodb: what the gently caress
|
|
#
?
Mar 20, 2018 18:34
|
|
|
eschaton posted:what the christ
|
|
#
?
Mar 20, 2018 18:35
|
|
|
queues aren't about handling failure they're about hiding failures which means that neither end of a queue has any idea if the other side is working also 'rest api' / 'restful api' has no real meaning and tbh never really did
|
|
#
?
Mar 20, 2018 18:35
|
|
|
if you replace "we will use a queue" with "we will use a shared buffer" then you may end up with less dangerous usage patterns out of not trusting a buffer all that much compared to the way people tend to trust an ~enterprise bus~ to do all the things
|
|
#
?
Mar 20, 2018 18:40
|
|
|
if message queues are bad what should you be doing instead
|
|
#
?
Mar 20, 2018 18:41
|
|
|
not using message queues. theres no problem they solve that isn't better solved by a database or a cache
|
|
#
?
Mar 20, 2018 18:42
|
|
|
— what do you mean you just took all of the data from all of the servers and you send it onto one single buffer? — well we wanted things to be easy to do so we thought that if we put all of it on one computer in one big memory space it would be nicer, all we need is for everyone to contact it all the time — what do you do when the buffer is full? — well technically we ought to return an error code or something but so far our plan is to monitor the buffer carefully and increase memory — okay and what do you do when that buffer is full on the biggest computer we can afford? — well we thought maybe we could either add buffers and then the client just needs to pick the one that works — okay and if that one is down or unavailable what happens? — ah well there, the client just needs to store things in their own local buffer, and send it to the big buffers only when they're ready — ... and so what would be the problem with removing the big shared buffers, and just keeping the client buffers anyway while preserving that retry logic on the client in the first place? — well the architecture diagrams get too many arrows on them
|
|
#
?
Mar 20, 2018 18:44
|
|
|
eschaton posted:what the christ i dunno, i wasn’t directly involved. terrible decisions all around
|
|
#
?
Mar 20, 2018 18:48
|
|
|
lol if you microservice in any way
|
|
#
?
Mar 20, 2018 19:05
|
|
|
we're looking at grafting an event system into our dumpster fire codebase, initially (and probably primarily) to support notifying external parties of stuff happening via webhooks our initial plan is to chuck events into a queue and let the webhook system pick them up from there. is a queue the wrong tool for this job? if it makes a difference, it's 99.9% going to be implemented using CREATE TABLE Queue in our existing db
|
|
#
?
Mar 20, 2018 19:11
|
|
|
Shaggar posted:not using message queues. theres no problem they solve that isn't better solved by a database or a cache Shaggar promoting Oracle Advanced Queue 
|
|
#
?
Mar 20, 2018 19:15
|
|
|
a queue in a db table is fine unless you don't care if the events get there in which case sticking in some in memory store to be handled best effort is ok.
|
|
#
?
Mar 20, 2018 19:15
|
|
|
MrMoo posted:Shaggar promoting Oracle Advanced Queue no. don't use anything w/ queue in the name
|
|
#
?
Mar 20, 2018 19:16
|
|
|
redleader posted:we're looking at grafting an event system into our dumpster fire codebase, initially (and probably primarily) to support notifying external parties of stuff happening via webhooks which db
|
|
#
?
Mar 20, 2018 19:23
|
|
|
mongodb
|
|
#
?
Mar 20, 2018 19:48
|
|
|
the insane 2d graphics proposal for c++ finally got presented to the wider committee at the most recent meeting and got dumpstered on:quote:Even though further work on this proposal was for six revisions, the room was quite brutal this time, and attempted to shut it down completely. Everyone said something along the lines of my quote above to justify the opposition, and experts from NVIDIA also expressed strong disagreement with the paper.
|
|
#
?
Mar 20, 2018 19:48
|
|
|
queue queue more
|
|
#
?
Mar 20, 2018 19:52
|
|
|
tef posted:which db shaggar-approved Microsoft (r) SQL Server (tm) Shaggar posted:no. don't use anything w/ queue in the name Shaggar posted:a queue in a db table is fine unless you don't care if the events get there in which case sticking in some in memory store to be handled best effort is ok. ok, now i'm confused. what's the difference between a queue using a db table and a queue using dedicated software? some of the fundamental issues (e.g. unbounded queue lengths) would be the same between the two, no?
|
|
#
?
Mar 20, 2018 22:41
|
|
|
Ploft-shell crab posted:he’ll yeah queuechat i mean i used the wrong terminology. i meant the issue where the data hose has been turned up to 11 case in point, the s3 outage today caused a bunch of backup from one of our services that lives in customer environments. they recovered when the outage ended but started spewing their data into the cloud as fast as possible. if it weren't for queues we wouldn't be able to keep up with transmission speed and data would have been lost. having a queue means we can scale consumers independently, keep messages on the queue and provide backpressure (term used correctly?) obviously it's a bandaid for a better data transmission system that respects transfer limits and doesn't murder downstream services but it sure beats the world burning down whenever there's an outage
|
|
#
?
Mar 20, 2018 22:48
|
|
|
redleader posted:shaggar-approved Microsoft (r) SQL Server (tm) the problem with using a db table as a queue is that you end up with lock contention between multiple consumers. one consumer does a select for update, locking the row it's working on. another consumer tries to dequeue something before the first one commits, finds a locked row, and sits around blocked. so you don't really get any concurrency out of the thing. the solution is to use "select for update skip locked" which does exactly what you want, and now your db table makes a fine queue, and you didn't even have to introduce another piece of infrastructure to maintain
|
|
#
?
Mar 20, 2018 22:56
|
|
|
backpressure is when overload of the receiving end pushes back on the sending end, slowing it down and equalizing the system (like if a server is at capacity, client requests take longer, reducing the potential load). a queue does the opposite of that - no information about load can flow back from consumer to producer, so your producers are happily able to vomit data at an uncontrollable speed.
|
|
#
?
Mar 20, 2018 22:56
|
|
|
DELETE CASCADE posted:didn't even have to introduce another piece of infrastructure to maintain but is this the reason people are complaining about queue's here?
|
|
#
?
Mar 20, 2018 23:01
|
|
|
redleader posted:shaggar-approved Microsoft (r) SQL Server (tm) skip locked is your friend quote:ok, now i'm confused. what's the difference between a queue using a db table and a queue using dedicated software? some of the fundamental issues (e.g. unbounded queue lengths) would be the same between the two, no? depending on the api you stick infront of it, nothing
|
|
#
?
Mar 20, 2018 23:04
|
|
|
Asymmetrikon posted:backpressure is when overload of the receiving end pushes back on the sending end, slowing it down and equalizing the system (like if a server is at capacity, client requests take longer, reducing the potential load). a queue does the opposite of that - no information about load can flow back from consumer to producer, so your producers are happily able to vomit data at an uncontrollable speed. yeah i used the wrong term :|
|
|
#
?
Mar 20, 2018 23:05
|
|
|
the issue is more about how error states are handled. purpose built message queues come with a ton of their own internal problems because they typically aren't just message queues, their some form of application bus. you end up putting application logic into the queue and you get this application/persistence hybrid system that sucks at both. when errors occur you have to figure out where in the pipe they're occurring. did a message get hosed up? is the queue distribution logic wrong? why isn't this endpoint available? who set this endpoint to retry forever? the best case scenario with a message queue is you don't use any of its features beyond basic message delivery and at that point its just a database of messages. a database that's probably not as good at being a database as sql server. let your db server hand db stuff and let your application handle application stuff.
|
|
#
?
Mar 20, 2018 23:10
|
|
|
Thermopyle posted:but is this the reason people are complaining about queue's here? its one reason. none of them are very good and they're a pain to maintain. sql server will do everything you should do in a message queue but better with no added headache of new maintenance overhead. adding 100 sql servers is 100000x easier than adding 1 message queue service
|
|
#
?
Mar 20, 2018 23:18
|
|
|
Shaggar posted:adding 100 sql servers is 100000x easier than adding 1 message queue service 
|
|
#
?
Mar 20, 2018 23:22
|
|
|
|
| # ? Jun 4, 2024 23:43 |
|
|
this thread gets shaggared again!
|
|
#
?
Mar 20, 2018 23:37
|
|