|
we use amazon sqs for lots of non-synchronous queueing tasks and it's needs suiting hosted solution so low maintenance burden, no logic in the queue itself like shagger's complaining about, simple read/process/acknowledge functionality in the consumers, retries N attempts and then all failed message get dumped to a dead letter queue for manual intervention, consumer load balancing group can scale on queue length or age of oldest message for bursts, cloudwatch alerts on queue length and dead letter queues for when poo poo breaks 
|
#
?
Mar 20, 2018 23:52
|
|
|
|
| # ? May 20, 2024 12:11 |
|
|
hate seeing application+ programmers using 'bus' tbh
|
|
#
?
Mar 20, 2018 23:56
|
|
|
JawnV6 posted:hate seeing application+ programmers using 'bus' tbh yeah and what's with electricians trying to be all trendy and hi-tech by calling things "bus bars" smh
|
|
#
?
Mar 21, 2018 00:00
|
|
|
JawnV6 posted:hate seeing application+ programmers using 'bus' tbh I know some people that agree with you: 
|
|
#
?
Mar 21, 2018 00:02
|
|
|
Sapozhnik posted:yeah and what's with electricians trying to be all trendy and hi-tech by calling things "bus bars" smh gonadic io posted:I know some people that agree with you:
|
|
#
?
Mar 21, 2018 00:20
|
|
|
Destroyenator posted:we use amazon sqs for lots of non-synchronous queueing tasks and it's needs suiting yeah this we previously used rabbitmq to do the same but the operating expenses was extremely high, especially during a network partition where it was literally "lol if u like to keep your data" sns/sqs is extremely needs suiting and doesn't provide half the pain rabbit did
|
|
#
?
Mar 21, 2018 00:22
|
|
|
my favorite part about sqs is how dequeueing a message doesn't actually dequeue it, it just sets the message invisible to other clients for a period of time, so you better hope processing the message doesn't take longer than that! our solution has been to explicitly delete the message from the queue immediately after dequeueing, handle errors on the application side, and re-enqueue the message if necessary. so we get none of that retry stuff or dead letter queue goodness, but wtf else are we gonna do
|
|
#
?
Mar 21, 2018 00:27
|
|
|
bad use of queue:code:code:
|
|
#
?
Mar 21, 2018 01:20
|
|
|
DELETE CASCADE posted:my favorite part about sqs is how dequeueing a message doesn't actually dequeue it, it just sets the message invisible to other clients for a period of time, so you better hope processing the message doesn't take longer than that! our solution has been to explicitly delete the message from the queue immediately after dequeueing, handle errors on the application side, and re-enqueue the message if necessary. so we get none of that retry stuff or dead letter queue goodness, but wtf else are we gonna do can you keepalive your lease? other than the issues you mention with your approach, a crash would cause messages to be silently dropped
|
|
#
?
Mar 21, 2018 02:49
|
|
|
JawnV6 posted:and one kid w/ a pellet gun caused so much more disruption Someone got mad at me for laughing when they said they had ptsd from that. Like, I used to see people get shot out front of my house bitch. Don't cry to me about your bb gun ptsd.
|
|
#
?
Mar 21, 2018 03:16
|
|
|
DELETE CASCADE posted:my favorite part about sqs is how dequeueing a message doesn't actually dequeue it, it just sets the message invisible to other clients for a period of time, so you better hope processing the message doesn't take longer than that! our solution has been to explicitly delete the message from the queue immediately after dequeueing, handle errors on the application side, and re-enqueue the message if necessary. so we get none of that retry stuff or dead letter queue goodness, but wtf else are we gonna do this is a really bad strategy you should tune your visibility timeouts and or renew them in flight if for some reason you’re taking forever to handle a message
|
|
#
?
Mar 21, 2018 03:31
|
|
|
cis autodrag posted:Someone got mad at me for laughing when they said they had ptsd from that. Like, I used to see people get shot out front of my house bitch. Don't cry to me about your bb gun ptsd.
|
|
#
?
Mar 21, 2018 03:55
|
|
|
Plorkyeran posted:the insane 2d graphics proposal for c++ finally got presented to the wider committee at the most recent meeting and got dumpstered on: you’re welcome also, got a link? I’m not finding the text you quoted eschaton fucked around with this message at 04:38 on Mar 21, 2018 |
|
#
?
Mar 21, 2018 04:30
|
|
|
eschaton posted:also, got a link? I’m not finding the text you quoted It hit pretty well on google for me with the first paragraph of text: https://vittorioromeo.info/index/blog/mar18_iso_meeting_report.html
|
|
#
?
Mar 21, 2018 05:10
|
|
|
thank you all for queuechat  we'll probably just implement a mediocre queue in the db and let ops sort it out
|
|
#
?
Mar 21, 2018 06:39
|
|
|
the talent deficit posted:bad use of queue: so...throttling?
|
|
#
?
Mar 21, 2018 11:29
|
|
|
I can’t believe you guys are suggesting putting a queue in a db
|
|
#
?
Mar 21, 2018 11:38
|
|
|
shagger and oracle can't BOTH be wrong
|
|
#
?
Mar 21, 2018 14:05
|
|
|
Sweeper posted:I can’t believe you guys are suggesting putting a queue in a db do any dedicated queues offer acid guarantees? 'cause the trouble is, i actually care about my data, so i put it in a db. if my data was social network spyware likes or mobile game high scores maybe i'd be more cavalier about losing a message here and there i guess
|
|
#
?
Mar 21, 2018 14:08
|
|
|
the talent deficit posted:bad use of queue: this wont work if the client cant keep their work until the queue is available again aka the sole reason to use a queue in the first place.
|
|
#
?
Mar 21, 2018 14:08
|
|
|
If you're able to carry backpressure and so on to the API in front of the queue, the queue might be an interesting tool to dispatch work and carry metrics, have some degree of persistence and so on. Nothing a proxy+consumers couldn't do on their own, but it's far less of a ticking time bomb into your system than just a queue alone, and removes a point of asynchrony that could otherwise cause problems in system behavior. The queue becomes an implementation detail of job management rather than the interface and core component by which it is done.
|
|
#
?
Mar 21, 2018 14:13
|
|
|
NihilCredo posted:do any dedicated queues offer acid guarantees? in the sense that they’ll dissolve your data and you shouldn’t touch them without protective equipment, yes
|
|
#
?
Mar 21, 2018 15:43
|
|
|
|
|
#
?
Mar 21, 2018 15:44
|
|
|
more like acrid guarantees!!
|
|
#
?
Mar 21, 2018 16:17
|
|
|
Apparently Java 10 is out now?  That was quick. Nothing terribly exciting in it though. http://www.oracle.com/technetwork/java/javase/10-relnote-issues-4108729.html
|
|
#
?
Mar 21, 2018 17:18
|
|
|
Sapozhnik posted:Apparently Java 10 is out now? they're moving to a firefox style schedule java 11 is going to be the long-term support release so i wouldn't expect to see much adoption by the big names until then
|
|
#
?
Mar 21, 2018 17:21
|
|
|
carry on then posted:they're moving to a firefox style schedule I.e. are they expecting that bundling a vm or using the AOT stuff is going to be the norm in the future?
|
|
#
?
Mar 21, 2018 17:28
|
|
|
mystes posted:Is this sort of an acknowledgement that java as a universal vm for desktop applications is pretty much dead yes, for those of you joining us from 1997 quote:I.e. are they expecting that bundling a vm or using the AOT stuff is going to be the norm in the future? java applications all run on servers. some will be deployed within containers running the latest and greatest upstream JVM base image, others will use whatever comes with the os (centos or ubuntu). co-installation of JVMs is already possible in traditional deployment contexts but with containers it becomes a non-issue.
|
|
#
?
Mar 21, 2018 18:01
|
|
|
AWWNAW posted:this is a really bad strategy didn't realize you could renew the visibility timeout for individual messages while in flight, ok we'll do that instead 
|
|
#
?
Mar 21, 2018 18:46
|
|
|
my favourite part about amazon sqs is the amazon techies completely dodging my questions re: 'is there any issue with only sporadically reading & purging a queue b/c your amazon rep keeps telling us the queue is clogged and we should purge it due to like 1k messages tops'
|
|
#
?
Mar 21, 2018 18:56
|
|
|
Shaggar posted:this wont work if the client cant keep their work until the queue is available again aka the sole reason to use a queue in the first place. what do you expect to happen if the queue can't accept new items and the clients try to add them anyways?
|
|
#
?
Mar 21, 2018 19:06
|
|
|
Simple = democratic Open = people's republic of
|
|
#
?
Mar 21, 2018 19:06
|
|
|
the talent deficit posted:what do you expect to happen if the queue can't accept new items and the clients try to add them anyways? I expect whoever setup the queue to be fired
|
|
#
?
Mar 21, 2018 19:08
|
|
|
Either the requests aren't important in which case you tell the clients to gently caress off until a synchronous handler can deal with them or the requests are important and you queue them somewhere for later processing. the entire point of the queue is to store critical requests that cant be immediately handled.
|
|
#
?
Mar 21, 2018 19:10
|
|
|
our queue based infrastructure is such a loving nightmare and i legitimately don't know how to fix it. it's also bad because we use kinesis instead of kafka so we only have 1 day of log retention. also even in our wildest success scenarios we'll probably never have more than a few thousand active clients. backpressure is a thing we will never experience.
|
|
#
?
Mar 21, 2018 19:15
|
|
|
Shaggar posted:Either the requests aren't important in which case you tell the clients to gently caress off until a synchronous handler can deal with them or the requests are important and you queue them somewhere for later processing. the entire point of the queue is to store critical requests that cant be immediately handled. given a queue is often a central point of failure, there is on average a higher probability that the queue is down as opposed to a majority of the servers or workers handling the requests.
|
|
#
?
Mar 21, 2018 19:17
|
|
|
Shaggar posted:Either the requests aren't important in which case you tell the clients to gently caress off until a synchronous handler can deal with them or the requests are important and you queue them somewhere for later processing. the entire point of the queue is to store critical requests that cant be immediately handled. either you just described backpressure ("you tell the clients to gently caress off until a synchronous handler can deal with them") and you agree with me or you think it's okay that the queue is just a single point of failure?
|
|
#
?
Mar 21, 2018 19:19
|
|
|
mystes posted:Is this sort of an acknowledgement that java as a universal vm for desktop applications is pretty much dead, so there's no need to follow a schedule that is based around the assumption that everyone is going to switch everything over to the new version all at once at the same time? yes, one of the big concepts behind java 9 modules was to make bundling a jvm less terrible by letting you only bundle the parts of it that your software uses of course I still don’t know of anyone who’s managed to switch to java 9 yet, let alone considering 10. we’re certainly stuck on 8 for the time being. that compatibility break was pretty bad.
|
|
#
?
Mar 21, 2018 19:28
|
|
|
yeah, the compatibility break was pretty bad, and the lack of anything really compelling outside of modules made uptake even worse even our bleeding edge product isn't going to support java beyond 8 until 11, and our legacy product that people actually use is probably not going to ever work with java 9+
|
|
#
?
Mar 21, 2018 19:54
|
|
|
|
| # ? May 20, 2024 12:11 |
|
|
the talent deficit posted:either you just described backpressure ("you tell the clients to gently caress off until a synchronous handler can deal with them") and you agree with me or you think it's okay that the queue is just a single point of failure? you cant have backpressure and queues. they're incompatible. either you have important data that needs to be queued in which case you can never under and circumstance tell the clients to gently caress off or you don't have important data and you can tell your clients to gently caress off at which point you should not be using a queue because it does not help you. just have the clients talk directly to the processor that the queue is feeding. the queue is a single point of failure in theory, but you already had a single point of failure in the processor that the queue is feeding. the idea is that the queue is more reliable and available than whatever is doing the processing and you are moving the single point of failure from the processor to the queue.
|
|
#
?
Mar 21, 2018 20:07
|
|