|
MALE SHOEGAZE posted:at my startup our architecture is microservices and a message queue.
|
# ? Aug 9, 2017 08:30 |
|
|
# ? Jun 6, 2024 05:44 |
|
tef posted:needs more containers and to be ported to kubernetes by next month tia also a service discovery layer using a state-of-the-art distributed key value database implementing a sophisticated consensus algorithms with a 500-line bespoke configuration file in order to share three strings.
|
# ? Aug 9, 2017 08:43 |
|
The thing I find interesting about the use of gRPC at Google is that from their SRE book, the entire set up requires an actively managed cluster, a very smart router dispatching requests to a fully managed kubernetes-like back-end of workers on a one-to-one basis, and so on. In the end it sounds like they're transforming the whole thing into what looks a lot more like AWS' Lambda than your traditional RPC set up; they just used RPC as the building block for it. Then you have an army of folks going in there and hand-tuning timeouts, because RPCs calling RPCs calling RPCs turn out to have very weird and tricky timeout challenges and semantics when it comes to time limitations and cancellation. Blog posts from google point out a need to respect special conventions with unified internal interfaces that let such values be harmonized across the stack (a first argument 'context' that must be weaved into all functions on the call path). The unmentioned challenges of getting that kind of RPC architecture to work at their best in practice are kind of scary. Maybe the next RPC will be better. MononcQc fucked around with this message at 12:13 on Aug 9, 2017 |
# ? Aug 9, 2017 12:10 |
|
i didn't read much of that article but it immediately conflates the actual rpc library programming model with the semantics of rpc protocols. i don't really give a poo poo about whether a given rpc is represented with a synchronous stub call or whether a message is built up incrementally and gets explicitly submitted to yield a promise that can be chained. grpc tends to look more like the latter if anything. retry and rollback semantics and atomicity guarantees and whatever don't go away if you simply pretend they don't exist. that's not an argument for or against rpc. coarse messages between well-separated concerns that minimize round trips and rollback complexity seem like they ought to be fine. deeply chained and branching call stacks sliced across tens of machines seem like they'd have the problems you describe yeah. a higher-level abstraction would be useful there. but hey, it's google and they employ some of the best distributed systems people in the world. i'm sure they have some sort of improvements in mind here.
|
# ? Aug 9, 2017 14:47 |
|
MALE SHOEGAZE posted:at my startup our architecture is microservices and a message queue. de my couple hole
|
# ? Aug 9, 2017 15:09 |
|
your org has it figured out: if your system can't be used it can't be broken
|
# ? Aug 9, 2017 15:21 |
|
Sapozhnik posted:retry and rollback semantics and atomicity guarantees and whatever don't go away if you simply pretend they don't exist. that's not an argument for or against rpc. This is not a pro-HTTP argument; HTTP can and would make for an often fairly lovely mechanism, but HTTP did the distsys poo poo better than most RPC systems do, even if there have been like 3 HTTPs, but over a dozen RPC protocols. Maybe the 24th RPC iteration will get there. It appears Finagle has a way to mark some failure types as retry-friendly now! Sapozhnik posted:deeply chained and branching call stacks sliced across tens of machines seem like they'd have the problems you describe yeah. a higher-level abstraction would be useful there. but hey, it's google and they employ some of the best distributed systems people in the world. i'm sure they have some sort of improvements in mind here. Until google have their improvements, I don't really feel it's that great of a model to migrate to use in all the places that don't employ "some of the best distributed systems people in the world" without the full caveats and architecture workarounds google required implementing for any level of non-trivial services. I mean for gently caress's sake, I went to a conference about reactive applications last year, and while half the presentations were about how Kafka got people to remove so many arrows from their architecture diagrams and replace them by one big Kafka box with a logo on it, at least a quarter of them were just about how to manage all the RPCs and other remote calls that were starting to be blocking and surprised everyone with the big nasty tail latencies when they used microservices. Microservices just make the problem much more apparent.
|
# ? Aug 9, 2017 15:48 |
|
i just fire http requests everywhere and it seems like it works pretty well, op
|
# ? Aug 9, 2017 16:38 |
|
in my api i am logging the data i send. the problem is some of these data sets are so big that serializing the data to the log it is causing out of memory exceptions i do need to log this so when poo poo breaks i can prove it's not my fault and it's super helpful in debugging too, i guess
|
# ? Aug 9, 2017 16:55 |
|
uhhh most programs don't run with logging set to TRACE for a reason
|
# ? Aug 9, 2017 16:56 |
|
HoboMan posted:in my api i am logging the data i send. the problem is some of these data sets are so big that serializing the data to the log it is causing out of memory exceptions Maybe cherry pick some fields from that blob that you know are relevant to your service metrics and can help you debug poo poo?
|
# ? Aug 9, 2017 17:06 |
|
HoboMan posted:in my api i am logging the data i send. the problem is some of these data sets are so big that serializing the data to the log it is causing out of memory exceptions convert your data to protobufs and then stream it to your logging micro service
|
# ? Aug 9, 2017 17:23 |
|
the problem with rpc as a design is that it encourages programmers to think of the action as something fairly simple and reliable it also encourages programmers to write code with horrible latency because it’s so easy to serialize things that really could be happening in parallel, but async/await has that problem, too
|
# ? Aug 9, 2017 17:35 |
|
i think on reading that rpc (and grpc) has nothing to do with what i wanted for my side project at work so oops I'm being switched to a different group soon though so I guess it's irrelevant now anyways~ it was still interesting to learn about though.
|
# ? Aug 9, 2017 18:39 |
|
i keep tcpdump running on every server and have it stream to kinesis for processing. of course those generate more tcp data so i'm hitting the 500 shard limit so i had to make some API calls to amazon to dynamically stand up more streams for each server. parsing the data in wireshark requires massive amounts of memory and is slow as dirt, but at least i can prove that it's not my fault when a server throws a 500 error.
|
# ? Aug 9, 2017 19:04 |
|
CRIP EATIN BREAD posted:i keep tcpdump running on every server and have it stream to kinesis for processing. you can try tshark for a lighter-weight command-line version of wireshark (it ships with it). You can use the same filters as you would in the GUI, but usually it struggles a lot less at handling huge dumps.
|
# ? Aug 9, 2017 19:10 |
|
MononcQc posted:I went to a conference about reactive applications last year, and while half the presentations were about how Kafka got people to remove so many arrows from their architecture diagrams and replace them by one big Kafka box with a logo on it, at least a quarter of them were just about how to manage all the RPCs and other remote calls that were starting to be blocking and surprised everyone with the big nasty tail latencies when they used microservices. Microservices just make the problem much more apparent. Would you say that "it makes the terribleness of your systems more obvious" is a benefit of micro services? (I am a baby coder who probably won't make decisions that big for years, so don't feel like you need to write an essay here lol)
|
# ? Aug 9, 2017 19:28 |
|
MononcQc posted:you can try tshark for a lighter-weight command-line version of wireshark (it ships with it). You can use the same filters as you would in the GUI, but usually it struggles a lot less at handling huge dumps. i hope you know i was joking
|
# ? Aug 9, 2017 20:49 |
|
i loving get it, but the dev on the other side of that api call is a little poo poo that refuses to even look onto a bug unless you can empirically prove that his poo poo is what hosed up
|
# ? Aug 9, 2017 21:10 |
|
CRIP EATIN BREAD posted:i hope you know i was joking eeh, I've seen similar approaches before, so it was a stretch. Like people replaying/rewriting packets on the fly so that a staging/pre-production stack gets actual production data coming in. Kinesis was a dumb thing though because iirc they have like 5 qps limits by default and that would just not be possible without paging log files
|
# ? Aug 9, 2017 21:16 |
|
Lutha Mahtin posted:Would you say that "it makes the terribleness of your systems more obvious" is a benefit of micro services? (I am a baby coder who probably won't make decisions that big for years, so don't feel like you need to write an essay here lol) It's a benefit if you know you can fix it, have the time to do it, and are going to stick with a microservice architecture out of need already. In a lot of cases, you just have people who want microservices because that's the new cool thing, but for whom a monolith would work really really well for many years. In this case, they're giving themselves distributed systems problems they could avoid by scaling vertically for a long time. OTOH, it's good to keep these problems in mind because they can impact product decisions in major ways so you can avoid them (some things that work locally just aren't possible with large distributed systems) if you know you're gonna get real big.
|
# ? Aug 9, 2017 21:19 |
|
MononcQc posted:eeh, I've seen similar approaches before, so it was a stretch. Like people replaying/rewriting packets on the fly so that a staging/pre-production stack gets actual production data coming in. Kinesis was a dumb thing though because iirc they have like 5 qps limits by default and that would just not be possible without paging log files Generating pcap logs is a sensible option for trading environments when you don't want latency from inline logging. With OpenOnload it becomes near-free like a port mirror.
|
# ? Aug 9, 2017 21:38 |
|
suffix posted:https://deadlockempire.github.io/ these were fun little games, but the ‘boss fight’ at the end is a bear. everything else was pretty straightforward, get one thread in the right place and cycle the aggressor, but that last one is the kind of failure you’d catch with batches of tests not inspection
|
# ? Aug 9, 2017 21:51 |
|
i put an executable up for one of my github projects and was able to download and run it on another computer feeling a little less terrible today
|
# ? Aug 9, 2017 23:45 |
|
Shaman Linavi posted:i put an executable up for one of my github projects and was able to download and run it on another computer i uploaded my xcode project, tried to download it again and xcode crashed every time it tried to load rip
|
# ? Aug 10, 2017 00:34 |
|
JawnV6 posted:these were fun little games, but the ‘boss fight’ at the end is a bear. everything else was pretty straightforward, get one thread in the right place and cycle the aggressor, but that last one is the kind of failure you’d catch with batches of tests not inspection actually, you'd catch it by inspection by noting that the critical section isn't actually guarded by the monitor. who gives a poo poo if you can't figure out the exact sequence that breaks it immediately when the underlying flaw is obvious? if you're relying on tests to catch a nondeterministic failure you're going to have a bad time at some point
|
# ? Aug 10, 2017 02:37 |
|
Jabor posted:actually, you'd catch it by inspection by noting that the critical section isn't actually guarded by the monitor. who gives a poo poo if you can't figure out the exact sequence that breaks it immediately when the underlying flaw is obvious? and yes, i have relied on tests to catch nondeterministic failures and you have used products that resulted from this methodology
|
# ? Aug 10, 2017 03:45 |
|
critical sections are the worst mechanism for multithreading that I can think of
|
# ? Aug 10, 2017 04:31 |
|
I GOT IT AAAAAAAA https://twitter.com/LuigiThirty/status/895525296505364480
|
# ? Aug 10, 2017 07:01 |
|
wooooo
|
# ? Aug 10, 2017 09:05 |
|
is there some kind of an universal sentence tokenization library that will spit out a stream of words from any text, written in any script? i know doing that correctly is Really Hard, but is there something that produces stable results, even if incorrect? i want to try some bayesian classification on tweets as an idiot side project, and it would be nice if it's not hardcoded to support only languages that separate words with whitespace. is there a word for what i'm looking for? (a unicorn lol)
|
# ? Aug 10, 2017 09:30 |
|
tps: i will never get over the fact that grails can't count how many unit tests it has so it reportscode:
|
# ? Aug 10, 2017 13:51 |
|
Wheany posted:is there some kind of an universal sentence tokenization library that will spit out a stream of words from any text, written in any script? somethin like this? https://nlp.stanford.edu/software/lex-parser.shtml although being probabilistic it may not be stable HoboMan fucked around with this message at 14:48 on Aug 10, 2017 |
# ? Aug 10, 2017 14:45 |
|
Wheany posted:is there some kind of an universal sentence tokenization library that will spit out a stream of words from any text, written in any script? nltk is probably your best bet quote:is there a word for what i'm looking for? (a unicorn lol) segmentation ? https://en.wikipedia.org/wiki/Text_segmentation#Word_segmentation
|
# ? Aug 10, 2017 18:27 |
|
tokenizer is the word i think we are looking for like this? https://nlp.stanford.edu/software/tokenizer.shtml
|
# ? Aug 10, 2017 18:45 |
|
i think it's finally time to learn c because unsafe rust is just confusing without a c background.
|
# ? Aug 10, 2017 18:59 |
|
icu's word split iterator is sort of surprisingly good, and even supports a few languages that don't separate words with whitespace (by having a giant dictionary)
|
# ? Aug 10, 2017 19:31 |
|
Jabor posted:if you're relying on tests to catch a nondeterministic failure you're going to have a bad time at some point tests are really good at finding non-deterministic failures, just not the one-shot unit test kind. if you haven't read through the jepsen blog, it's worth a look: https://jepsen.io/analyses it's all randomized property testing, and it's basically state-of-the-art for ensuring real-world concurrent systems actually work.
|
# ? Aug 10, 2017 19:40 |
|
MALE SHOEGAZE posted:i think it's finally time to learn c because unsafe rust is just confusing without a c background. C is a good language to learn because it demonstrates the low-level inner-workings of operations that modern higher-level languages have abstracted away and also it teaches you how much you enjoy not having to write in C.
|
# ? Aug 10, 2017 19:57 |
|
|
# ? Jun 6, 2024 05:44 |
|
Yeah unit tests can hide the concurrency problems in a piece of software that is divided into lots of modules. I remember writing an test that spun up hundreds of clients hitting the same endpoints to prove that there was a race condition in our code that was using row level locking incorrectly, and the bug would pop up consistently and faster the more clients I added. Then once I applied a possible fix I used the same test to show that this particular problem was fixed.
|
# ? Aug 10, 2017 20:01 |