|
Shaggar posted:actually, out of curiosity how are you collecting their logs right now? do you hijack common logging libs and/or provide them an official heroku log api? Stdout (or whatever) data is piped to individual syslog services, which then push it to the router. Our router basically aggregates various standard syslog transports from different components in the system (app itself, routing logs, DB logs, etc.) and re-routes them to user-configured endpoints. These can be the command line 'heroku' thing (lets you see the recent ones, tail them live), done over an API call, you can ask the router to redirect the log streams to whatever endpoints you want (TCP Syslog, Syslog/HTTPs if you ask for it), in any combination whatsoever.
|
#
?
Sep 17, 2013 18:50
|
|
|
|
| # ? Jun 11, 2024 15:11 |
|
|
ah, so you're looking for a way to handle the redirect to a custom endpoint and what format and protocol that should look like?
|
|
#
?
Sep 17, 2013 18:53
|
|
|
if that's the case i think sticking w/ syslog is fine cause everything has a way to parse syslog. if you want to cleanup the stuff before it you're looking at a project to develop a common logging library for all hosted languages
|
|
#
?
Sep 17, 2013 18:55
|
|
|
Shaggar posted:ah, so you're looking for a way to handle the redirect to a custom endpoint and what format and protocol that should look like? Pretty much. Multiple endpoints, in fact. You could have 4 TCP-Syslog endpoints that go to different service and an HTTP one to your own special sauce thingy, while still using the CLI tools to read them live. Or you could have none and just shut it all down. The HTTPS pipe was added for specific users who felt their logs were somewhat sensitive and didn't feel safe sending them over TCP across different data centers. TCP/TLS syslog could have been used, but at this point I have no idea what went behind the decision making as it happened way before I joined. Shaggar posted:if that's the case i think sticking w/ syslog is fine cause everything has a way to parse syslog. if you want to cleanup the stuff before it you're looking at a project to develop a common logging library for all hosted languages Yeah afaik, they went with syslog especially because everything uses and supports it.
|
|
#
?
Sep 17, 2013 18:56
|
|
|
MononcQc posted:It's hairy for the custom logging protocol to be built on HTTP, but it shares some things for practical reasons. Syslog messages are individually stateless, and HTTP shares that. Syslog APIs are explicitly client-server, because that's how logs are produced and consumed, and HTTP shares that. if this is the case, where did your gigantic insane list of bullet points come from? because the system you just described is way simpler and probably inadequate to the requirements you specified earlier
|
|
#
?
Sep 17, 2013 18:58
|
|
|
tef posted:still, if your dataset fits in memory or a single disk then hadoop isn't likely going to be cost effective yeah if you can fit that poo poo into memory you load it up and run w/ it
|
|
#
?
Sep 17, 2013 19:07
|
|
|
fritz posted:yeah if you can fit that poo poo into memory you load it up and run w/ it Don't underestimate what fits in memory. You can get an x86 box with 3.0 TB of RAM now. That box is 3 TB of RAM + 40 cores inside 5 rack units. Pretty soon it will be 3TB + 48 or 64 cores.
|
|
#
?
Sep 17, 2013 19:11
|
|
|
Notorious b.s.d. posted:if this is the case, where did your gigantic insane list of bullet points come from? These can be user demands or requirements depending on their apps and use cases. The system I just described uses multiple protocols to support all/most of the bullet points from my list on a per-demand basis depending on what each customer needs. You can use whatever. Here's the breakdown and how these 2-3 transport formats manage to work right: quote:maintain long-lived connections (reconnecting is always slow) You don't implement all of these in a single protocol, you give a choice and let individual users make decisions based on what they need once they feel dissatisfied with the standard option (TCP-Syslog). I have then been told by this thread that giving user choices is a dumb thing and I should be ashamed and worked on problems better, and that the list of requirements above was an horror despite us being able to do an acceptable job at dealing with them in production. MononcQc fucked around with this message at 19:19 on Sep 17, 2013 |
|
#
?
Sep 17, 2013 19:15
|
|
|
Notorious b.s.d. posted:Don't underestimate what fits in memory. our 'big machine' at work has 32gb ram and a 1tb drive.
|
|
#
?
Sep 17, 2013 19:18
|
|
|
Shaggar posted:representation as in json vs xml is not what we're talking about. we're talking about representation of Person v1 vs representation of Person v2. those are located at different services. using accept to route them is retarded. using urls to route them is not. Not a fan of this av change, but nicely done whoever is responsible. Accept is the exact way that a client says 'hey i can understand v1 of a person in json, or a v2 of a person in xml, or just gimmie back some random text if that's the best we can do'. Why does the client have to know or care where a given implementation version of a service lives?
|
|
#
?
Sep 17, 2013 20:07
|
|
|
because only certain versions of the service understand person v1 or person v2. assuming that theres one service that understands everything is stupid. custom version routing using accept is retarded. urls are the correct way to do it.
|
|
#
?
Sep 17, 2013 20:12
|
|
|
 tef is shaggar and mononcQc is fishmech. wtf is happening in this thread!?
|
|
#
?
Sep 17, 2013 20:37
|
|
|
Brain Candy posted:
Didn't you get the headers, they negotiated a new version of posting but kept the names the same
|
|
#
?
Sep 17, 2013 20:41
|
|
|
Malcolm XML posted:Didn't you get the headers, they negotiated a new version of posting but kept the names the same lmao
|
|
#
?
Sep 17, 2013 20:45
|
|
|
Malcolm XML posted:Didn't you get the headers, they negotiated a new version of posting but kept the names the same
|
|
#
?
Sep 17, 2013 20:46
|
|
|
fritz posted:our 'big machine' at work has 32gb ram and a 1tb drive. let's do the math here:
if they can't afford 5% of your salary for development environments, how loving broken is their accounting? how many other things in the organization are this dysfunctional? if they value you so little they won't pay for even one virtualization server, it's very probable they just don't know how much your team costs. time to find a job where you can deliver real business value Notorious b.s.d. fucked around with this message at 21:02 on Sep 17, 2013 |
|
#
?
Sep 17, 2013 20:57
|
|
|
incidentally i used 48 month financing because that's what dell defaults to, i would have to actually apply for a loan to get a more realistic 36 month quote on the other hand, you can probably get that server for 60% of list if you buy more than 1 or 2 of them. it probably balances out in the end
|
|
#
?
Sep 17, 2013 20:59
|
|
|
that calculation is why in memory DBs are the future. RAM is so fuckin cheap
|
|
#
?
Sep 17, 2013 21:00
|
|
|
Nomnom Cookie posted:that calculation is why in memory DBs are the future. RAM is so fuckin cheap i don't know anyone whose database fits in 768 GB of RAM. but it *is* why SQL databases are still awesome in the age of "big data." 768 GB is more than enough cache to have a useful working set out of your 3 TB database
|
|
#
?
Sep 17, 2013 21:03
|
|
|
Notorious b.s.d. posted:i don't know anyone whose database fits in 768 GB of RAM. IIRC microsoft calculated that the working set of something like 90% of their "big data" calc was under 32GB
|
|
#
?
Sep 17, 2013 21:28
|
|
|
the next version of sql server has heck of ton which lets you put all ur poo poo in memory all the time and apparently uses some lockless system that still guarantees integrity. sounds like black magic, but we'll see.
|
|
#
?
Sep 17, 2013 21:34
|
|
|
Malcolm XML posted:Didn't you get the headers, they negotiated a new version of posting but kept the names the same Nice!
|
|
#
?
Sep 17, 2013 21:59
|
|
|
Brain Candy posted:
for a second I thought I actually had received a new avatar here
|
|
#
?
Sep 17, 2013 22:03
|
|
|
Nomnom Cookie posted:semantic versioning for apis gently caress me. /v2.4.5/boners/69/stroke /v2.4/boners/69/stroke /v2/boners/89/stroke
|
|
#
?
Sep 18, 2013 01:15
|
|
|
the best thing is dealing with combined versions. Client v2.3.7 deals with API v1.12.2, but so does v2.3.6 or something 
|
|
#
?
Sep 18, 2013 01:19
|
|
|
a good use for content negotiating is serving webp images to browsers that support them, and jpeg to browsers who don't. users want the image, not a file, and browsers can use any of the image formats they support interchangeably. (well, mostly. when Facebook tried it, users got confused because they were saving thumbnails and getting webp files. drat those implicit requirements!) i don't like using content negotiation for api versioning because 1. your special snowflake protocol probably doesn't need its own mime type. if you use a generic type like text/xml or application/json you can read it with browsers or generic adapters. 2. most formats you would be using (form-data, json, xml, protocol buffers, thrift) are extensible enough to support backwards compatibility by adding optional fields, or in the worst case, an explicit version field. any upgrade severe enough to require a format change is probably going to change your URI layout as well.
|
|
#
?
Sep 18, 2013 01:54
|
|
|
MrMoo posted:gently caress me. nope make the wildcards explicit
|
|
#
?
Sep 18, 2013 02:04
|
|
|
make the api purposefully return bad results or behave wrong as a built-in chaos monkey so that users learn to be fault tolerant
|
|
#
?
Sep 18, 2013 02:30
|
|
|
im pretty sure the storage people at work are already doing that
|
|
#
?
Sep 18, 2013 02:58
|
|
|
suffix posted:a good use for content negotiating is serving webp images to browsers that support them, and jpeg to browsers who don't. Regarding #1, just support both. I've usually had application/json or application/xml just return the latest version of the resource in json or xml respectively. Then everything works nicely in the browser and real clients who depend on specific versions can use your custom media type with the embedded version info in it. The rels for any links you embed in your resources will likely be vendor specific anyway (and they definitely aren't part of the application/json media type), so that seems to fit better with a custom media type.
|
|
#
?
Sep 18, 2013 04:23
|
|
|
Notorious b.s.d. posted:let's do the math here: I had to agitate for months to get that machine, prior to that I was doing everything on an 8g MacBook pro from 2011. just a couple weeks ago I discovered that they had several 64+gb servers in some hosting somewhere that had been sitting idle for literal years, right now I think they're stacked in a back room. my stock was supposed to vest today. well see how much they value me when I give two weeks notice tomorrow.
|
|
#
?
Sep 18, 2013 06:36
|
|
|
fritz posted:I had to agitate for months to get that machine, prior to that I was doing everything on an 8g MacBook pro from 2011. just a couple weeks ago I discovered that they had several 64+gb servers in some hosting somewhere that had been sitting idle for literal years, right now I think they're stacked in a back room. our cto is selling us as a big data company, right now people are all agitated because we're bringing a new thing online that'll come to like four (4) terabytes per year.
|
|
#
?
Sep 18, 2013 06:39
|
|
|
jesus loving christ can we get past http urls vs headers and talk about actual programming languages again here let me start: php is not a programming language, it's a set of functions that you can call, but probably should not
|
|
#
?
Sep 18, 2013 08:06
|
|
|
OK, I skimmed through most of the HTTP chat for the past few pages until I got to this one and just had to post, furiously swiping my thumb up the screen of my phone, so please excuse even less reading than usual, but: 1. I think lots of things should use BEEP. 2. Use a real man's protocol: PCIe Gen 3.0 x16. DMA bitch. If and/or how serious I am about either of these is left as an exercise to the reader.
|
|
#
?
Sep 18, 2013 13:03
|
|
|
use php
|
|
#
?
Sep 18, 2013 13:05
|
|
|
tripletef
|
|
#
?
Sep 18, 2013 13:17
|
|
|
i should have revived cheesebot and made an account for it :/
|
|
#
?
Sep 18, 2013 13:18
|
|
|
i have to poast onge again that i hate how every programming related course is so loving concerned about performance when discussing things like patterns. strategy pattern pros: you can change the implementation on the fly, cons: you have to call a function from a virtual function table  (which is what they mean by slightly decreased performance)
|
|
#
?
Sep 18, 2013 13:22
|
|
|
lol, wheany, so performance doesn't matter, well perhaps you have heard of a little thing called the travelling salesman problem. check-mate
|
|
#
?
Sep 18, 2013 13:24
|
|
|
|
| # ? Jun 11, 2024 15:11 |
|
|
Actual PL inquiry: has anyone here used Julia for anything (on Windows)? Does it run/install? I'm hoping it's a better situation than trying the current stable Rust builds on Windows but I haven't tried yet. Also, people interested in (simple, 2D) games programming should check out and play around with strlen's Lobster.
|
|
#
?
Sep 18, 2013 13:27
|
|