|
Way back when I took a course in Prolog, and I've always wanted to pick Erlang up to have a more practical use for that stuff. I threw 5.3 into a file and compiled it, and it always kills the interpreter after around 130-140k recursions. Am I running into some kind of memory limit or infinite recursion protection for the interpreter or something? It should be tail recursive, so I'm not sure why it's crashing. code:
code:
|
# ? Jul 22, 2013 23:45 |
|
|
# ? May 10, 2024 00:28 |
|
Paul MaudDib posted:Way back when I took a course in Prolog, and I've always wanted to pick Erlang up to have a more practical use for that stuff. I've just tried this snippet. It works fine (went well over 10 times your problem) on R15 and R16 versions of the VM. I'm pretty dumbfounded that something like that would kill your node. How did you get Erlang installed? Do you have any code being loaded in a .erlang file in your home directory?
|
# ? Jul 27, 2013 14:26 |
|
I stole this entry from the PL thread in YOSPOS where I posted it yesterday, and it fits fairly well in here. For the last couple of weeks, I've been playing a game of optimizing and hunting down all kinds of bottlenecks, memory hogs, processes with too many messages, analyzing crash dumps and whatever om some of our systems so that we'd finally get rid of pretty much 99% of non-actionable alerts over some clusters. It's the kind of poo poo you end up to do on any drat software stack every once in a while once it gets to be under load for a good while in production, and is fairly unavoidable as an activity. What changes, though, is how you do it. One thing I wanted to do, for example, is figure what connections (out of more than 20,000) end up taking most of the bandwidth on the server at any given time, either on input or output, to be able to characterize the extremes of the load we may receive. Doing this in many languages or systems would generally require to wrap whatever socket operations are being done in a counter of some sort if it's not done already, and then poll or log the results somewhere. If your application is mostly IO bound over the network, logging that data for arbitrary levels of precision can have a rather damaging effect on the quality of service. Erlang has this nice thing where stats are automatically accumulated for all socket activity, in advance. Every Erlang socket is called a port, and they're a language construct that wraps a socket, a file descriptor, or whatever else in something that looks like an Erlang process to the rest of the language. The statistics for any of them are available by calling a function called inet:getstat/1 on any given port. Moreover, this function can be combined with some other calls to list all the ports around. By doing this, you can get the data for all the ports in a program from the Erlang shell, ask for how much data they've seen, make a list and sort it up. You could also wait for a few milliseconds, make a second list, make a difference between both, and get a snapshot of how what ports saw the most data at any interval of time. I ended up writing such functions and putting them in a small library called recon. I wrote both functions in a short while, loaded the module and could find the biggest culprits in a matter of seconds by calling recon:inet_window(Metric, Number, IntervalToMeasure): code:
code:
code:
The fun thing about these inspection functions to make time windows and whatnot is that they're usable for anything else. So if you're looking for processes leaking memory, you can either look at them in the absolute or over a sliding time window: code:
Oh yeah, and all of this can be done remotely over entire clusters to find the worst everywhere: code:
The best thing about all of that is that it's non-blocking, read-only, and generally safe to run on any number of production nodes remotely without impacting quality of service at all. I don't know how many other languages can give you that kind of run-time introspection, but I know I feel like poo poo every time I need to go back to "reproducing poo poo locally" or "debugging via logging or printf". poo poo is much harder and requires an ungodly amount of redeploys, compared to just digging into a running system for whatever information you need, even across a cluster to give you more data points. Of course if you had a decent stats/graphing/reporting system in place already, and that all the data you need is in there, that's even better, but you're likely not going to get the same level of granularity. So far I'm pretty happy with Recon as a library and I'm trying to inject it into more work projects this post brought to you by your local department of Erlang propaganda.
|
# ? Aug 13, 2013 12:30 |
|
MononcQc posted:...Cool poo poo... Post this in your blog. E: Pressed post too soon: What projects do people use Erlang in? By that I mean, what happened that made you go "I need Erlang for this!"? Workaday Wizard fucked around with this message at 16:25 on Aug 13, 2013 |
# ? Aug 13, 2013 16:22 |
|
Shinku ABOOKEN posted:Post this in your blog. From Justin Sheehy, CTO at Basho: quote:We use the Erlang/OTP programming language in building our products here at Basho. We made that choice consciously, believing that it would be a tradeoff – significant benefits balanced by a handful of costs. I am often asked if we would make the same choice all over again. To answer that question I need to address the tradeoff we thought we were making.
|
# ? Aug 13, 2013 17:21 |
|
Shinku ABOOKEN posted:Post this in your blog.
|
# ? Aug 13, 2013 19:53 |
|
Shinku ABOOKEN posted:Post this in your blog. Maybe I'll add it to my blog, reword it and whatnot. It's been a while since I've been truly excited about using one of my libs for myself, rather than doing it then moving on. I've used Erlang for:
Most use cases I've made of Erlang had a few common points in terms of massive concurrency, a lot of time spent over heavy load, strict time constraints, requiring to be always up (shutting down to upgrade means losing money/user data). I've been served well so far. Cocoa Crispies posted:From Justin Sheehy, CTO at Basho: ^ this is my favorite bit of the whole thing. E: fixed a link to the wrong blog post MononcQc fucked around with this message at 03:38 on Aug 14, 2013 |
# ? Aug 14, 2013 03:17 |
|
As I've (slowly, mea culpa) read LYSE and discussions/papers here and in the PL thread, I've started coming around to the view that there is a lot of overlap between building classical distributed systems and building industrial embedded systems (stuff with hard reliability requirements, eg. SIFs). I'm surprised there isn't more on the web relating the two, but then again I am a crazy dude that thinks there is a fundamental equivalence between statistical signal processing and classical frequency-domain signal processing vv
|
# ? Aug 14, 2013 03:30 |
|
Thanks for the replies.MononcQc posted:[*] Real Time Bidding software, because low-latency requirements (soft real time), massive levels of concurrency, and constant system overload (which Erlang rules at) I don't know why I was under the impression that Erlang couldn't do low latency stuff. Glad to hear it does.
|
# ? Aug 14, 2013 03:31 |
|
Otto Skorzeny posted:As I've (slowly, mea culpa) read LYSE and discussions/papers here and in the PL thread, I've started coming around to the view that there is a lot of overlap between building classical distributed systems and building industrial embedded systems (stuff with hard reliability requirements, eg. SIFs). I'm surprised there isn't more on the web relating the two, but then again I am a crazy dude that thinks there is a fundamental equivalence between statistical signal processing and classical frequency-domain signal processing vv I'd like to hear about that more. FWIW, there's work to make Erlang more available for bigger 'embedded' platforms (http://www.erlang-embedded.com/). There's also a German guy I know who uses Erlang in hard real-time systems for the automotive industry by using it in RTEMS (a real time OS). The gist of his thing is that he writes the hard-real time components in C, C++, or even Ada, and gives them a priority higher than Erlang. Because they're usually smaller core components, he can then run everything that is soft-real or lower priority on the same embedded OS with a lower priority, within the Erlang VM. There's been other interesting work in the topic, trying to make the Erlang VM work for hard real time, but I haven't heard about it in a long while and it's frankly outside of my area of expertise.
|
# ? Aug 14, 2013 03:47 |
|
Shinku ABOOKEN posted:Thanks for the replies.
|
# ? Aug 14, 2013 04:47 |
|
Rapsey posted:Goldman Sachs uses erlang in their high frequency trading platform. With that and a bunch of different languages, for different HFT platforms :p
|
# ? Aug 14, 2013 13:02 |
|
Shinku ABOOKEN posted:What projects do people use Erlang in? By that I mean, what happened that made you go "I need Erlang for this!"? I wanted to use Erlang in my last project, a distributed web crawler. Unfortunately, Erlang wasn't on the cards. So I poorly reimplemented process supervision in python
|
# ? Aug 15, 2013 16:37 |
|
Any opinions on Elixir? Is it performant/stable enough to be worth learning in addition to/in place of Erlang at this point? For reference, Elixir's a new-ish language implemented on top of the Erlang VM, with a Ruby-ish syntax.
|
# ? Aug 20, 2013 05:15 |
|
The Insect Court posted:Any opinions on Elixir? Is it performant/stable enough to be worth learning in addition to/in place of Erlang at this point? From last page: MononcQc posted:Elixir is not adding too much to Erlang, IMO. Its biggest contributions are in macros, multiple modules per file, and the ability to have contracts, but otherwise most of its features will be a variation of something available in Erlang through the BEAM VM and Core Erlang (the intermediary language many can compile to, if they don't just generate an Erlang abstract parse tree), and its weaknesses will likely be a similar variation. I still stand by that position. I'm interested to see how Elixir grows and how the community goes about it. I've privately held the theory for a while that Erlang is a 'different' language and that different syntax helps people drop the baggage they usually carry around (the same way it's obvious when a C programmer programs C in C++, it's obvious when a OO programmer programs OO in Erlang). I'm very eager to see how people who adopt Elixir for its friendlier syntax will deal with the [relatively] more surprising semantics of the language.
|
# ? Aug 20, 2013 06:14 |
|
MononcQc posted:words about Elixir
|
# ? Aug 20, 2013 08:15 |
|
Elixir is nicer than Reia. Reia was an attempt to make an entirely new language with new semantics -- object-oriented with each object being its own process. Compared to that, Elixir keeps its semantics closer to Erlang (not OO, and processes are used similarly as Erlang, not as Reia did it).
|
# ? Aug 20, 2013 08:47 |
|
MononcQc posted:Elixir is nicer than Reia. Reia was an attempt to make an entirely new language with new semantics -- object-oriented with each object being its own process. The object-as-process model is pretty much the actor model though; I wonder what might have come out of the Reia folks looking at something like E and still targeting a Ruby-ish syntax...
|
# ? Aug 21, 2013 02:21 |
|
Erlang is sometimes said to be object-oriented in the original meaning of it (each process acts as an object communicating through message passing), but you'll hit a wall if that's how you approach things. Erlang's processes are meant as a way to separate individual components to provide fault-tolerance; not to compose them and have them interacting on a level as low as function calls all the time. Representing a list or a tree node as a process is useless, while they could very well be objects in any OO language. Erlang's processes are a way to provide fault-tolerance first. This can be tolerance to some weird hardware failure, a programmer error, corrupted data, etc. That an OO-like system emerges from it is purely accidental. We can think of it as "OO done right" if we want, but using it in practice as if it were truly OO will likely lead to a shitload of unwarranted friction that would have been avoided by using a functional style over data structures, and keeping processes as isolated small programs that can talk to each other with messages. That being said, Reia eventually died off and got abandoned after its author tried to get ruby blocks into Erlang and being told 'no' by members of the community, most notably by the well-informed Richard O'Keefe[1][2][3][4][5]. This discussion, and the appearance of Elixir, prompted Tony Arcieri to declare that Erlang is a ghetto and he left the community to work on Celluloid, which tries to bring Erlang to Ruby, rather than his former approach of bringing Ruby to Erlang.
|
# ? Aug 21, 2013 06:04 |
|
quote:However, the problem with Erlang's fun syntax is, well, it isn't fun. Ton'y a great guy and a good friend of mine but let's just say keyboards aren't the only buttons he's good at pressing.
|
# ? Aug 21, 2013 21:57 |
|
MononcQc posted:Sorry for the thread necromancy, but I was wondering if anyone in here planned to be around for the Erlang Factory Lite in September, in New York City? I'm not a frequent poster to the forums, but I'll be there and it would be fun to say hi. I've only just started learning Erlang, and your book was highly recommended by my coworkers.
|
# ? Aug 29, 2013 19:21 |
|
Mniot posted:I'm not a frequent poster to the forums, but I'll be there and it would be fun to say hi. Did you have coworkers at Erlang DC?
|
# ? Aug 30, 2013 04:17 |
|
Mniot posted:I'm not a frequent poster to the forums, but I'll be there and it would be fun to say hi. Nice! Good to hear. --- Oh and I can't believe I forgot to post this here, but I'm giving a free webcast for O'Reilly on Tuesday on Modern Server Application Design with Erlang for a high-level tour of building Erlang apps and how that compares to traditional things, then some general design ideals to keep in mind when using Erlang for that. I hope it's gonna be good, although I'm still working on it as I type this.
|
# ? Aug 30, 2013 13:06 |
|
MononcQc posted:Nice! Good to hear. Does O'Reilly archive webcasts? I can't attend this one
|
# ? Aug 30, 2013 13:41 |
|
MononcQc posted:I'd like to hear about that more. It's been a while, but I didn't forget this I've been real busy, but in my spare mental cycles I've been thinking more about it and to cut a long story short I think the similarity I saw boils down to a) some systems, especially SIFs, being actual distributed embedded systems of the AP variety even though they aren't normally described that way and b) unreliable communication even in systems that don't look "distributed", where it is impossible to discern whether there is excessive latency on a bus or whether the chip you're trying to talk to is transiently failing or really broken in the "magic smoke is gone" way. I will try and elaborate on this Saturday or Sunday
|
# ? Aug 30, 2013 14:00 |
|
Cocoa Crispies posted:Did you have coworkers at Erlang DC? Unlikely; they're mostly in California. I'm in the Boston area, myself. I know they went to the last SF conference.
|
# ? Aug 30, 2013 14:47 |
|
Shinku ABOOKEN posted:Does O'Reilly archive webcasts? Other talks that were free seem to have been made public (there's a couple of Haskell ones), so in the best of cases, it should be archived and made available. I don't know the details though.
|
# ? Aug 30, 2013 15:11 |
|
I'm looking to parse HTML and RSS feeds. It looks like mochiweb_html is the goto html parser, but is there anything more standalone that doesn't require bringing in mochiweb? For the RSS, is there a specific RSS library out there, or should I just stick to xmerl?
|
# ? Sep 6, 2013 12:37 |
|
more like dICK posted:I'm looking to parse HTML and RSS feeds. It looks like mochiweb_html is the goto html parser, but is there anything more standalone that doesn't require bringing in mochiweb? For the RSS, is there a specific RSS library out there, or should I just stick to xmerl? I went with mochiweb_html for my HTML parsing requirements personally, but I didn't research it very hard. It's annoying and you'll need to fetch the repository, so what you can do is either just import that one module (which could create conflicts if you're writing a library), or you import the whole drat thing and later on fix it with releases where you tell Reltool to just bring in a single file from that app. That assumes you're willing to build releases with Reltool or Rebar, though. For XML, don't use xmerl. xmerl has a thing where all tags get to be transformed into Erlang atoms, which are not garbage collected, and that can be used as an attack vector to bring your nodes down. It's really dumb like that, and I'm not sure why it's still part of the standard library without a huge warning. Go use erlsom instead. It's safer, and seems to work reasonably well. Regarding RSS, you'll have to deal with datetime support in there too. I've used dh_date in my webcast demo, but it doesn't deal with timezones. Instead I've recently found qdate, and while I haven't tried it, it seems to deal with them much better. A quick look at the code tells me high-volume requests would probably hammer its central server a bit, but that would be somewhat easy to refactor if you ever needed it to be done. MononcQc fucked around with this message at 13:08 on Sep 6, 2013 |
# ? Sep 6, 2013 13:05 |
|
This is just for a standalone app, so I suppose I can just cheat and put mochiweb_html alongside my own sources without having to muck around with even more Rebar config. Thanks for the heads up on erlsom and qdate.
|
# ? Sep 6, 2013 14:18 |
|
I've got a ram_copies Mnesia table on a node, and I'd now like it to be duplicated on another node as well. It looks like this is easy enough to do by connecting to the first node, and running mnesia:add_table_copy(mytable, Node2, ram_copies). My main concern is that I've got about 1.5GB in the table at any given time, and that data now needs to be moved to Node2. How does Erlang manage this? Will it saturate the network or degrade the connection between the nodes in any way? edit: DO just announced private networking a couple hours after I made this post Posting Principle fucked around with this message at 16:39 on Sep 10, 2013 |
# ? Sep 10, 2013 14:53 |
|
What're the consequences if the two tables get out of sync due to latency, slow node startup, or other issues?
|
# ? Sep 10, 2013 17:07 |
|
I redid this in a way that is less dumb. The only thing I'm worse at than Erlang is design.
Posting Principle fucked around with this message at 03:05 on Sep 11, 2013 |
# ? Sep 10, 2013 17:39 |
|
more like dICK posted:This is just for a standalone app, so I suppose I can just cheat and put mochiweb_html alongside my own sources without having to muck around with even more Rebar config. Thanks for the heads up on erlsom and qdate. This is what I do, and it's worked out really well for me. I'd still be interested in something cleaner if someone finds something though.
|
# ? Sep 11, 2013 21:16 |
|
leper khan posted:This is what I do, and it's worked out really well for me. I'd still be interested in something cleaner if someone finds something though. The "cleanest" way I can think of is if you end up using OTP releases, which basically mean you take your entire node, and crystallize it by declaring what applications it should contain, along with a few more settings regarding the kind of runtime you want. They're the canonical way to ship an Erlang system, even though a lot of people and companies (those I worked at included) don't do it that way. If you're using reltool, you can specify custom filters about what part of applications to include or not. I have examples in the cookbook part of my book for reltool, and just today, Riak started using this method to avoid including Mnesia's include files in its project. I put "cleanest" in quotes, because there's a significant overhead to use Reltool in terms of what you need to know just to ship something. Rebar can actually wrap around it and newer releases of Erlang should contain a self-executable to do it. Relx is a new build tool for releases that is far easier to use, but is also far less powerful than reltool. So for your use case you might need reltool, but I'd recommend playing around with relx to get accustomed to releases and how they work.
|
# ? Sep 20, 2013 00:33 |
|
Not Erlang per se but is anyone else going to Ricon 2013 in SF this week?
|
# ? Oct 28, 2013 11:38 |
|
Cocoa Crispies posted:Not Erlang per se but is anyone else going to Ricon 2013 in SF this week? I won't be there (Toronto awaits me instead), but many of my coworkers will be there. I need to go to there next year :(
|
# ? Oct 28, 2013 23:45 |
|
So I posted this stuff in plenty of places already, but I had forgotten about this thread where this is where it makes more sense. Here's the blog post: https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole And here's the relevant stuff about how Erlang's memory works: quote:The amount returned by erlang:memory/0-1 is the amount of memory actively allocated, where Erlang terms are laid in memory; this amount does not represent the amount of memory that the OS has given to the virtual machine (and Linux doesn't actually reserve memory pages until they are used by the VM). To understand where memory goes, one must first understand the many allocators being used: Hopefully someone else than me finds this stuff super interesting (images are from my S3 account at work -- no leeching here)
|
# ? Nov 8, 2013 15:00 |
|
Anyone going to the Toronto Erlang Factory Lite?
|
# ? Nov 8, 2013 16:25 |
|
|
# ? May 10, 2024 00:28 |
|
Yeah, I'm speaking there: http://www.erlang-factory.com/conference/Toronto2013/speakers/FredHebert
|
# ? Nov 8, 2013 16:50 |