I still recall a CS jobs fair at my college ~8 years ago, where some company representative took a very bad liking to Erlang on my CV (I took a "Programming Languages" course where Erlang was one of ~6 languages we chewed through. s/o venkat!)
He kept asking "why use this over Java at my work X". He seemed to have the idea I wanted to join his company and rewrite their web administrative tools used by 5 people in Erlang. Not sure why he got so annoyed, and didn't ask me about anything other than a language he thought was silly!
In one CS job application in college I wrote in the list of skills "can write good prose". By this I meant both that I knew English, which was not obvious at that time and place, and that I can structure text logically, e.g. for documentation.
The interviewer kept mocking it and asking what did I mean by prose and whether I can also write in rhymes if needed, and ended up not making an offer. Twenty years later, good writing is still the most important skill in my career and the one that I rely on most often.
About 10 years ago a friend of mine asked me to apply for a job in his team (at this big famous company starting with A). I had exactly the right qualifications and we had worked before so it was basically a done deal. I send my application through the normal channels and their HR team rejected me before reaching my friend.
Among my qualifications I had jokingly added that I could fake a Pink Floyd riff or two on the guitar...
Seems to be a general knee-jerk reaction from typical CS-majors and other Java/C++ "bros" around Erlang. A more mature reflection could easily see lots of potential niches for Erlang, with the big caveat from a business perspective of the relative difficulty in finding devs. I had the pleasure to encounter Erlang professionally many many moons agos, in it's originally intended role within telecoms switching, and was nothing but impressed by it.
The argument here might be less about Erlang and more about Maslow's hammer - i.e., if all you've got is a hammer then all you can see are nails.
"CS majors"? Can you specify which universities you think have that mindset?
At least for us, CS is just algorithms and maths, we had only one single course for a specific language in my five years (C) but you where required to just pick up whatever the professor chose.
If you check the list of computer science awards they all focus on algos & maths so it feels weird if you could get a degree with a "language is what matters" mindset.
Seems like the standard CS degree course in the UK (at least last time I researched this stuff which was some years ago) heavily focuses around teaching the most popular tech stacks and employability and does not really focus as much at fostering a general interest in CS.
I mean yes, but at least a few years ago when I last looked at this, CS degrees in this country were heavily SE oriented and standalone SE courses were rare. I think the problem was lots of people were applying for CS degrees expecting SE courses and universities adjusted to cater for this. It meant that actual CS degrees seemed sparse for a while.
Yes, there are courses on Java and C/C++ (and also LATEX, MATLAB and Prolog!), but these are a few out of 60 courses in total, which include things like algorithms, information theory, distributed systems and optimizing compilers.
...although maybe Cambridge isn't a "standard" CS degree?
Firstly yes, Cambridge isn't "standard". It gets to assume its graduates will likely be employed because hey, Oxbridge degree, and in a STEM subject too, so it needn't care about "employability".
Both Oxford and Cambridge currently begin with an ML as first language (Cambridge chooses Ocaml), for the reasons I was taught an ML first too. These languages have sound fundamentals, their type system is good, they can handle recursion and other natural ways to express software. Also, almost none of the entrants know an ML [this will have started to change somewhat because of Rust] and so the introductory material holds everybody's attention and you don't need to spend as much time breaking bad habits.
Places which begin with just Java need to wrestle with the fact that Java has some very weird assumptions which you need to either mention but move past or else ignore and then they leave a weird hole in the knowledge of your students. (For now) All Java's user defined types live on the heap - that's very weird, do you call it out?. Some of your cohort know this language well, and so they can do your exercises without paying attention and they lose interest right at the start of your course which isn't good. But others are new to programming entirely as well as Java, for them every concept you introduce is a general idea - this isn't "How variables work in Java" for them it's "How variables work". Beware the resulting prejudices you have created.
This is exactly it. A language is a tool for engineering. I still have a soft sport for Erlang and it would make a lot of sense in many places I work on now, but it has to be seen in totality. Something that includes the entire ecosystem and thus we are not using it at the moment.
Maybe experience with language enthusiasts? When a language has fewer users, the users that it has are more likely to be enthusiasts.
Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp.
I guess you’ll become allergic to minority programming languages if that happens often enough.
> Reminds about me I was talking to a company at a booth. They were using Java and up comes this guy who started professing that that was all wrong and they should rewrite their whole stack in Lisp.
I frequently get told that the newest version of the JVM can do everything the Erlang VM can do and I should consider switching. Even here, in the comments of hacker news, I've witnessed JVM fanboys proclaim that Erlang, if "properly" implemented, for the JVM, would be faster and more performant then the Erlang VM.
From my experience a lot of Java guys have no idea about the wider world of languages out there. They can be total experts in Java and get very confused why you are not writing something in Java.
The vast majority of programmers don't read HN, reddit or really follow any industry news. They work in Java or C# which are entirely self sufficient with very comprehensive SDKs.
It's a pretty big leap though to take someone's academic interests as an indicator about their ability to do work. If that guy sees that I took two semesters of medieval history, is he going to be afraid that I'll turn the workplace into a Renaissance fair?
Having to hide your interests to look more bland might be the saddest thing I read today. I Just imagine that those recruiters have some sort of list of subversive activities.
1. Was an active member of the Red Army Faction OR
2. Read a book about Erlang or Lisp, is now too dangerous to be reintroduced into the general developer population
A link to a github account is far more valuable IMO. If you have nothing on github (or another code sharing service), then you probably don't have a passion for software.
> There are enough young enthusiasts who will rewrite something to weird tech without asking
This is a communication problem, and not something you can infer from a CV.
I can't see why having experience with any particular language would be a bad thing—quite the contrary. There are concepts to learn from each language, and the more languages under an engineer's belt (particularly "weird" languages), the more open they are to novel solutions. The tool analogy works well: you would want someone who's familiar with an axe and a chisel, and not only a hammer.
Someone thinking this is a bad thing shouldn't be making hiring decisions. It speaks more of the company, so OP probably dodged a bullet.
The software industry is for sure not the only industry where people confuse the quality of their tools for the quality of the job, and go looking for nails that match their preferred hammer instead of focusing on doing a good job of the task at hand.
However it might be one of the only professions where this tool-obsession is so valorized and where there's a whole subculture of it, and where it proliferates at such a high rate. There are people who've made half their career out of chasing the technology of the month or advising that a rewrite in language/framework/technique will solve all problems.
It's of course natural because we talk about the tech industry, and so people become focused on the technology rather than its application to human problems, or see all problems as a technology choice issue rather than a issue of implementing business logic with technology choices... So it depends on the discipline of the engineer how deep down that rabbit hole people go, and the quality of management about how well they can keep their staff focused on the problem domain first, and tools second.
All this to say: I've been that engineer, I've also suffered from those engineers, and because of that I'm now careful. I think Erlang is great. And my career now is Rust. But I hope I'm wise enough to recognize that how harmful rewrite in X and if only this was built in X can be to customers' projects and budgets.
Not to say the original-comment was that person, or that the person sneering about the Erlang was right, but I do understand where the wariness comes from.
All that said, I have more fear of the proliferation of novel JS frameworks than I do of stuff like Erlang or Lisp. And Java is another place where esp in the early days the proliferation of frameworks-as-solutions was a real problem.
My father worked at Ericsson for a long time back then, and while he wasn't involved in any of the AXE-N or Erlang stuff, he did hear water-cooler talk about the ongoing political battles. So this is like 3 layers removed from the people who were actually involved in it all, but anyway:
There was a huge push to move on from the in-house PLEX to industry standard software engineering. Object orientation, component architecture, UML, design patterns, etc.; think of a software engineering kool-aid from that time period, they were drinking it. And C++ was the industrial strength vehicle to deliver all of this.
AXE-N wasn't just a project that flopped, it was the crown jewel of this giant effort to switch to the new "industry standard" way to write telephony software. So the failure was a giant cultural shock.
But even after the AXE-N failure, the phalanx behind it weren't done for, and apparently it was they who managed to push through a ban on Erlang, that embarrassing thorn in their side.
As for what happened later, I don't know. Anybody who knows what language Ericsson is currently using to program their exchanges?
I think all modern Ericsson mobile network base stations use Erlang (at least for the configuration part) so most likely just using your phone you come in contact with an Erlang system everyday (depending on what gear your ISP use, but Ericsson is one of the largest providers of telecom equipment).
There were and are good reasons not to use Erlang. It's much (like an order of magnitude) harder to find developers who know it and almost impossible to find developers who can use it well (e.g know what a gen_server is). Much Erlang code I saw at Ericsson were technically written in Erlang but very much resembled imperative style code you'd find in a Java project. Erlang is also not very good at interacting with the surrounding system which is pretty important for a SoC.
I left Ericsson many years ago, but I bet that they are still using ClearCase for version control, FrameMaker for documentation, and that there are still five managers for every developer. Pretty sure they are still drinking the kool-aid, trying to put AI in their exchanges glued together with TypeScript.
Ericsson is a large company with many different products but I didn’t see any usage of ClearCase when I was there. It was mostly git, C++, etc. Some product was written in Erlang but most weren’t (what I know of at least)
TBH, I'm not sure CORBA is as bad as its reputation tells. It's a hard problem space overall, and seems nowadays we're reinventing (parts of) CORBA with things like gRPC etc.
It was a hard problem that could have had a much simpler solution than CORBA. There is no need for a Grand Unified Thingy that is so complicated nobody actually understands it.
Even back then you probably could use something as simple as xmlrpc and get the job done at 1/100 of cost of an CORBA solution.
I love erlang so much. It's just such a bizarre combination of things that should't mix as well as they do. Pattern match as assignment still feels futuristic; I know they didn't invent that but it must have taken serious vision & confidence to commit to it. Plus agents as the core abstraction, immutability semantics.
It all feels very CS-theory elegant and then you get to OTP, the most grimy roughneck Système D ass standard library I've ever seen. But that has no actual problems anywhere in my experience other than that you'll go years not knowing something is in there bc you didn't guess what five letter abbreviation it would be under.
Plus the syntax is like what you would get if you described a programming language to someone who had never seen one. Code from another planet. Flawless.
For confused fellows: In the context of Erlang, OTP stands for "Open Telecom Platform."
The key components of OTP include:
OTP Behaviors: Behaviors are generic programming frameworks that define the structure and behavior of a particular type of process. OTP provides several predefined behaviors, such as gen_server, gen_fsm, and gen_event, which encapsulate common patterns of communication and state handling.
Supervision Trees: OTP introduces a hierarchical structure called a supervision tree, which allows for the supervision and management of processes. Supervision trees help handle process failures and enable the system to automatically restart failed processes or take appropriate actions in response to failures.
Application Framework: OTP provides an application framework that facilitates the organization and configuration of Erlang applications. The framework allows developers to define dependencies between applications, manage application startup and shutdown, and handle system-wide configuration.
OTP Design Principles: OTP promotes the use of certain design principles, such as "Let It Crash" and "Fail Fast," which encourage building fault-tolerant systems that can recover from failures quickly and gracefully.
you'd think that an article about how facebook bought the erlang-implemented whatsapp because facebook messenger was less popular would bother to mention that facebook messenger used bob ippolito's mochi media's mochiweb, an erlang web server, to handle the comet connections that allowed facebook messenger to deliver asynchronous notifications to web browsers
that is, facebook messenger was also to a significant extent written in erlang, though i don't have any insight into how much of the internal facebook messenger system was erlang-based (i imagine probably not that much)
also why does the article say
> In 2022, it received a notable update that speeded up the execution of Erlang programs significantly through the introduction of a ‘just in time’ compiler.
and not mention that the hipe jit compiler was added in 02000, 22 years earlier, with x86 support added the next year? there have literally been jit compilers for erlang since last millennium. https://www.researchgate.net/publication/225129672_The_HiPEx...
There was a Commercial Uses of Functional Programming workshop talk back in 2009 where the engineers behind Facebook Chat talked about their use of Erlang.
>Then suddenly, in 1998, Erlang was banned within the Ericsson. The reason? To avoid the costs of development of a language that was unique to Ericsson, and instead to access the shared effort being put into the development of other languages.
Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).
> Sounds like some manager had drunk the FOSS/bazaar kool-aid at that point, but wanted to keep the costs down by just taking from FOSS efforts in other languages (as opposing to continue to use more resources to maintain Erlang at the previous degree, and giving to FOSS).
AFAIU, nothing to do with FOSS, which wasn't really on the radar for a somewhat conservative telecom company at the time. It was about custom in-house stuff (PLEX, Erlang, etc.) vs. "industry standard" (C++, OO, UML, etc.).
Motorola Labs had a similar-ish internal programming language around this time. It also died because devs didn’t want to use an internally-developed language. The important things were hiring, training, support, external tools, etc. Being a “good” language was at the bottom of the list. They went with C++. Even though I worked on that PL I knew they were right not to bet a giant business on a bunch of researchers.
I didn't know him personally, but I loved how he came across to me as both a cantankerous 'old' man AND an infectiously 'young' enthusiast in his talks.
I had the pleasure of escorting him around Chicago one day, and his enthusiasm was in full force. Such an amazing fellow, hearing the news of his death was a cruel blow.
>Like WhatsApp, Facebook Messenger had used Erlang and ejabberd but moved away from it after finding it difficult to recruit engineers.
Since WhatsApp handled much more users with just 35 engineers, I would have thought it wouldn't be so much of a deal to recruit some Erlang engineers. They didn't need to recruit thousands of engineers.
Facebook Messenger moved to a C++ server, built like the rest of Facebook C++ servers.
The story I heard was they were looking for people who already knew Erlang and were having trouble finding them, so they rewrote the thing in a language that they could hire for.
From my recollection, when I was at WhatsApp, we only hired two server engineers who had used Erlang before. I was considered knowledgable about Erlang, because I vaguely remembered seeing the Slashdot post when Ericsson open sourced it. Pretty much we hired smart people and gave them an Erlang book. As a result, I don't think we did OTP applications properly, and we might have missed out on a couple other things, and our code might not be very idiomatic always, but I think we did ok.
But, beyond hiring, Erlang is a weird fit inside Facebook's datacenter infrastructure. The Messenger server team was (and I believe still is) very small, and rewriting into something with less impedence mismatch for a small team to deal with probably made sense. For WhatsApp, we spent quite a lot of effort making our software fit at Facebook, and we had to give up FreeBSD and from what I've heard, hotloading, too.
I think the assumption Sun's Java could become something that eclipsed most languages including Erlang is over, as the prototype distributed JVMs just became another navel-gazing exercise.
The Erlang/Elixir OTP has design constraints like any other solution, but has survived 4+ business cycles. One can only hope Julia ends up with something similar, but more user friendly.
Distributed systems is hard, and training obstinate people is like trying to teach a Raccoon TCL. If a design really needs Erlang/Elixir, than there are few alternatives that won't degenerate into a polyglot dumpster-fire.
In practice, hot code reloading and mnesia are pretty cool concepts but very far from trivial. The vm and OTP provide mechanisms to work with such advanced concepts, but it still requires an intense amount of discipline and process to execute effectively.
Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture).
For given hard requirements, e.g. upgrading the application without dropping TCP connections - this can be a killer feature that removes considerable development effort, but usually it's a very cool thing that you don't need.
That being said, hot code reloading on the individual module level I have found to be a useful and dirty trick to change the behaviour of running applications (add log statements / telemetry where they weren't before to diagnose production issues). This still has all the caveats and danger of the above, but is more limited in scope.
> Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture).
This was an interesting discussion I had with a few of our B2B customers at a customer event.
Even with redundancy and everything, something like a major postgres upgrade is a tricky thing. You either have to do some amount of work of setting up several clusters, setup logical replication, decide a cutover point (most of them things pg_easy_replicate (if I remember the name right) does), while juggling the backup solution on the side and such. It's overall understandable, but a decent number of steps to do right, and if you don't you might even lose transactions to the upgrade. And losing transactions is way worse than being down.
The alternative though is somewhat brute: After testing everything, shut the app down, shut the cluster down, upgrade binaries, run pg_upgrade. Restart cluster, check cluster, start application. This requires downtime, but as a procedure, it is very straightforward, very testable and overall predictable. Running a readonly version of the application sadly wouldn't be possible because of choices in the application.
Interestingly, many of our larger customers obviously preferred if vendors don't go down, naturally. But for a complex migration or update, most of them have been burnt by horror stories of upgrades going entirely sideways. If an outage is of predictable time, well-planned and communicated in advance, they can prepare for it and deal with it. One just shrugged and said "So what, then we just tell our customers we don't provide that service for half a day. No big deal with a month in advance or so"
This gave me a very valuable perspective of how much energy to put into avoiding downtimes for rare procedures and interactions. Talk with your customers, and sometimes the correct answer is indeed "don't".
Exactly, it's similar to Linux live kernel patching. It's certainly desirable, but with the support of modern orchestration platform, most of the time it's simpler to migrate services off the nodes, update, reboot then redeploy.
Having worked with hot loading for a long time, and now working without, even if you bring up new nodes, migrate services, destroy old nodes, it just takes so much longer, and you have to spend so much more time dealing with the multiple versions at once problem.
Especially if you have state in your service. Shutting down servers with active clients is disruptive, and moving clients immediately is difficult. Simple HTTP request/response is easier, but a lot depends on your load balancing.
We run a distributed cluster of 32 Elixir nodes across 4 data centers. Typically, we do rolling deployments -- but when we need a fast fix for something, pushing patches against the live system is real life saver!
These build on the facilities you're quoting: Phoenix uses hot reload to enable smooth local development, Phoenix pub/sub builds on OTP distribution, Docker builds use OTP releases, ETS tables are everywhere...
Mnesia is the only major piece that I agree is left out, for good reasons (it's fine software but really not built for cloud deployments).
Aside from the main article, I find it very convincing marketing strategy to announce a continuation of the subject in next chapter that will be paid. I could already see the quality of the authors and I feel like I want to pay now to read more. I'm all in for this kind of marketing.
He kept asking "why use this over Java at my work X". He seemed to have the idea I wanted to join his company and rewrite their web administrative tools used by 5 people in Erlang. Not sure why he got so annoyed, and didn't ask me about anything other than a language he thought was silly!