This is such a cool project. Haven't used it for any serious things but just the ability to have a high performance media streaming framework inside of Elixir is amazing. You literally could build Twitch just using BEAM and nothing else (a Postgres database is probably good to have though).
Mnesia is not a database by any modern definition of the term and it should generally be avoided. It is at least 4 if not 5 orders of magnitude away from "being able to run Twitch". That is, yes, I'm serious, if you tried to run a Twitch clone "but 10,000x smaller" I would still expect Mnesia to completely fall over.
Mnesia worked well enough for us at WhatsApp while I was there; although we didn't use it to store messages; long term message storage is on the end points (generally sqlite), messages in transit (offline) were stored in a file per user with the import/export written in C IIRC. We did add redundant in memory storage of messages in transit; but I don't remember the storage there; may have just been ets.
We mostly used mnesia as a replicated key-value store, but we got a lot of value from having the data and the business logic colocated. Other nodes would send logical operations to processes on the mnesia node and those processes could run each operation one at a time on the data. Any concurrent logical requests for a given piece of data were implicitly serialized by the process mailbox. But almost all of our data was easy to shard, no high volume operations needed to address multiple tables.
We heard a lot of things about mnesia scalability limits that just didn't match up with our experience; so I don't know what other people were doing, but you can get a glimpse of what we were doing in the Rick Reed talks at Erlang Factory. We certainly had some scalability challenges, but many (most?) are discussed in those talks; and my general recollection is that most of them were more like we were the only people running mnesia with tables of enormous size, so we had to make things work; but that's kind of how OTP is. The trickiest one to find, IMHO, was that IIRC mnesia_frag and ets (and our request sharding) all use(d?) the same hashing function, so adding more fragments would make distribution of keys per ets slot worse, ets wanted power of two slots, and would split based on average keys per slot, but we would have lots of keys on some slots and no keys on most slots. Changing the hash seed for ets was a 2 line code change that drastically improved performance on all of our sharded mnesia systems.
Another fun one is that if you use mnesia to store data for long periods, you have to be very careful with the binaries you store; it's easy to end up with refc binaries that have extra space for append operations; storing them in mnesia means that append space is allocated but unusable; you might also store a sub binary that's a small part of a refc binary, the underlying binary can't be disposed of until the sub binary is. For both of those cases, cleaning the binaries before storing them with binary:copy/1 can really reduce your memory use. There's probably some cases where you do want to store a sub binary though?
Mnesia doesn't (or didn't) include a good way to handle when two mnesia nodes sharing a schema disconnect and reconnect. We mostly solved that by ensuring our network was stable enough that that rarely happened. If your network is not stable, you will have a bad time with distributed Erlang in general, and Mnesia in specific.
If I were building Twitch but smaller, and on the BEAM, I would absolutely put account databases in Mnesia; but messages and media would probably live as files. I wouldn't tend to put those into a SQL database on a server either though.
That is not true. I dont use Mnesia because it is so different and thus harder for me than sql. But it is used. Klarna uses it. They are a pretty big company. Would I recommend it for most? No, but I also would not recommend cassandra and a bunch of other databases for most apps
mnesia is great, and you can get very far before you would make the jump to anything else. And it can be way faster then any other database. For obvious reasons.
Although it would be great if it spoke SQL, maybe one day it will: https://github.com/elixir-dbvisor/sql since we can already pass it and get the AST.
And when you do have to jump off, you'll be screwed. Because the jump off is usually from in-memory Mnesia with guaranteed sub-millisecond responses to a proper database with at least a magnitude higher latency. And you realize that your app is completely dependent on sub-millisecond responses :)
Low latency is addictive. And by the time when you think you might need to change you’re probably become such a skilled engineer that you realize that everybody that shit on mnesia does not know what they talk about. :)
There's also the legendary bleacherreport abandoning elixir and totally shooting themselves in the foot.
Several fintech companies moved off - brex, ramp. I think for brex they were told by VCs to hire XYZ CTO and the CTO couldn't elixir. Hilariously I ran into ramp people totally in the wild who complained that "they couldn't find elixir devs". I told them "you just randomly ran into one". I think their hiring processes were likely broken, but what's new in silly valley?
You can always find stories of people moving off of stacks. Sometimes they just legitimately evolved in a direction the stack wasn't the best solution for. Sometimes they should never have picked the stack in the first place. Sometimes a new leader came in who had preconceived notions that the company needed to conform with. You really have to look at the specifics of the story to know if it's relevant to you.
In my very opinionated opinion, it's actually reasonably uncommon for me to read a story of someone leaving a stack and not classifying it as one of the things I listed above. Of the cases I would consider "legitimate", it's usually a performance issue; some languages and runtimes are just intrinsically slower than others, or at least, intrinsically slower without an unrealistic amount of effort. (Elixir would be middling here. BEAM is kind of between the dynamic scripting languages and the compiled languages. The interpreter is simple enough that it can run much faster than the dynamic scripting languages but it would be completely unacceptable performance for any compiled language. You can run out of performance in BEAM, but it does take a system that needs performance and some growth to get there.) The rest are probably complexity explosion of some framework, and this is almost always a UI framework problem.
I don't understand the fixation on hiring $LANGUAGE devs. If you can't find any developers using your current stack, pay for a course or a book for them and train them on it. Training a competent developer to use a new programming language has to be easier, cheaper, and faster than rewriting your entire software stack.
We’ve been running elixir in production since 2017.
During the pandemic, our elixir app sent/received 45 million text messages, helped schedule 1.5 million vaccination appointments, and a few million COVID testing appointments.
It all scaled and performed flawlessly. Any bugs were our fault :-)
I recall doing a tutorial for Exilir, the Phoenix framework in particular, a few years back and I actually enjoyed using it. Anyone know any good up-to-date tutorials someone could use?
Fantastic project, and the team behind it is really good! The developers I have worked with are passionate about building things the right way (not just making it work/adding kludge).
I felt like I was seeing the future when I saw the visualization/rendering of PeerConnection stats on the server side. The video compositor is really neat also how they have it working with live modifications.
I wish I had more time/a chance to use it on a project myself.
I looked at using this for a client project a few months ago. We use Erlang and Elixir at work, and it's my go-to for anything serious.
Be aware that parts of their stack use a custom license for some components... but a large portion of it is OSS Apache 2.0, which is nice if you can stick to those parts!
What do you mean by "develop environment"? Are you referring to IDE support or features like mix, IEx, pry, releases, etc.?
If the latter, Elixir has one of the best development environments, in my opinion. Mix is fantastic, releases are easy, and Elixir's error messages in IEx are the clearest I've seen in my 30+ year career.
I use Emacs to write code, and beyond syntax coloring, I don't want anything else, so you may have a point if you're talking about IDE support.
This is really cool! Not just that, we actually really needed something like this in Elixir for a lot of projects and always had to end up going with some NodeJS implementation. Thank you <3
This is such a cool project. Haven't used it for any serious things but just the ability to have a high performance media streaming framework inside of Elixir is amazing. You literally could build Twitch just using BEAM and nothing else (a Postgres database is probably good to have though).
The BEAM even includes a database - Mnesia...
https://www.erlang.org/doc/apps/mnesia/mnesia.html
Mnesia is not a database by any modern definition of the term and it should generally be avoided. It is at least 4 if not 5 orders of magnitude away from "being able to run Twitch". That is, yes, I'm serious, if you tried to run a Twitch clone "but 10,000x smaller" I would still expect Mnesia to completely fall over.
Mnesia worked well enough for us at WhatsApp while I was there; although we didn't use it to store messages; long term message storage is on the end points (generally sqlite), messages in transit (offline) were stored in a file per user with the import/export written in C IIRC. We did add redundant in memory storage of messages in transit; but I don't remember the storage there; may have just been ets.
We mostly used mnesia as a replicated key-value store, but we got a lot of value from having the data and the business logic colocated. Other nodes would send logical operations to processes on the mnesia node and those processes could run each operation one at a time on the data. Any concurrent logical requests for a given piece of data were implicitly serialized by the process mailbox. But almost all of our data was easy to shard, no high volume operations needed to address multiple tables.
We heard a lot of things about mnesia scalability limits that just didn't match up with our experience; so I don't know what other people were doing, but you can get a glimpse of what we were doing in the Rick Reed talks at Erlang Factory. We certainly had some scalability challenges, but many (most?) are discussed in those talks; and my general recollection is that most of them were more like we were the only people running mnesia with tables of enormous size, so we had to make things work; but that's kind of how OTP is. The trickiest one to find, IMHO, was that IIRC mnesia_frag and ets (and our request sharding) all use(d?) the same hashing function, so adding more fragments would make distribution of keys per ets slot worse, ets wanted power of two slots, and would split based on average keys per slot, but we would have lots of keys on some slots and no keys on most slots. Changing the hash seed for ets was a 2 line code change that drastically improved performance on all of our sharded mnesia systems.
Another fun one is that if you use mnesia to store data for long periods, you have to be very careful with the binaries you store; it's easy to end up with refc binaries that have extra space for append operations; storing them in mnesia means that append space is allocated but unusable; you might also store a sub binary that's a small part of a refc binary, the underlying binary can't be disposed of until the sub binary is. For both of those cases, cleaning the binaries before storing them with binary:copy/1 can really reduce your memory use. There's probably some cases where you do want to store a sub binary though?
Mnesia doesn't (or didn't) include a good way to handle when two mnesia nodes sharing a schema disconnect and reconnect. We mostly solved that by ensuring our network was stable enough that that rarely happened. If your network is not stable, you will have a bad time with distributed Erlang in general, and Mnesia in specific.
If I were building Twitch but smaller, and on the BEAM, I would absolutely put account databases in Mnesia; but messages and media would probably live as files. I wouldn't tend to put those into a SQL database on a server either though.
That is not true. I dont use Mnesia because it is so different and thus harder for me than sql. But it is used. Klarna uses it. They are a pretty big company. Would I recommend it for most? No, but I also would not recommend cassandra and a bunch of other databases for most apps
mnesia is great, and you can get very far before you would make the jump to anything else. And it can be way faster then any other database. For obvious reasons.
Although it would be great if it spoke SQL, maybe one day it will: https://github.com/elixir-dbvisor/sql since we can already pass it and get the AST.
And when you do have to jump off, you'll be screwed. Because the jump off is usually from in-memory Mnesia with guaranteed sub-millisecond responses to a proper database with at least a magnitude higher latency. And you realize that your app is completely dependent on sub-millisecond responses :)
Low latency is addictive. And by the time when you think you might need to change you’re probably become such a skilled engineer that you realize that everybody that shit on mnesia does not know what they talk about. :)
Honestly I think mnesia is one of those "don't use it unless you know what you're doing" things. Just use postgres.
That would be the dream. Do you know of any major apps using elixir besides telecom?
my startup is using elixir in production for the last 5 years. we are a cloud based restaurant POS.
no regrets. the ecosystem has been pretty solid for everything we've wanted to do. Stability/performance has been very good.
also: if you're looking for a high profile startup using elixir, supabase is almost entirely elixir and discord uses it for some critical parts.
Are you willing to share the name of your startup?
https://blinqme.com/
At the BBC we use Elixir quite extensively. I'll talk about this at ElixirConfEU in a few days if you are interested.
There are several companies that are known to use elixir in production...
https://elixir-lang.org/cases.html
Tubi (relevant), Tvlabs, pagerduty, divvy
All not on the list.
There's also the legendary bleacherreport abandoning elixir and totally shooting themselves in the foot.
Several fintech companies moved off - brex, ramp. I think for brex they were told by VCs to hire XYZ CTO and the CTO couldn't elixir. Hilariously I ran into ramp people totally in the wild who complained that "they couldn't find elixir devs". I told them "you just randomly ran into one". I think their hiring processes were likely broken, but what's new in silly valley?
You can always find stories of people moving off of stacks. Sometimes they just legitimately evolved in a direction the stack wasn't the best solution for. Sometimes they should never have picked the stack in the first place. Sometimes a new leader came in who had preconceived notions that the company needed to conform with. You really have to look at the specifics of the story to know if it's relevant to you.
In my very opinionated opinion, it's actually reasonably uncommon for me to read a story of someone leaving a stack and not classifying it as one of the things I listed above. Of the cases I would consider "legitimate", it's usually a performance issue; some languages and runtimes are just intrinsically slower than others, or at least, intrinsically slower without an unrealistic amount of effort. (Elixir would be middling here. BEAM is kind of between the dynamic scripting languages and the compiled languages. The interpreter is simple enough that it can run much faster than the dynamic scripting languages but it would be completely unacceptable performance for any compiled language. You can run out of performance in BEAM, but it does take a system that needs performance and some growth to get there.) The rest are probably complexity explosion of some framework, and this is almost always a UI framework problem.
I don't understand the fixation on hiring $LANGUAGE devs. If you can't find any developers using your current stack, pay for a course or a book for them and train them on it. Training a competent developer to use a new programming language has to be easier, cheaper, and faster than rewriting your entire software stack.
If you're a cto hired in to a company you need to make your mark somehow.
Divvy still uses Elixir extensively. I use to work there and still have many contacts there.
Currently work here and we're definitely still building and supporting Elixir applications and enjoying it!
Thanks. Updated. Something about the best way to find an answer is to write an incorrect answer on the internet! ;-)
We’ve been running elixir in production since 2017.
During the pandemic, our elixir app sent/received 45 million text messages, helped schedule 1.5 million vaccination appointments, and a few million COVID testing appointments.
It all scaled and performed flawlessly. Any bugs were our fault :-)
Cars.com, Discord, Supabase are some of the top of my head. And of course Whatsapp is Erlang (which is same thing but with, imo, more ugly syntax)
OkNext.io is built using Elixir and Phoenix framework, if you're considering building a Web App and looking for examples.
PagerDuty, Pinterest, TheRealReal, Discord, Cars.com, Bleacher report
Discord, Facebook Messenger iirc was ejabberd and I think Google Talk was at one point? WhatsApp was ejabberd too iirc.
Discord was Elixir (and some Python) from the beginning. WhatsApp started with ejabberd though.
ESPN also uses it, iirc for their sportsbook system
Cars.com did a pretty extensive rewrite to Elixir
I recall doing a tutorial for Exilir, the Phoenix framework in particular, a few years back and I actually enjoyed using it. Anyone know any good up-to-date tutorials someone could use?
For Phoenix/LiveView, the pragmaticstudio's courses are great. I just completed https://pragmaticstudio.com/courses/phoenix
The official ones are still the best: https://hexdocs.pm/phoenix/overview.html and https://hexdocs.pm/elixir/introduction.html
Second the Pragmatic Studio courses. If you want something free, this is a good channel: https://www.youtube.com/watch?v=0rpt5sMb7cw
Fantastic project, and the team behind it is really good! The developers I have worked with are passionate about building things the right way (not just making it work/adding kludge).
I felt like I was seeing the future when I saw the visualization/rendering of PeerConnection stats on the server side. The video compositor is really neat also how they have it working with live modifications.
I wish I had more time/a chance to use it on a project myself.
I really wish Nvidia had gone in this direction instead of gstreamer :/
I looked at using this for a client project a few months ago. We use Erlang and Elixir at work, and it's my go-to for anything serious.
Be aware that parts of their stack use a custom license for some components... but a large portion of it is OSS Apache 2.0, which is nice if you can stick to those parts!
During my foray into Elixir I never found the develop environment as smooth as other languages.
What do you mean by "develop environment"? Are you referring to IDE support or features like mix, IEx, pry, releases, etc.?
If the latter, Elixir has one of the best development environments, in my opinion. Mix is fantastic, releases are easy, and Elixir's error messages in IEx are the clearest I've seen in my 30+ year career.
I use Emacs to write code, and beyond syntax coloring, I don't want anything else, so you may have a point if you're talking about IDE support.
as others have said, there’s a lot of work going into improving the dev experience.
There were 3 LSP implementations. they’re getting combined, so IDE support should improve.
There’s technically a step through debugger, but it’s extremely slow if you actually get it working.
There is a new type system too.
A bunch of work is currently going into improving that.
This is really cool! Not just that, we actually really needed something like this in Elixir for a lot of projects and always had to end up going with some NodeJS implementation. Thank you <3