Phoenix is practically built with this idea in mind.
My startup is built out of a single phoenix monolith. Only external dependency is postgres.
Despite this, I've managed to completely avoid all the typical scaling issues of a monolith. Need a service or background worker? I juts make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process, pubsub to said service and error handling in the event of a crash.
I'm basicly getting all the best advantages of a monolith
* one codebase to navigate, our team is TINY
* all functionality is in libraries
And all the advantages of microservices without the associated costs. (orchestration, management etc)
* bottleneck use cases can live on their own nodes
* publish subscribe to dispatch background jobs
* services that each focus on doing one thing really well
We've accomplished this all through just libraries and a little bit of configuration. At no point so far have we had to..
* setup a message broker (phoenix pubsub has been fine for us. we can send messages between different processes on the server and to the clients themselves over channels)
* setup external clusters for background workers. (Oban was drop in and jsut works. I'm able to add a new worker without having to tell devops anything. the supervision tree allocates the memory for the process and takes care of fault tolerance for me. That leaves me to focus on solving the business problems)
* setup a second github repo. Its a monorepo and at out scale, its fine. We have one library for communicating to the database that every system just uses.
Eventually we'll probably have to start rethinking and building out separate services. But I'm happy that we're already able to get some of teh benefits of a microservice architecture while sticking to what makes monoliths great for mvps. It will be awhile before we need to think about scaling out web service. It just works. Leaves me more time to work on tuning out database to keep up!
In my experience, this is also a major benefit of running Akka on Scala or Java. I had the realization that it's basically a single-language Kubernetes, with some really nice abstractions built on top.
By using Kubernetes, you get a scalable infrastructure.
By using OTP/Akka, you get a scalable application.
While there is common problems to both domains, they are still 2 different domains.
For example, using only Kubernetes, you won't have the ability to react to a Pod restart within your application (unless your application is aware of Kubernetes).
Using only OTP/Akka, you still need a workflow for deployment and infrastructure management, and you still need to implement node discover for clustering.
NB: For Elixir, you have libcluster[1] that can use different strategies for node discovery, including using a Kubernetes Service to discover the nodes (as Pods).
EDIT: Using Kubernetes, libcluster, and Horde[2], you get the best of both worlds IMHO.
Thanks! I was just going to point out that Elixir/Phoenix + Libcluster + K8s is like a match made in heaven. I haven't tried Horde yet but I'm quite intrigued now.
Horde uses a CRDT[1] (Conflict-Free Replicated Data Type) to provide a distributed Supervisor/Registry, this allows you to run an OTP supervision tree only once in your cluster, with automatic takeover. Basically, you run your supervisor on each nodes, but only one will run it (thanks to the CRDT).
I find it very useful because "Distributed OTP Application" are a pain IMHO (they must be started during the boot of the BEAM).
wow thankyou! this is actually handles a usecase we're trying to solve in an upcoming sprint. (each tenant needs to maintain a persistent websocket connection to a third party api but though us) I was trying to use registry to do it but was was wondering how I'd scale it once it got big enough.
this sounds really neat and I'm going to go read about it!
I also wanted to call out that this ~sentence has just... a bunch of things that seem like jargon/names within the community? As an outsider, I have no idea what they mean:
> make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process
Elixir inherits a library called OTP from erlang which is a set of primitives for building massively concurrent systems.
A Genserver is sort of like a Base Class for a object that runs as an independant process that plugs into OTP. By inheriting/implementing the Genserver behavior, you create an independant process that can be mounted in an OTP supervision tree which runs at the top of your application and monitors everything below it. Out of the box, that means your process can be sent messages by other processes and can send messages itself. If a crash happens, the supervisor will kill it, resurrect it and redirect messages to the new process.
Creating a Genserver is as easy as adding an annotation and implementing a few callbacks.
Genservers are the base on which a lot of other systems build on. Oban, a job worker essentially builds on Genserver to use a postgres table as a job processor. Since its just a Genserver with some added behavior, adding a background worker is as simple as adding a file that inherits from Oban and specifying how many workers for it should be allocated in a config file. The result is that adding a background worker is about as much work for me as adding a controller. No additional work for devops either.
And yes, it is very sci fi. Honestly I'm shocked elixir isn't more widespread. there's very little hype behind it but the engineering is pretty solid. Every scaling bottleneck we've had so far has been in the database (only because we particularly make heavy use of stored procedures)
In the world of OTP (Open telecom platform), a process is the term for what essentially is a green-thread, not an OS process!
So it is:
a) much, much more lightweight (IIRC ~ 1Kb)
b) scheduled by the Erlang virtual machine (so called BEAM)'s scheduler. The BEAM's schedulers run on a per-thread-basis, inside the BEAM's process
c) independently garbage collected, no mutable memory sharing
This isn't a knock against Elixir or the Erlang ecosystem but I would definitely say that Elixir gets a decent amount of hype. Each time a new release comes out it invariably shoots to the front page of HN.
1. if I'm calling directly, I can just use a raw sql query
2. views can be backed with a read only ecto model
3. triggers can be set to run without your ecto code even being aware of it.
4. for custom errors, you can add overides for the error handling in ecto to transform things like deadlocks to 400 errors
Some of these ideas are written up in the late Joe Armstrong's dissertation about Erlang and OTP "Making reliable distributed systems in the presence of software errors". It's a few hundred pages but quite readable to a programming audience -- it isn't filled with acres of formal proofs or highly specialised jargon.
See chapter 4 "Programming Techniques" section 4.1 "Abstracting out concurrency" and chapter 6 "Building an Application" section 6.2 "Generic server principles"
They do sound very sci-fi. I would say it made me more or less inclined to dig a little deeper myself hahahaha.
Genservers are an abstraction encapsulating the typical request/response lifecycle of what we would consider a "server", but applied to a BEAM-specific process. Like "general server".
Oban allows for job processing, instrumented much like you would any other process in the BEAM. This is an external library while genserver is built in.
Really great talk! He talks about some problems about the BEAM Distribution, but didn't get into details about it. Do you have any idea about those problems?
I have done much the same in plain old Java, at a couple of jobs, going back ten years or so.
No OTP, GenServer, etc. Just a webserver and some framework for scheduled jobs, third party or write your own. Config enables individual routes and jobs at the top level. You can deploy one instance with everything enabled, or a hundred instances with the customer-facing web routes enabled in 60, API routes in 20, admin routes in 5, report generation jobs in 10, maintenance jobs in 5, or anywhere in between.
The only discipline you have to stick to is that you must pass information between components in a way which will work when the app is distributed. A shared database is the most obvious, and may be enough. We also used message queues, and at one point one of those distributed caches that were all the rage (Hazelcast / Terracotta / Infinispan / EHCache / etc - anyone remember JavaSpaces and Jini?).
We have a similar philosophy with our Python code base. Redis and Postgres are our only real dependencies. We use celery but itβs basically just Python. Maybe takes a little more work setting it all up, but once done provides similar benefits.
How do you keep track of background jobs? Is the queue persisted on disk somewhere? If it's not and a background worker crashes for some reason, then is the job lost?
Thatβs a very nice library. BEAM + Elixir + OTP supervision trees + Obama for background jobs + PostgreSQL for persistent storage seems a killer combination. And like you said you can scale horizontally naturally with the Erlang plateform without even using k8s. Really interesting.
This was possible due to the fact that BEAM does allow you to do things like the ones you describe and enjoy advantages of both worlds (monolithic and microservices). If someone does not use Elixir/Erlang however or if the the product consists of parts written in multiple languages (for whatever reasons) then it's simply not possible to have the advantages of microservices in a monolithic approach.
We moved to microservices, despite my love for a monolith with libraries:
- to enable different teams to deploy their smaller microservice more easily (without QA, database migrations etc affecting the whole app);
- to solve the human temptation of crossing abstraction boundaries.
Not black and white, and both approaches can solve these problems. Open to any feedback!
Yeah I moved to microservices for the exact same reasons. People are undisciplined, and making it harder (and much more obvious) for them to do the wrong thing is more important than having all your code in one place. Plus, if you need to upgrade some dependency or if you want to try a new language or library or idiom, you can do so without the risk, effort and sunk cost of upgrading the entire application.
> And all the advantages of microservices without the associated costs. (orchestration, management etc)
Really curious about this! Whatβs the deployment experience? What environment do you use for your production? How do you run/maintain the erlang VM and deploy your service?
We use eks with some customizations specific to elixir and use a autodiscovery service for new nodes to connect to the mesh.
One of the big differences we had to make was allocating one node per machine as the beam likes to have access to all resources. in practice this isn't a problem because its internal scheduler is way more efficient and performant than anything managing OS level processes.
That said, a more complex deployemnt story is defiantly one of the downsides. But the good news is that once setup, its pretty damn resilient.
Now of course, our deployment setup is more complex specifically to take advantage of beam such as distributed pubsub and shared memory between the cluster. If you don't need that, you could use dokku or heroku.
To add to this, we use Phoenix in a typical dockerized, stateless Loadbalancer- X Webserver - Postgres/Redis setup and it works great. Deployment is exactly the same as any other dockerized webapp. What OP is using is the "next level" that allows you to really leverage the BEAM but you don't have to.
I hadn't heard of this but it looks cool. Looks like something that will do the job when we do need it. so far we just store the parameters from certain heavily used mutations into a job table and run the actual insertion in a background worker. We're no where near needing this yet but it'll be good to have it on hand when we (hopefully) get to that point.
A zero-cost alternative that has worked well for me so far is to use a front-end load balancer to distribute requests to multiple Phoenix instances (in k8s), and then just let those requests' background tasks run on the node that starts them.
The whole app is approximately a websocket-based chat app (with some other stuff), and the beauty of OTP + libcluster is that the websocket processes can communicate with each other, whether or not they're running on the same OTP node.
not automatically but its pretty easy to configure in your supervision tree file. I don't know the details because whatever happens by default has taken care of our needs so far.
IT does automatically distribute processes across all the cores on a cpu though.
What I like about the Erlang platform is that it seems like it has the most sensible βmicroserviceβ story: deploy your language runtime to all the nodes of a cluster and then configure distribution in code. Lambdas, containers, etc. all push this stuff outside your code into deployment tooling that is, inevitably, less pleasant to manage than your codebase.
My startup is built out of a single phoenix monolith. Only external dependency is postgres.
Despite this, I've managed to completely avoid all the typical scaling issues of a monolith. Need a service or background worker? I juts make a Genserver or Oban worker and mount it in the supervision tree. The Beam takes care of maintaining the process, pubsub to said service and error handling in the event of a crash.
I'm basicly getting all the best advantages of a monolith
* one codebase to navigate, our team is TINY * all functionality is in libraries
And all the advantages of microservices without the associated costs. (orchestration, management etc)
* bottleneck use cases can live on their own nodes
* publish subscribe to dispatch background jobs
* services that each focus on doing one thing really well
We've accomplished this all through just libraries and a little bit of configuration. At no point so far have we had to..
* setup a message broker (phoenix pubsub has been fine for us. we can send messages between different processes on the server and to the clients themselves over channels)
* setup external clusters for background workers. (Oban was drop in and jsut works. I'm able to add a new worker without having to tell devops anything. the supervision tree allocates the memory for the process and takes care of fault tolerance for me. That leaves me to focus on solving the business problems)
* setup a second github repo. Its a monorepo and at out scale, its fine. We have one library for communicating to the database that every system just uses.
Eventually we'll probably have to start rethinking and building out separate services. But I'm happy that we're already able to get some of teh benefits of a microservice architecture while sticking to what makes monoliths great for mvps. It will be awhile before we need to think about scaling out web service. It just works. Leaves me more time to work on tuning out database to keep up!