Why and How Discord Uses Patch to Test Elixir

Discord’s Realtime Infrastructure Team is responsible for delivering all the events that power Discord’s text chat.  These systems are largely built in Elixir, a dynamic, functional language for building scalable and maintainable applications.  These services are improved and deployed every day.  Tests allow us to develop and deploy these services with confidence.

So let’s talk about testing Elixir code.

Elixir comes with ExUnit and that’s what we use to test our code. ExUnit has all the normal features you would expect from a testing library, with one notable exception, mocking.  Mocking is something Elixir developers have to figure out for themselves, and that’s what this post is all about.

Step 1: Make Discord

To test some code we need to have some code to test.

Step 2: Test Discord

Great, we have a nice looking Discord that’s all ready to test!  Let’s write some tests for send_message/1.

Ok, so how should we write this first test?  We could craft a message in our test that fails one of the validation checks.  This would work, but it’s got a number of drawbacks.  Which validation inside of validate_message/1 should we craft the message to fail?  How much additional fixture data do we need to construct to cause this failure? Do we need a member list so the validate_author_is_member/1 check can fail?  What happens when someone comes along and adds a new check before the validate_author_is_member/1 check and now our send_message/1 test fails because it’s returning {:error, :new_check_failed} instead of {:error, :not_a_member}?

We can take a step back and realize that we don’t even want to test the validate_message/1 function or any of the validate_* functions inside of it.  We just want to have validate_message/1 return an error and assert that when that happens it gets returned by send_message/1.  Mocking would be a good fit for this problem.

It would be great if we could write some code that looked something like this.

Let’s grab a mocking library and try this out.

If you google Elixir Mock, you’ll probably end up at Mock.  Let’s install that and give it a try.

Not so hard, let’s try running it.

Uh oh, why is the function undefined?  Well it is a private function, so we can’t mock that.  Ok, let’s make it public, not ideal, but it is what it is.

Ok, so now Discord is not available.  Time to learn about partial mocking and the passthrough option.  Let’s change our code slightly to partially mock the module.

One more try!

What’s wrong now?!  Oh, the NOT SUPPORTED section of the documentation will enlighten you to the fact that this is an internal function call (also known as a local function call) and those can’t be mocked. 

Step 3: Ask for help while gesturing broadly at everything

At this point a lot of our engineers, especially those that aren’t primarily working in Elixir, throw their hands up and ask for help.  They normally end up at my virtual door, wondering why mocking is so much more difficult in Elixir than it is in their favorite language.  For a while I would just try to help them refactor their code into something that could be tested and get them on their way with what Mock had to offer.  This experience planted a seed, a small thought in the back of my mind, that this shouldn’t be so hard.

It also provided the guiding principle for the project the rest of this post is about.  

Patched functions should always return the mock value they are given.

Step 4: Sand down the sharp edges

In part to make my colleagues more successful at testing and in part to prevent having to have yet-another-conversation-about-Mock, I wrote a small, simple library called Patch.  This library started out with a relatively modest goal; make it easier to use the functionality that Mock provides. 

Engineers were consistently stumbling over things like partial mocks and were rarely using the expectations part of Mock, so I built a wrapper to set what I considered more sensible defaults.  Another complaint was that tests using Mock were very noisy and mocking was difficult to compose.  Mock thinks about mocking modules so if you wanted to mock a few functions in a module or multiple modules you end up with code like this.

Can you see the test? It’s that one line surrounded by punctuation. This is a lot of mocking noise in the test itself and the way it’s structured makes abstracting and encapsulating this concern rather difficult.

Here’s the equivalent mocking via Patch.

Since patch calls are just normal function calls you can make a patch_module_a function that encapsulates this behavior, it can take arguments, the entire expressive power of the language is at your disposal.

This was a much more pleasant experience for our engineers and the adoption of Patch grew and grew, and with that feature requests began to pour in.

Step 5: Devise a plan just crazy enough to work

Patch made the hammer easier to hold, but it was still just a hammer.  When we started our journey we just wanted to patch a local function call and our engineers still wanted to do that. Now that they had a direct line to the person writing their testing library, they had all kinds of other neat stuff they wanted to do too.

For uninteresting technical reasons Patch was originally built on Meck, the Erlang library that powers Mock.  Meck had the exact same limitations when it came to local function calls.  Well when the tools you have can’t do the job you can either give up or make better tools, and I was never one to give up.

Meck has a very nice and easy to read open source Erlang code base that allowed me to understand why the limitation exists and formulate a plan to replace it with a new strategy that didn’t have these limitations.  

Mock and Meck work by creating a copy of the module you are mocking and redirecting calls into a GenServer that can record the calls and respond with mock values for functions that are mocked.  Patch takes a similar approach but with some key differences.

Let’s continue to use our Discord module as an example and see what Patch does under the covers to allow local function calls to be mocked.

When you mock a module with Patch, three new modules will be dynamically generated, the Facade, the Delegate, and the Original, along with a GenServer for the module.

These modules are all generated off the BEAM file, and so no Elixir code is ever actually generated, but for the sake of simplicity we will look at the equivalent Elixir code instead of the BEAM abstract format.

Here’s what our Facade module would look like for Discord.

Not very exciting, just wraps the Delegate module, so let’s look at that next.

Again, not super exciting, but a little different.  The Delegate Module has all the functions, both public and private, and it just forwards the call to the GenServer for the module.  That GenServer actually holds the patch values and call history and so we can see how someone calling Discord.send_message/1 would now end up actually calling the GenServer.

Let’s peek into the delegate/3 function on Patch.Mock.Server to see what exactly it does (this is a simplified version of that function for clarity)

Pretty simple stuff, it calls the GenServer which either returns {:ok, reply} if the function has been mocked, or :error if it hasn’t.  If it returns :error then it just calls the function on the Original module.  

So let’s look at the Original Module.

This looks really familiar, looks almost exactly like the Discord module we started with.  There are two big differences though, all the functions are public now and every local function call has been transformed into a remote function call to the Delegate module we’ve already seen.

This is where Patch becomes capable of doing something no other Elixir mocking library can do, mock local function calls.  Now we can revisit our example and see if we can get our first test to work.

Let’s give this a run.

Nice.

How did that just work?  We can follow the call graph to see what exactly happened.

The call graph is a bit complicated, but the most important part is what happens in the Original module’s send_message/1.  Since the local function call to validate_message/1 was transformed into a remote function call to the Delegate’s validate_message/1, the Delegate and Server can intercept that call and return the patched value.  

The strategy works, it allows the test author to patch a function and that function always returns the patched value.   If it’s a remote function call the call path goes from Facade to Delegate to Server, which can return the mock value.  If it’s a local function call, the call path goes from Delegate to Server, which can return the mock value.

Step 6: Exploit new found powers for good

Once you start dynamically generating modules, it’s hard to stop.  The way that Patch works means that you can already patch private functions without any extra steps, that was a nice surprise.  Having to change the visibility of your functions just for testing always felt like a hack.

There is one other time that we change the visibility of a function for testing, testing private functions.  This isn’t always a good idea, but from time to time some important logic or a bug is found in a private function and to prevent future regressions engineers want to test that function.  This is often accompanied with a comment like “Function is only public for testing, do NOT call directly!!!”

Since Patch is generating the Facade, it could just expose some private functions as public for the purpose of testing.  This allows you to test important pieces of code directly while still enforcing that the non-test code in your project can’t call it directly.

This new ability to expose private functions as public was also implemented in the aptly named expose/2 function in Patch. 

Step 7: Use Patch

Once our engineers got a taste of being able to test things in a way that was ergonomic and powerful, they started testing more and more things.  Many of our tests are still using Mock but new tests are being written with Patch and as tests get updated engineers are voluntarily converting from Mock to Patch.  This is the best endorsement you can hope for when building a developer tool.

Patch started from a simple idea, when you patch a function it should return the patched value.  It has grown more capable, but at its heart is this simple idea.  Just like Elixir itself, Patch strives to provide simple guarantees that are easy to reason about. 

There are more cool things inside of Patch that I could write a whole series of blog posts about. Call assertions that work like ExUnit’s assert_receive that have the capability of binding arguments for additional inspection. Functions for working with processes that allow you to listen to all the messages being sent to a process or easily change the state of a running GenServer.  Mock values that go beyond functions and simple static return values, sequences, cycles, raising exceptions, throwing values.  The Quickstart provides a handy reference to all the features in Patch.

Patch is open source, MIT licensed, and available right now on Hex; it comes with a comprehensive test suite, plenty of documentation and a Guidebook.  Installation is as easy as adding the dependency to your mix.exs and from there you can enjoy writing unit tests in Elixir just that much more.

‍

If you want to work with people that are determined to make things like Unit Testing effective, easy, and fun, we are hiring.  If you are the kind of person that sees something isn’t working well and wants to make it work better, we are hiring.  If you just want to work somewhere with tests, we are hiring.