Over the last four months, a couple of our engineers tucked themselves into a cave and coded some incredible backend voice chat infrastructure improvements. These are the kind of fixes that just make Discord work better. You’ll never notice them because… well, that’s the point. Your voice experience should just be smoother than ever.
We want to provide some insight into what we actually did so you can understand how much we care about making sure your voice chat experience is as painless as possible. This kind of work isn’t the sexiest stuff but it’s some of the most important work we do here.
With an expert three-pronged approach, two of our engineers, Jesse and Andrei, were able to greatly reduce voice issues and improve performance across the board. This was done by creating a curator, leveling up our infrastructure, and improving our client technology.
Watch the video for a basic breakdown from our engineers Jesse and Andrei. Then, read below for further details.
Creating the Curator
Server health is important. If you’re connected to a server that’s experiencing issues for whatever reason, you might encounter things like robo voice or even disconnects (which is, clearly, the last thing we want our users to experience). Maintaining connection in the middle of a gank can be the defining factor between success and failure.
The curator detects and removes bad servers. It does this by sending all servers some network data similar to the voice data that users send. The curator then inspects the return traffic. If it’s not healthy, it alerts the clients that the server is bad and the server is removed.
Discord engineers can also manually blacklist bad servers, providers, and regions when issues arise.
Another sweet feature of the curator is that it improves our ability to roll out updates without disturbing our users.
Most of the time we can update our voice servers while users are on them. Sometimes though, an update requires us to take the server offline. The curator can mark a server as “draining”, allowing currently connected users to finish up their voice session on the server. However, new sessions will be started on non-draining servers. Once all the sessions on the server have finished, we take the server offline and do the update.
Leveling Up the Infrastructure
One of the key things that Discord provides over self-hosted alternatives is something called “failover.”
When issues arise with one particular provider or data center, Discord will move users connected to these servers to healthier servers.
This happens seamlessly on our end — as a user, you’ll never notice this occurs (unless you’re staring mindlessly at your voice connection information for some reason).
We improved this process in a few ways.
First up, we added multiple providers and data centers to almost all of our regions so we can have more options.
Second up, we increased the number of servers in each region. Increased capacity means more, more, and more options.
Finally, we consolidated servers in some regions to provide cleaner and less confusing server options.
Overall, this means our backend has a lot more flexibility to maintain connection for our users.
Improving the Client Technology
The last bastion of these improvements is our client itself.
The client is now much more resilient to spotty internet and will resume connections without losing any voice traffic.
Network changes, Wi-Fi blips, IP changes, and short internet outages are now recognized quickly and our voice reconnection responds just as fast. In practice, you’ll notice far less connection issues when experiencing these things.
You’ll Never Know What Hit Ya
As we said in the beginning, the key with these improvements is that you will never notice them taking effect. We do all the heavy lifting behind the scenes and we do it all seamlessly so you can do the thing you’re trying to do — play games with a stable, high quality voice connection.
If you have a friend who may have had an issue connecting with Discord in the past, now would be a great time to have them give Discord another shot.
May your connections be strong and your ganks successful.
Wanna be a part of our next engineering endeavor? Check out our job listings here.