Discord
Tags

Have you ever found a user at your company who actually likes using multi-factor authentication (MFA), either time-based one-time passwords (TOTP) or push-based MFA? Either method adds friction for users by necessitating a second device for logins while increasing the cost to attackers. However, both have problems. SMS MFA is widely regarded as insecure because of the proliferation of SIM jacking attacks that allows for intercepting SMS-delivered MFA codes— but TOTP MFA is also phishable.

MFA push notifications are also trivially easy to overcome. People’s attention can easily be exhausted. Look no further than Uber’s recent security incident in mid-September last year; that was the likely entrypoint Lapsus$ exploited in order to gain access to Uber’s internal systems.

Moreover, TOTP and push MFA are difficult to use frequently. They don’t vet that you’re logging in on a known device in a known location. They require use of a third-party authenticator app. Users lose their phones, and it’s difficult to have codes available on multiple devices. Some authenticator apps do support backing up TOTP secrets to a cloud account although backups are not typically enabled by default, which leaves users hanging out to dry if they lose access to their authenticator app.

Back when I joined Discord, a staff engineer on my team had been pushing for us to adopt “WebAuthn”, and soon. I’d heard of it of course, but they pointed us to a thread from Dev at Figma chronicling their adoption of WebAuthn. WebAuthn and other hardware-based MFA mechanisms significantly reduce the incidence of account takeovers (ATOs) at companies like Google and Coinbase:

Authentication Type chart displaying Percentage of ATOs associated using the noted authentication mechanism as of November 2022. SMS / Text Based Codes = 95.65%. Time Based One Time Passwords = 4.13%. Coinbase App Push Authentication = .18%. Physical Security Key = .04%.
source: coinbase.com

In 2022, after completing our migration from Google Workspace to Okta as our primary Identity Provider, our focus turned to answering this question - Our corporate user authentication strategy is great, but how can we continue to improve on it? We decided then was a good time to roll out WebAuthn everywhere.

This has been a transformative step for our security posture, and we want to walk you through how we did it, including both choices we made that worked out great and choices we made that we wish we hadn’t made. Read on, and we can hopefully help you with your migration!

A Primer on WebAuthn

The Web Authentication API (also known as WebAuthn) is a specification written by the W3C and FIDO, with the participation of Google, Mozilla, Microsoft, Yubico, and others. The API allows servers to register and authenticate users using public key cryptography instead of a password.

source: webauthn.guide

Web Authentication relies on three major properties: Strong (Authentication is ideally backed by a Hardware Security Module, which can safely store private keys and perform the cryptographic operations needed for WebAuthn), Scoped (A keypair is only useful for a specific origin, like browser cookies. A keypair registered at 'webauthn.guide' cannot be used at 'evil-webauthn.guide', mitigating the threat of phishing), and Attested (Authenticators can provide a certificate that helps servers verify that the public key did in fact come from an authenticator they trust, and not a fraudulent source).

WebAuthn leverages public key cryptography to generate a unique keypair per device-site pairing. The private key is securely stored on your device; the website gets the public key, and is used to validate the handshake initiated during sign-in.

The advantages of an approach like this are that WebAuthn is:

  • Strongly tied to a specific domain for a site. There’s no concern about lookalike domains for phishing
  • Hard to phish. There’s no TOTP code via SMS or authenticator app to phish
  • Easy to use. Using an authenticator, especially one built to your platform, is faster than using an authenticator app with expiring codes

https://webauthn.guide is a great site with a lot more info; anything I state here will just restate what they say, so I highly recommend diving in there.

Making the Case for WebAuthn

We had a number of challenges to overcome in pushing for Discord to adopt WebAuthn internally.

At the time Discord had been using two-factor authentication (MFA) for a long while, primarily via authenticator apps like Google Authenticator or Authy. We’d been using MFA with Google Workspace for years; we swapped people over to using MFA with Okta just a few months prior to the kickoff of this project in April 2022. So, we were pushing for behavior change after behavior change.

We wanted to change user behavior and incur additional hardware/support spend by rolling out WebAuthn. The idea of tying authentication to a user’s device worried people as well; would this lead to more lockouts? Would it lead to more support costs incurred to our hard-working IT team?

We wanted to show that WebAuthn was safer, easier, and more cost-effective when compared to the status quo of TOTP MFA.

WebAuthn: Safer

Thanks to a lot of great research from Google and other companies that they’ve published, it’s quite easy (and compelling!) to compare successful account takeover (ATO) or account compromise (ACO) attempts by different authentication factors:

Chart source: https://security.googleblog.com/2019/05/new-research-how-effective-is-basic.html
source: security.googleblog.com

Even more compelling, I got to include this very-definitive statement from the same research in my document:

“We have had no reported or confirmed account takeovers since implementing security keys at Google.”

It’s rare you get to be that definitive in security!

WebAuthn: Easier

I don’t know about you, but while I appreciate the security advantages of TOTP MFA, it does not spark joy to fish my phone out of my pocket, unlock it, and unlock my authenticator app, and feel the Damoclesian “T” in Time-based One Time Password counting down just to log into a service.

TOTP MFA is fairly secure as it is difficult to brute force; but by comparison, the typical experience for a user with WebAuthn set up using a platform authenticator here is:

  1. App (like Okta) challenges for WebAuthn factor
  2. User taps WebAuthn-compatible peripheral like Macbook TouchID sensors
  3. …that’s it!

Even a cellphone holster can’t compete on friction like that.

WebAuthn: More Cost-Effective; Easier to Support

Committing to spend money to buy hardware not only for everyone today but everyone forever is a big ask. We were looking at a cost of about $100/user that would scale linearly with headcount. We also had the specter looming over us of more IT-serviced lockout requests.

But the comparison isn’t to one of doing nothing. Inaction leaves us perpetually exposed, at a much higher likelihood, to a successful phishing attack targeting somebody who works for us. It’s a very common vector nowadays for breaches of companies.

According to IBM, the cost of a US-based company experiencing a data breach averages $9.44M in 2022. You could buy a heck of a lot of WebAuthn-compatible peripherals for that much money. At $30/pop, that’s over eight times the length of the Golden Gate Bridge in Yubikeys stacked end to end!

Moreover, relying upon additional great research from Google, we were able to show that moving to hardware-based MFA methods would actually decrease support burden over time by fewer lockouts:

Source: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45409.pdf
source: Lang, Juan, et al. Security Keys: Practical Cryptographic Second Factors for the Modern Web

Google’s research (linked above) summed up the improvements in support burden below:

For the deployment, we found the increased user productivity, and decreased support cost, were worth the increased hardware cost.

This meant that we would not only relieve some pressure with IT with this change, but also prevent more MFA lockouts that were already hurting our corporate users! Reasons like lost phones, new phones, lost credentials, etc. were now obviated— as long as we ensured users had multiple WebAuthn authenticators.

The Plan

So, people are sold. What’s the plan?

Step 1: get everyone and every app* (it’s never every app) into Okta

Step 2: get two+ WebAuthn authenticators to everybody who works for us. (some people, like super-admins and executives, get three authenticators for redundancy)

Step 3: a graduated rollout plan to each group periodically:

  • Security Engineering (always test on yourself!)
  • Infrastructure Engineering
  • IT
  • Executives
  • All of Engineering
  • All other employees
  • All different types of contractors

Step 4: use an Exception Group in Okta that exempts members from WebAuthn checks

  • Actually, we had a Rollout Group until about half our company was onboarded to WebAuhn. Then, we switched to an Exception Group

Step 5: shrink the Exception Group down to zero!

Easy, right?

Tools of the Trade

Here at Discord, we’ve got a healthy blend of MacOS, Windows, and Linux devices. Some users are on portable laptops; some are permanently docked. Some are even on desktops!

Our goal was to ensure everyone had at least two functional authenticators at any time.

We had a few tools in mind, with a few escape hatches.

Platform authenticators: these are WebAuthn authenticators embedded in the device we’re attempting to sign in with. The vast majority of our users were on Macbooks with TouchID or laptops with Windows Hello, meaning we could rely upon users usually having one of these two factors to register.

Roaming authenticators: these are WebAuthn peripherals not embedded in the device. For us, we used Yubikeys as our roaming authenticators.

Specifically, we chose to give most people the Yubikey Security Key C NFC as their roaming authenticator. We chose this for a few reasons:

  1. It supports NFC, not just USB-C, meaning you can use it to bootstrap a new iPhone or Android phone to service as a mobile platform authenticator
  2. It’s big and bulky, so people are less likely to leave them in their laptops, meaning users are less likely to combine their “roaming” and “platform” authenticator into one laptop-shaped bundle
  3. It doesn’t support OTP mode, so there’s no “Yubispam” to deal with

In some cases, like docked users or devices without platform authenticators, we provided keys like the USB 5c Nano to use in place of the Platform Authenticator instead while also supplying Security Key C NFCs as backups.

Our docs reflected our typical choices, but we always worked with users to find something appropriate and usable to make them successful.

How’d ‘The Plan’ Go?

So, here’s our planned timeline:

Project Kickoff May 2022; Security, IT, Infra, Execs June 2022; All FTEs July 2022; Everyone August 2022

And here’s how it actually went:

Project Kickoff May 2022; Security, IT, Infra, Execs June 2022, All FTEs July 2022, Everyone 2023

So, what happened? We have a few learnings to share:

  1. Fulfillment to a distributed global team is really hard. Work from home makes this even more complicated! My advice: pay for importers in any difficult-to-ship-to countries
  2. Always test on yourself first before shipping keys out to users. Because we did this for weeks, we were able to aggressively roll out to all FTEs on our original timeline!
  3. Listen to concerns, and find a path forward for unusual cases. We were changing people’s behavior. If they are docked 100% of the time and requested a WebAuthn-compatible keyboard we let them buy their favorite and expense it.
  4. Lean on enterprise shipping partners where you can. We used Yubico Enterprise Delivery, and their portal helped with US, UK, EU, etc. adoption tremendously. But: they don’t support every country we needed, so we had some complications to work through ourselves
  5. For enforcement: stick to your dates, but allow for exceptions. The “Exception Group” in the plan let us move the needle forward without blocking on the longest tail cases— invaluable!

What’s it like now?

Nowadays, newly-onboarded Discord corporate users receive a laptop and at least one Yubikey alongside that laptop. IT onboards users to Okta and instructs them to register at least two WebAuthn authenticators; typically, this is their Macbook’s TouchID/Windows Hello sensor and also their Security Key C NFC.

We also instruct corporate users to set up Okta Verify for use only as a fallback MFA in the event that all their authenticators fail at once. This way, we never have user accounts lacking at least one strong form of multi-factor authentication.

Lockouts at the worst require overnighting a Yubikey to the user; all the while they’re still protected by hardware MFA. We set up an “exception” group that we track with expiring membership to ensure users cannot stay without WebAuthn protecting their Okta sessions for quite a while.

The rollout took some effort, but the change has been very impactful, and users love the ease of use when compared with TOTP or other forms of MFA!

A Call to Action

Phishing attacks are prevalent, and the threat is urgent to address. We’ve stopped thousands of otherwise-successful phishing attacks thanks to our adoption of WebAuthn in Okta.

And, if you’re somebody who has a product that could support WebAuthn and/or passkeys better: please do! (we might even be building/planning this ourselves 😱)! Patchwork support is the biggest hurdle to adoption.

Together, we can make phishing (mostly) a thing of the past!

For More…

THE AUTHOR
MORE FROM