Security in system design

Most design rounds never mention security until the interviewer asks "and how is this protected?" with ten minutes left. A good answer is not a list of buzzwords. It is a short walk across the design: where untrusted data comes in, who checks what at each step, and what an attacker would try first. That walk takes two minutes, and it is the subject of this chapter.

The individual ideas each have an article: authentication vs authorization, sessions vs JWT, API keys vs OAuth vs mTLS and hashing vs encryption vs encoding. Here is where they sit in a design.

Draw the trust boundaries#

A trust boundary is any line where data or a request crosses from something you control less to something you control more. Draw them on your design and the security answer writes itself, because each boundary asks the same three questions: who is calling, what are they allowed to do, and what could they reach if a check were missing.

Who is calling: check identity once, at the edge#

The gateway checks every request's session or token, rejects anything invalid, and passes a verified user id to the services behind it. Services then trust that id because it came from the gateway, so two rules follow:

• Strip identity headers that come from outside. If services read the user from a header like X-User-Id, the gateway must delete any copy the caller sent and set its own. Otherwise anyone can claim to be anyone with one header.
• Short lived tokens. A stolen token is a stolen session until it expires. Access tokens that last minutes, refreshed from a longer lived one that can be revoked, keep that window small. Access tokens vs refresh tokens covers the pattern.

What may they do: check permission next to the data#

Knowing who someone is says nothing about what they may touch. That check has to happen where the object is loaded, in the service that owns it, because only that service knows whose order 1042 is.

That bug, often called IDOR (insecure direct object reference), is the most common serious flaw in real APIs, and it sits at number one on OWASP's API Security Top 10 for 2023 as broken object level authorization. It is easy to see why: every endpoint that takes an id needs the check, and one missing check leaks everything behind that endpoint. Two habits stop it:

• Load through the owner. Fetch "order 1042 belonging to user 7", not "order 1042" followed by a check that someone might forget. A query that cannot return other people's rows cannot leak them.
• Answer 404, not 403, for something the caller may not see, so counting through ids does not even reveal which ones exist. Random ids help too, but they hide the problem rather than fix it.

Services talking to services#

Inside the network is not the same as trusted. An attacker who gets one foothold, through a vulnerable library say, will try to call everything else from there. So services prove who they are to each other as well:

• Mutual TLS. Both sides of every connection present a certificate, so the payment service knows the caller really is the order service, and the traffic is encrypted even inside the data centre. A service mesh can do this for every service without code changes.
• Least privilege. The order service may call "charge this order" on payments. It may not call "refund anything". Each service gets only the permissions its job needs.

Secrets and data#

• Secrets live in a secret store, never in code, config files in the repo, or container images. Services fetch them at start up, and they are rotated on a schedule, so a leaked key stops working on its own. Better still are credentials that expire in minutes, issued to the service's identity, so there is no long lived key to leak at all.
• TLS on every connection, outside and in.
• Encryption at rest with envelope encryption. Each piece of data is encrypted with its own data key, and the data keys are encrypted by a master key that never leaves a key management service. Rotating the master key does not mean re-encrypting every file.
• Passwords are hashed with a slow function like bcrypt or Argon2, never encrypted and never a fast hash. Slow is the point: it makes guessing billions of passwords from a stolen table impractical.
• Keep less. Data you never stored cannot leak. And keep tokens, passwords and card numbers out of logs, which are copied to more places than any database.

Floods and abuse#

Not every attack wants your data. Some want you down, and some want to use your features at a scale you never planned for.

• Floods of raw traffic are absorbed by the CDN's network before they reach you. On 28 February 2018 GitHub took a 1.35 terabit per second flood, made by bouncing tiny requests off thousands of exposed memcached servers that replied with huge answers. It was unreachable on and off for about ten minutes, until its traffic was routed through a DDoS protection service that absorbed the flood.
• Floods of valid looking requests are harder. In 2023 attackers used a trick in HTTP/2, opening a stream and cancelling it at once, over and over. Google saw a peak above 398 million requests a second, and Cloudflare above 201 million. Defences sit at the edge and in the server software, which is one more reason to have an edge.
• Abuse of features is the most common. Login gets credential stuffing: passwords leaked elsewhere tried against your users. Sign up gets fake accounts, and ticket holds get bots. Rate limits per account and per IP, checks against known leaked passwords, and limits on what one account can hold at once are part of the design of those features, not an add on.

Break it#

Trade-offs#

Interview replay#

Checkpoint#