Walkthrough: File Storage & Sync

"Design Dropbox" is the blob-storage patterns question. If you internalize one split, the whole design falls out: metadata lives in a database, file bytes live in blob storage, and the two are connected by chunk fingerprints.

Step 1: Requirements

Functional:

1. Users can upload and download files — assume up to 50GB.
2. Files sync automatically across a user's devices.
3. Users can share files with other users.

Non-functional:

• Uploads are resumable: a dropped connection at 90% resumes, not restarts.
• Sync is bandwidth-efficient: editing one paragraph of a big file must not re-upload the whole file.
• Cross-device sync is eventually consistent — a few seconds of lag between devices is fine; say so.

Step 2: Core entities

User, FileMetadata (name, owner, size, version, list of chunk fingerprints, status), Chunk (a 5–10MB piece of the file, stored in S3, addressed by its fingerprint), Share (file, grantee, permission).

Step 3: API

POST /v1/files                { name, size, chunk_hashes[] }
                              -> { file_id, presigned_urls_for_missing_chunks[] }
GET  /v1/files/:id            -> { metadata, download_urls[] }
GET  /v1/changes?since=cursor -> changed file metadata   (sync)

Step 4: High-level design

Start by killing the wrong design out loud: uploading a 50GB file through your API servers fails three ways — it pins gigabytes of server memory per transfer, it doubles bandwidth (client → server → S3), and it guarantees timeouts. So the app server never touches bytes. It hands the client presigned URLs (temporary direct-upload links to S3), the client uploads chunks directly, and an S3 event notification tells the backend each chunk landed.

The upload flow:

1. Client splits the file into 5–10MB chunks and computes a SHA-256 fingerprint (a content hash) of each.
2. Client sends the fingerprint list to POST /files. The server records metadata with each chunk's status (pending) and returns presigned URLs — but only for chunks it doesn't already have (this is dedup; more below).
3. Client uploads chunks in parallel. As S3 events arrive, the server flips each chunk's status to uploaded, verifying S3's ETag (its content hash) against the fingerprint.
4. All chunks present → file status: complete.

Downloads mirror it: presigned URLs, ideally fronted by a CDN, with selective compression (compress text-like files with Zstandard; skip already-compressed video).

Deep dive 1: Resumable uploads

Because per-chunk state lives in the metadata DB, resume is free: the client reconnects, asks "which chunks do you have for file X?", diffs against its local list, and uploads only the gaps. No custom protocol — just fingerprints and status rows. This is the concrete flow interviewers want narrated step by step.

Deep dive 2: Deduplication

Fingerprints make chunks content-addressed: identical bytes → identical hash → same chunk object in S3. If a thousand employees upload the same 200MB deck, you store its chunks once and add a thousand metadata rows pointing at them. Dedup works within and across users and can cut storage dramatically. Mention the honest caveat: cross-user dedup means the server can tell that two users hold identical content — a privacy consideration, and it's incompatible with per-user client-side encryption. Raising that unprompted is a strong signal.

Deep dive 3: Delta sync

Requirement 2's efficiency clause: edit one paragraph in a 2GB file — what uploads? With fixed chunking, only the chunks whose bytes changed have new fingerprints, so the client uploads just those. One subtlety worth naming: an insertion near the start of a file shifts every byte after it, changing every fixed-boundary chunk. The senior-flourish fix is content-defined chunking — cut chunk boundaries with a rolling hash of the content itself, so boundaries move with the data and an insertion only disturbs nearby chunks. Name it in one sentence; don't derive it.

Deep dive 4: Conflicts

Two devices edit the same file offline, then both sync. Options:

• Last-writer-wins: simplest — latest timestamp overwrites. Silently destroys one user's work; unacceptable for a file product.
• Versioning / conflict copies: keep both — the second write becomes "report (conflicted copy from laptop)". This is what Dropbox actually does, because files (unlike database rows) can't be merged automatically.

Detect the conflict with a version check: each upload says "I'm replacing version 7"; if the server is already at 8, that's a conflict, so fork a copy. How do other devices find out about changes at all? WebSocket push for online devices plus periodic polling of /changes as the fallback — eventual consistency, as scoped.

Wrap-up

Recap: uploads and downloads are direct-to-S3 via presigned URLs with metadata in the DB; chunking plus SHA-256 fingerprints delivers resumability, dedup, and delta sync from one mechanism; conflicts fork versioned copies instead of losing work; sync rides push with a polling fallback. With more time: folder hierarchies, permissions inheritance for shares, and client-side encryption trade-offs.