Encrypted notes shouldn't mean unsearchable notes

The privacy wall we almost built

When we started Scribr, the tension was obvious. Knowledge workers record meetings, calls, and voice memos on their phones. That audio becomes a transcript. That transcript becomes a searchable database of what you've said, what others have said to you, what you've decided. It's intimate. It's sensitive. Some of it is legally protected.

So when we built Vault Mode for Pro users, we went full encryption: AES-GCM at rest. The notes live on your phone encrypted. They sync encrypted to our server. No key is held by us. If someone asks for your data, we literally cannot read it.

The problem arrived quietly. Users started reporting that the best feature of Vault Mode (the encryption) was making the feature they loved most (the search) feel broken. You'd record a meeting six months ago about a contract negotiation. You'd remember a detail but not the exact meeting. You'd search, get nothing, and assume the note was gone. It wasn't. It was just locked behind your own encryption.

Why traditional search dies behind encryption

Full-text search needs to read the text. Encrypted text is gibberish to the search engine. You can decrypt locally and search locally, but that means your phone has to run the search every time. Scale that to a thousand notes across a thousand recordings, and the battery drain makes the app unusable.

We looked at the obvious workarounds. You could store searchable metadata alongside the encrypted blob. But then you're leaking information: someone could infer what's in your notes from the metadata alone. You could ask users to decrypt everything before searching, but that defeats the point of encryption; it's asking them to choose between privacy and usability, and most users will pick usability.

What we settled on was semantic embedding. When you record audio and we transcribe it, we create a dense numerical representation of the meaning of that text. That embedding is tiny, compared to the full transcript. And here's the key: the embedding gets encrypted too. But it's not used for decryption. It's used for search.

How the search actually works (without compromising the lock)

When you search for 'childhood home' inside Vault Mode, your phone takes that search phrase and converts it to an embedding as well. Your phone then compares that embedding to all the encrypted embeddings in your vault. Embeddings that are semantically similar light up. Your phone decrypts only the notes that match. You get results without the server ever seeing your plaintext or your search terms.

The encryption stays intact. The server has no window into your vault. And search feels instant because you're not decrypting and scanning thousands of notes anymore.

We didn't invent semantic search, obviously. But we had to make a specific choice: keep it on-device, or push the search to the cloud. We chose on-device. That meant your search queries, your vault's contents, and the embeddings stay on your phone. The tradeoff is that you search only what you've synced locally. For most people, that's everything. For someone with thousands of hours of recordings, it's a choice to make.

The question we're still asking ourselves

The therapist who wrote us that email is now a paying Team user. She uses Vault Mode for client notes and search works as she needs it to. But building this feature forced us to confront something real: most privacy features are friction features. People don't want them because they're good; they want them because they're necessary.

Vault Mode works because it solves two problems at once. It encrypts your notes so you own them fully. And it lets you find them, so you actually use them. The moment you make either half worse, users have to choose, and privacy usually loses.

We've kept the embedding search fully on-device, which is slower on older phones but truly private. We could move it to the cloud and make it faster; the embeddings alone tell you almost nothing. But then we'd be deciding which part of your notes is okay to share with our servers. We don't think that's our call to make.

What this means for how you capture and store your work

If you're a consultant, lawyer, therapist, or researcher, you already know that what gets said in your meetings is yours to protect. Scribr lets you capture that on your phone without uploading to the cloud unless you choose to (Free tier is fully on-device). Pro users get cloud transcription and AI features like summaries and action-item extraction, plus Vault Mode. Your notes stay encrypted; your searches stay private. Team users get contact intelligence and note sharing, which means you can share specific notes with colleagues without sharing the whole vault.

The point isn't that we're more private than everyone else. The point is that we've tried to make privacy and usability stop fighting each other. You shouldn't have to choose.