§ 00 / WINDOWS RECALL
Designing semantic search for everything you’ve seen.
- ROLE
- Senior UX/Product Designer
- PLATFORM
- Windows 11 (Copilot+ PCs)
- TIMELINE
- 2023 – 2025
- TEAM
- Cross-functional team of design, research, and ML engineering
- MY FOCUS
- Semantic search experience: ranking, relevance, and trust
- STATUS
- Shipped at Build 2024; reshaped post-launch around privacy
Where was that thing I saw last week?
The problem was simple and unsolved. You'd seen something on your computer: a presentation, a snippet of code, a reference in an email. But you couldn't find it. You'd try different search terms, retrace your steps, open files one by one. Minutes wasted. Information you knew existed but couldn't retrieve.
Recall aimed to capture everything appearing on screen and make it searchable through meaning rather than filenames. But solving that technically wasn't the real challenge. We had to build something users actually trusted.
My role focused specifically on the semantic search experience: how users search their memories, how results are ranked and displayed, and how relevance is communicated in a way that feels understandable and trustworthy.
§ 02 / 09
System
On demand intelligence
RECALL QUERY LIFECYCLE
04 · SEARCH & INDEX SERVICE
Stores enriched content in a semantic index and retrieves it by matching the meaning of a query against similarity scores, finding the right files even when the exact words don't match.
03 · MEANING ANALYSIS
Interprets raw extracted text by running meaning analysis to identify concepts and intent, then maps those concepts to semantic vectors for downstream understanding.
02 · OCR PROCESSING
Converts raw image input into structured text by straightening and denoising the image, identifying characters through optical recognition, and reconstructing the output into formatted, flowing text.
01 · SCREENSHOT CAPTURE
Continuously monitors and captures the screen at intervals, saving raw visual snapshots as the source material that enters the processing pipeline.
Search wants precision. Memory offers fragments.
File search is built on certainty. You give it a filename or keyword. It matches exactly. Done.
But that's not how memory works. You remember a chart, maybe the color was blue, maybe it was from an email or a browser. Nothing precise enough for traditional search to grab hold of.
Recall flipped this. Instead of requiring exact queries, it indexed everything the system could see and made it searchable through semantic understanding.
If you're indexing everything, how do you let people search without overwhelming them? The system needed to think like a person, not force people to think like the system.
Cards as moments, not documents
A Recall card needed to hold several pieces of information: a screenshot, app name, timestamp, extracted text, and relevance signals.
We built around the screenshot as the primary anchor. Not a cropped asset preview, but the actual desktop as it appeared. That context is what lodges in memory.
App name, timestamp, and extracted text stayed visible but secondary. The hierarchy pushed away from “found document” and toward “revisited moment.”
The result is something between a timeline and a search interface: visual enough to scan like memory works, structured enough to act predictably.
AI-powered search has a trust problem
Technically correct results can feel wrong. Search for “blue chart spreadsheet” and the system might return something from an unrelated app that simply had a blue element.
We didn't try to eliminate every false positive at the model layer. Instead, we made results understandable.
Every card explained how it matched. Text matches were labeled as text matches. Visual matches were labeled as visual matches. Separating and showing these signals let users judge relevance themselves.
Perfection wasn't the goal. Legibility was.
Merged results killed clarity
First version blended everything. Text matches and visual matches went into one ranked list. Clean, elegant, simple.
Testing proved it didn't work. Text matches dominated the ranking. Visual matches got buried. Users couldn't figure out why something appeared.
We split text and visual matches into separate sections. That single structural change made the system's logic transparent. Cognitive load dropped. Trust went up.
The AI was technically correct even in the first version. But opaque correctness erodes trust faster than transparent mistakes.
We made waiting feel like progress.
Fast enough to feel alive
Embedding indexing is computationally heavy, so we tuned retrigger cadence to refine results per keystroke without ever blocking the user.
PER KEYSTROKE
200MS WINDOW
NO BLOCKING WAIT
§ 08 / 09
Trust
Privacy was the whole product
Recall captures everything. That only works if people trust where the data sits, who can see it, and what control they actually have.
On-device processing wasn't optional. All capture, all indexing, all retrieval happened locally. Nothing left the machine.
When public scrutiny hit, we made the call to flip Recall from opt-out to opt-in, ship per-app exclusion, and give users a pause control. I owned the IA for the new privacy surface.
We made those boundaries tangible. Cards showed where results came from and when. Excluded content got explicit explanation instead of silent gaps. Trust wasn't a single setting. It lived in every interaction.
- DATA STORAGE
- Everything stays on-device. Local capture, local indexing, local retrieval. Nothing leaves the machine.
- USER CONTROL
- Opt-in by default. Users can exclude apps, pause indexing, and delete any memory at any time.
- TRANSPARENCY
- Every card shows where the result came from and when. System boundaries are visible, never hidden.
- DELETION
- Delete individual memories or wipe everything. No hidden caches. No silent retention.
We killed RAG to keep search fast
Early on we explored RAG synthesis across screenshots, but latency broke the core expectation that search should feel instantaneous. We abandoned synthesis for speed and legibility. Surface relevant moments, let people interpret them.
The system helps people rediscover what they saw. It doesn’t rewrite their history for them.
From rediscovery to reference pattern
Separating visual and text matches proved it wasn't just philosophy. In testing, users could explain why each result appeared and quickly reject things that didn't fit. Mysterious AI behavior became rational.
People recovered information they'd written off as lost. A code snippet they saw once in documentation. Files from months ago they'd forgotten existed.
The work rippled further. Privacy and trust patterns from Recall became reference points across Windows teams.
The search approach itself became foundational. Principles of relevance transparency and match-type separation showed up in Windows Search and File Explorer updates.
