This doc takes a step back to ask whether focusing on search efficiency and hit rate really makes the most sense. It looks at the bigger picture—like what users are actually trying to do and what makes search truly useful—and explores whether there might be better goals to aim for when improving the search experience.
✅ Search Efficiency (e.g., Time to First Relevant Result, Search Abandonment Rate)
Helps reduce friction in search workflows.
Aligns well with UX improvements.
Improves time-to-insight, which is crucial for researchers.
✅ Hit Rate (e.g., Query Success Rate, Precision@K)
A higher hit rate means better search relevance.
Helps ensure that searches are retrieving useful results.
Can be used as an internal measure of search model quality.
🔴 Doesn't Directly Measure Scientific Outcomes
A researcher finding a dataset doesn’t mean it's useful for their work.
Hit rate might be misleading if it counts "any" retrieved results instead of "meaningful" ones.
🔴 Might Overemphasize Low-Hanging UX Fixes
Improvements in efficiency might not translate to better scientific insights.
Efficiency gains do not necessarily make the search system smarter.
🔴 Fails to Account for Data Reuse & Discovery Impact
A search might lead to a dataset download, but that dataset's actual use is not tracked.
It does not capture whether search facilitates new biomedical discoveries.
To ensure that biomedical researchers find, understand, and reuse data effectively, better metrics that track meaningful success should be considered. These include:
📌 Search-to-Download Conversion Rate
Definition: % of searches that result in a dataset download.
Why? Ensures that search helps users find datasets they deem valuable enough to use.
📌 Search-to-Use Rate (via Citation or API Calls)
Definition: % of downloaded datasets that are later referenced in publications or accessed via API repeatedly.
Why? Directly links search success to scientific impact.
📌 Query Reformulation Rate
Definition: % of searches where users modify their query due to poor results.
Why? A high rate signals frustration and poor search quality.
📌 Dataset Exploration Rate
Definition: % of users who click through metadata, documentation, or previews before downloading.
Why? Indicates whether users understand what they are obtaining before committing.
📌 Semantic Match Rate
Definition: % of queries where users engage with conceptually similar (not just keyword-matched) datasets.
Why? Tests whether semantic search is effectively surfacing relevant datasets.
Prioritizing KRs like Search-to-Use, Dataset Engagement, and Reusability over efficiency and hit rate may better align with the goal of driving biomedical discoveries. However, if the focus is on improving the search system in the short term, tracking efficiency can still be useful—provided it serves as a foundation for better discovery and impact.