/
Rethinking Search Success Metrics
Document toolboxDocument toolbox

Rethinking Search Success Metrics

Owned by Mieko Hashimoto
Last updated: Apr 21, 2025

 

This doc takes a step back to ask whether focusing on search efficiency and hit rate really makes the most sense. It looks at the bigger picture—like what users are actually trying to do and what makes search truly useful—and explores whether there might be better goals to aim for when improving the search experience.

 

1. Evaluating Search Efficiency & Hit Rate as KRs

Pros of Targeting Search Efficiency & Hit Rate

✅ Search Efficiency (e.g., Time to First Relevant Result, Search Abandonment Rate)

  • Helps reduce friction in search workflows.

  • Aligns well with UX improvements.

  • Improves time-to-insight, which is crucial for researchers.

✅ Hit Rate (e.g., Query Success Rate, Precision@K)

  • A higher hit rate means better search relevance.

  • Helps ensure that searches are retrieving useful results.

  • Can be used as an internal measure of search model quality.

Cons & Potential Gaps

🔴 Doesn't Directly Measure Scientific Outcomes

  • A researcher finding a dataset doesn’t mean it's useful for their work.

  • Hit rate might be misleading if it counts "any" retrieved results instead of "meaningful" ones.

🔴 Might Overemphasize Low-Hanging UX Fixes

  • Improvements in efficiency might not translate to better scientific insights.

  • Efficiency gains do not necessarily make the search system smarter.

🔴 Fails to Account for Data Reuse & Discovery Impact

  • A search might lead to a dataset download, but that dataset's actual use is not tracked.

  • It does not capture whether search facilitates new biomedical discoveries.

2. Alternative or Complementary KRs to Consider

To ensure that biomedical researchers find, understand, and reuse data effectively, better metrics that track meaningful success should be considered. These include:

A. Researcher Success & Engagement Metrics

📌 Search-to-Download Conversion Rate

  • Definition: % of searches that result in a dataset download.

  • Why? Ensures that search helps users find datasets they deem valuable enough to use.

📌 Search-to-Use Rate (via Citation or API Calls)

  • Definition: % of downloaded datasets that are later referenced in publications or accessed via API repeatedly.

  • Why? Directly links search success to scientific impact.

📌 Query Reformulation Rate

  • Definition: % of searches where users modify their query due to poor results.

  • Why? A high rate signals frustration and poor search quality.

B. Dataset Understanding & Accessibility Metrics

📌 Dataset Exploration Rate

  • Definition: % of users who click through metadata, documentation, or previews before downloading.

  • Why? Indicates whether users understand what they are obtaining before committing.

📌 Semantic Match Rate

  • Definition: % of queries where users engage with conceptually similar (not just keyword-matched) datasets.

  • Why? Tests whether semantic search is effectively surfacing relevant datasets.

3. Recommendation

Prioritizing KRs like Search-to-Use, Dataset Engagement, and Reusability over efficiency and hit rate may better align with the goal of driving biomedical discoveries. However, if the focus is on improving the search system in the short term, tracking efficiency can still be useful—provided it serves as a foundation for better discovery and impact.