Skip to end of banner
Go to start of banner

Synapse Usage Metrics Dashboard

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Description

Many modern web applications have ways to aggregate and present metrics on how the application is being used in practice to those developing the application.  For example, sites like Amazon have rich and detailed information on what features get used, and translate into greater levels of purchasing on the site.  Synapse currently has a small user base and we can currently get a lot of information by talking directly to the users.  However, real metrics on actual user behavior provide another window into how the system is (or isn't) being used and can answer a variety of questions of interest to the business.

At the beginning of the year we defined two measurements of Synapse use that we wanted to track, and that fed into our yearly objectives.  These form the basis for initial dashboard design.

User Activity

Let's define the following:

  1. New User - Someone who's created a new Synapse account, has not logged in 3 times, and is less than 30 days from their account creation date.
  2. Non converted user - Someone who's created a new Synapse account more than 30 days ago, but has never become an Active user.
  3. Active User - someone who's logged into Synapse 3 times in previous 30 day window
  4. Lost User - someone who was previously Active, but is no longer.

We'd like to track the number of these over time, at the granularity of a day.  We'd like to know who they are, and especially highlight changes in a user's categorization.

We'd like to be able to break these out by the user's organization.  We can either use email domain as a proxy for organization when possible, or collect organization as part of user's profile if we can get it there.

In our company objectives, we define a use metric N based on # of active users.  In this calculation we exclude platform team and administrative users from the metric, and count users as double if they represent a unique organization.  So, if we have 5 Sage scientists, 3 people from U of X, and 1 person from U of Y as active users, N=12.  Our September "Achieved" and "Stretch" goals are N = 30 and 50 respectively.  We want to plot N over time on the dashboard.

Requirements

  1. The Dashboard will surface information about how users are actually behaving on Synapse so that Sage's Engineering and Management teams can drill into this data to make a variety of discussions related to the product roadmap and development of partnerships.  The dashboard must serve both technical and non-technical users (e.g. it must be useable by the CEO). 
  2. In the short term, we want to limit visibility of the dashboard to Sage employees, but make it very easy for all Sage employees to get at the dashboard.  Embedding the dashboard in either the Sage intranet or a Confluence wiki page would be a great way to accomplish this.
  3. In the long term we might want to also surface some metrics about Synapse on Synapse itself, or expose a public API for others to access and mine our metrics for a variety of purposes.  For example, a large data generator might want to be able to find out who is using their data.  Journals or funding agencies might want to assess the impact of work performed on Synapse.  However, we probably will always want a separate Sage-only dashboard that may be more specific or tailored to our needs than what we'd put on Synapse itself for public consumption.
  4. We are interested in observing long-term trends in user behavior over the course of months or years.  We will want to demonstrate uptake of the technology for purposes like raising grant money to continue Synapse development.  We are also interested in short term snapshots, e.g. what users have recently become active / inactive in last 30 days that might require someone making contact with the user and understanding what has happened. 
  5. It's not necessary that this be an operational dashboard for technical people to monitor and trouble shoot the performance of Synapse or it's components.  Cloud Watch type metrics on things like load on different infrastructure components are a different category of metric, and can be managed separately.
  6. We expect the specific metrics gathered to start off high level and general, and to continuously evolve and become more granular as we generate more questions to ask of Synapse about its users.  We want to make it easy for new developers to incrementally add to the dashboard.  For example, a new developer might develop a new feature and add new custom metrics to measure how the feature is actually used in production by live users.

Design Options

 

  • No labels