Users

• Challenge Organizers

  • Challenge Initiator - An individual who wants to create/host a challenge. This user is the person who will create the challenge page and write the challenge materials. They will also engage with the challenge community to generate interest and participation. This user is most likely the driving energy behind a challenge and may be a funder, data provider, scientific lead, or project manager.

  • Challenge Reviewer - This user is responsible for reviewing the draft challenge pages prior to deploying them to production. This may be the PI on the challenge project. They are one of the Challenge Organizers.

  • Challenge Project Manager - This user is responsible for coordinating between all the other Challenge Organizers.

  • Challenge Scientific Lead - This user is one of the Challenge Organizers and is responsible for developing the scientific question and evaluation methods for the challenge.

  • Challenge Infrastructure Lead - This user is responsible for building and managing the infrastructure that runs the challenge. This may be remote servers that run the orchestrator tool, or they may be AWS environments running the orchestrator. This may involved coordinating between multiple institutions to run multiple remote servers.

  • Challenge Data Provider - This user is the point of contact at the organization/lab providing data for the challenge.

  • Challenge Funder - This user is the organization funding the challenge. They are one of the Challenge Organizers. (Should this be an organization or the user representing the organization?)

• Prospective Challenge Participant - This user is a potential challenge participant who needs to be able to explore current, past, and future challenges. They would need to know what the different questions were, what data was available, what the relevant domain expertise of the challenge was.

• Challenge Participant - This user is the researcher, student, or data scientist taking part in the challenge. This user needs to download data, create models, and submit either a prediction file, a docker image, or some other type of submission.

• Benchmarking Participant - Much like a challenge participant, this user is the researcher, student, or data scientist taking part in a benchmarking project.

• Challenge Archivist - This user is responsible for archiving the final challenge.

• Challenge Results Consumer - This is a challenge participant who is checking the results of the challenge. They will look at the leaderboard and check to see how their methods ranked in comparison to the other submissions.

• Challenge Governance Liaison - This user is a contact on the Sage governance team who is responsible for review the governance policies of the challenge. They will also be responsible for creating click wraps and gathering signatures for the different MOUs, DTAs, or DURs.

• Organization Manager - This user is the manager for the organization’s account on the challenge platform.

Responsibilities

• Data quality control

• Evaluation metric design

• Discussion board monitoring

• Infrastructure dry runs

Narratives

Challenge Creation

...

I have some new data. I want to start to a challenge. This is my question. Help me.

• Create challenge landing page draft

• Submit challenge landing page draft for review

• Review challenge landing page draft

• Create challenge data

• Identify challenge infrastructure needs

• Identify challenge questions

• Identify challenge evaluation metrics

Challenge Management

• Spin up the infrastructure

• Work with governance to establish the data terms of use

Infrastructure installation and management

• Initializing the orchestrator workflow engine in a Sage provisioned cloud (AWS, GCP, Azure) environment.

• Initializing the orchestrator workflow engine in a remote server provisioned by a collaborating institution.

• Initializing the orchestrator workflow engine on multiple, federated remote servers provisioned by multiple collaborating institutions.

Challenge Management

• Challenge Launch

Challenge Completion

• Post Challenge analysisChallenge archiving / publishing

Completing and publishing the challenge results after the completion of the challenge.

After a challenge is complete and the winners have been established the challenge initiator or challenge project manager click the “Archive Challenge“ button. A popup form prompts the user to fill out a few details including the synapseid synapse_id of the winning team(s), the synapseid synapse_id of the winning team(s) project wiki write up, and the submissionid submission_id of the winning submission or a link to the winning code github link. After clicking confirm, a prompt asks the user if they are sure they want to archive the challenge. Upon confirmation, the platform closes all queues, updates the status of the challenge to complete, creates a final results page where the leaderboard, the winning team, the winning team’s method write up, and the winning submission (link to docker container with instructions for downloading, link to github repo with winning model, etc.) are displayed along with relevant visualizations of the overall performance of the challenge results (aggregated AUROC curves, calibrations curves, etc.). A table of links to other teams' write ups is also published that includes the team names and their scores. The results web page is editable by all challenge organizers and can be edited at any time. After the flagship manuscript of the challenge is pubishedpublished, the doi to the paper is also added under a “Challenge publications“ section.

...

• Registering for a challenge

• Forming a team

• Forming a single person team

• Forming a team as a professor for a class