An Evaluation queue allows for people to submit Synapse Files, Docker images, etcSynapse Files or Docker images. for evaluation. They are designed to support open-access data analysis and modeling challenges in Synapse. This framework provides tools for administrators to collect and analyze data models from created by Synapse users created for a specific goal or purpose.
...
To create a queue, you must first create a Synapse Project or Synapse Project or have edit permissions on an existing Project. To create a Synapse Project, follow the instructions on the Project and Data Management page.
Once you’ve created your project, navigate to it and add /challenge to the url (e.g. www.synapse.org/#!Synapse:syn12345/challenge). Click Challenge Tools in the right corner and select Add Evaluation Queue.
...
An Evaluation queue can take several parameters that you can use to customize your preferences.
Name – : Unique name of the evaluation
Description – : A short description of the evaluation
Submission Instructions – : Message to display to users detailing acceptable formatting for submissions.
Submission Receipt Message – : Message to display to users upon submission
...
Optionally, you can restrict how things are submitted by using submissions by adding a quota to each round of your challenge.
...
An Evaluation queue can only have one quota. You may specify the length of time the queue is open, the start date, round duration, and number of rounds. These are required parameters. It is optional to set a submission limit.
First Duration (Round Start Date/Time - The date/time at which the first round begins.
Number of Rounds - The number of rounds, or null if there is no limit to set.
and Round end) - Select a date and time for the start and end of the round.
Submission Limit - The maximum number of submissions per team/participant per round. Please keep in mind that the system will prevent additional submissions by a user/team once they have hit this number of submissions.Round Duration - The duration of each round. If you set a round duration you will want to set the number of rounds or it will assume an infinite number of rounds
Advanced Limits - You may set additional quotas for daily, weekly, or monthly submissions per team/participant. These limits can be combined by clicking the + sign next to the Maximum Submissions field.
Share an Evaluation Queue
Each Evaluation has sharing settings, which limit who can interact with the Evaluation.
...
| Info |
|---|
Important: When someone submits to an Evaluation Queue, a copy of the submission is made, so a person with Administrator or Can score Score access will be able to download the submission even if the submitter deletes the entity. |
Close an Evaluation Queue
While there isn’t technically a way of “closing” an evaluation queue, there are multiple ways to discontinue submissions for users.
...
The submission function takes two optional parameters: name and team. Name can be provided to customize the submission. The submission name is often used by participants to identify their submissions. If a name is not provided, the name of the entity being submitted will be used. As an example, if you submit a File named testfile.txt, and the name of the submission isn’t specified, it will default to testfile.txt. Team names can be provided to recognize a group of contributors.
Python
| Code Block | ||
|---|---|---|
| ||
import synapseclient
syn = synapseclient.login()
evaluation_id = "9610091"
my_submission_entity = "syn1234567"
submission = syn.submit(
evaluation = evaluation_id,
entity = my_submission_entity,
name = "My Submission", # An arbitrary name for your submission
team = "My Team Name") # Optional, can also pass a Team object or id
|
R
| Code Block | ||
|---|---|---|
| ||
library(synapser)
synLogin()
evaluation_id <- "9610091"
my_submission_entity <- "syn1234567"
submission <- synSubmit(
evaluation = evaluation_id,
entity = my_submission_entity,
name = "My Submission", # An arbitrary name for your submission
team = "My Team Name") # Optional, can also pass a Team object or id
|
...
Every submission you make to an Evaluation queue has a unique ID. This ID should not be confused with Synapse IDs which start with synthe prefix “syn” (for example, syn12345678). All submissions have a Submission and SubmissionStatus object.
...
Navigate to a file File in Synapse and click on File Tools in the upper right-hand corner. Select Submit To Challenge to pick the challenge for your submission. Follow the provided steps to complete your submission.
...
View Submissions of an Evaluation Queue
Submissions can be viewed and shared with users through Submission Views creating dynamic leaderboards. Submission annotations can be added to a SubmissionStatus object and are automatically indexed in the view.
...
You can input your own query statement such as SELECT * FROM syn22155139 ORDER BY score DESC. Remember, 22155139 syn22155139 should be replaced with the id synID of the submission view:
...
Submit to an Evaluation Queue from a Wiki Page
...
You may embed a Submit To Evaluation widget a Submit To Evaluation widget on a Wiki page to improve visibility of your Evaluation queue. The widget allows participants to submit to multiple Evaluation queues within a Project or a single Evaluation queue.
...