An Evaluation evaluation queue allows for people you to submit Synapse Files Synapse files or Docker images . for evaluation. They are designed to support open-access data analysis and modeling challenges in Synapse. This framework provides tools for administrators to collect and analyze data models created by Synapse users for a specific goal or purpose.
...
To create a queue, you must first create a Synapse Project or have edit permissions on Synapse Challenge project and have edit permissions on an existing Project. To create project.
For instructions on setting up a Synapse Project, follow the instructions on the Project and Data Management page.Once you’ve created your project, navigate to it and add /challenge to the url (e.g. www.synapse.org/#!Synapse:syn12345/challenge). Click see /wiki/spaces/DOCS/pages/2055471258.
If you do not see the challenge tab in your project:
Navigate to the bottom of the page and click experimental mode: off to turn on experimental mode.
Click Project Tools in the right corner and select Run Challenge and follow instructions.
Once your project has the Challenge tab, navigate to it and click Challenge Tools in the right corner and select Add Evaluation Queue.
...
An Evaluation evaluation queue can take several parameters that you can use to customize your preferences.
Name: Unique name of the evaluation
Description: A short description of the evaluation
Submission Instructionsinstructions: Message to display to users detailing acceptable formatting for submissions.
Submission Receipt Messagereceipt message: Message to display to users upon submission
...
submission—the name of your evaluation queue MUST be unique, otherwise the queue will not be created
...
Setting Quotas on an Evaluation Queue
Optionally, you can restrict submissions by adding a quota to each round of your challenge.
An Evaluation evaluation queue can only have one quota. You may must specify some required parameters: the length of time the queue is open, the start date, round duration, and number of rounds. These are required parameters. It is optional to set a submission limit.
...
Share an Evaluation Queue
Each Evaluation evaluation has sharing settings, which limit who can interact with the Evaluationevaluation.
Administrator sharing should be tightly restricted, as it includes authority to delete the entire Evaluation evaluation queue and its contents. These users also have the ability to download all of the submissions.
Can Score allows for individuals to download all of the submissions
Can Submit allows for teams or individuals to submit to the Evaluationevaluation, but doesn’t have access to any of the submissions.
Can View allows for teams or individuals to view the submissions on a leaderboard.
To set the sharing settings, go to the Challenge tab to view your list of Evaluationsevaluations. Click on the Share button per Evaluation evaluation and share it with the teams or individuals you would like.
| Info |
|---|
Important: When someone submits to an Evaluation Queueevaluation queue, a copy of the submission is made, so a person with Administrator “Administrator” or Can Score “Can Score” access will be able to download the submission even if the submitter deletes the entity. |
...
Users are only able to submit to a queue if they have
can submitpermissions to it. If you have the ability to modify the permissions of a queue, you will still be able to submit to the queue due to youradministratoraccess.If the quota is set so the current date exceeds the the round start + round duration, no one will be able to submit to the queue. This includes users with administrator permissions.
Deleting a queue will also discontinue the ability to submit to it. Be careful when doing this, as deleting a queue is irreversible - and you will lose all its submissions.
Submitting to an Evaluation Queue
Any Synapse entity may be submitted to an Evaluation Queueevaluation queue.
In the R and Python examples, you need to know the ID of the evaluation queue. This ID must be provided to you by administrators of the queue.
The submission function takes two optional parameters: name and team. Name can be provided to customize the submission. The submission name is often used by participants to identify their submissions. If a name is not provided, the name of the entity being submitted will be used. As an example, if you submit a File file named testfile.txt, and the name of the submission isn’t specified, it will default to testfile.txt. Team names can be provided to recognize a group of contributors.
...
Every submission you make to an Evaluation evaluation queue has a unique ID. This ID should not be confused with Synapse IDs which start with the prefix “syn” (for example, syn12345678). All submissions have a Submission and SubmissionStatus object.
Navigate to a File file in Synapse and click on File Tools in the upper right-hand corner. Select Submit To Challenge to pick the challenge for your submission. Follow the provided steps to complete your submission.
...
Submissions can be viewed and shared with users through Submission Views creating through submission views creating dynamic leaderboards. Submission annotations can be added to a SubmissionStatus object and are automatically indexed in the view.
...
You may embed a Submit To Evaluation widget on a Wiki wiki page to improve visibility of your Evaluation evaluation queue. The widget allows participants to submit to multiple Evaluation evaluation queues within a Project project or a single Evaluation evaluation queue.
Currently, this Wiki wiki widget is required to submit Synapse Projects projects to an Evaluation evaluation queue. Synapse Docker repositories can not be submitted through this widget.
...
The “Evaluation Queue unavailable message” is customizable. A queue may appear unavailable to a user if:
The Project project doesn’t have any Evaluation evaluation queues.
The user does not have permission to view a Project’s Evaluation queues. Learn more about sharing settings.The Evaluation evaluation queue quota is set such that a user can not submit to the queue. Learn more about quotas.
See Also
To learn how to create a Wiki page, please visit the Wikis article.
...
The user does not have permission to view a project’s evaluation queues.
Learn more about sharing settings /wiki/spaces/DOCS/pages/2024276030.