...
Synapse Managed Plans require a new object to encapsulate the definition of the plan (the limits (storage, egress, others) to manage data and users governance across projects. Organizations are the proposed solution. Organizations are a paradigm that is used by other applications (GitHub, HuggingFace). Organizations allow us to support users with complex data curation needs that span projects.
What are Organizations? Organizations are objects that encapsulate Projects, Users, Teams and Storage in order to enforce Plans. Organizations serve no other purpose, they are not intended for governance or access (other than project creation as part of plan enforcement), data curation or dissemination.
Glossary
Content: User-contributed files, datasets. (as opposed to Data which is too ambiguous)
Limit: The upper allowed amount of storage (in GB), or egress (in GB/Month) or projects.
For example, an Organization may have a limit on storage, egress, certified users, and/or other limits.
Organization: An object to encapsulate the collection of users, project, teams and storage. An Organization is defined by a Plan and enforces the Plan’s Limits.
Plan (Managed Plan, plans): A plan is the construct to map accounting and limits to an Organization. All users of Synapse can create and access a (free) Basic plan (and associated Organization). Users who wish to purchase a paid plan (Self-Managed or DCC) will need to contact Sage to create the Organization associated with the Plan. The Plan accounts for the limits and length of the agreement, the Organization implements the Plan with Synapse.
Project: A Synapse project. Definition of a project is not changed for this discussion.
Storage (buckets): Storage can be one of the following types:
Shared Storage: Shared storage is any storage managed by Sage on behalf of the user. Shared storage MAY HAVE many Organizations data included in it. Storage and Egress is limited by the Organization’s Plan. Sage’s shared storage bucket proddata, as well as the AWS Open Data space managed by sage are examples of shared storage.
Custom Storage: Storage managed by the user. Users control and manage the storage. Storage and egress costs are on the user and no limits are imposed by Synapse. Data from enforce Plan limits and encapsulate a collection of projects, storage locations or both.
Users may not desire Organizations (or managed plans as usage of Synapse was previously unrestrained). But once Organizations are deployed, they represent substantial assets that users will want to protect from misuse.
In order to protect Sage from runaway or unmanaged costs of data egress or storage, we need to implement controls on the system to prevent users from uploading too much data, egressing data at a rate faster than we can financially manage and support the managed plans through enforcement.
Eliminate unlimited free storage and egress by Synapse users.
Configurable manage total size of contents in Synapse to meet requirements of managed plans
Display size limits and current size in understandable and actionable by users in SWC
Prevent users from uploading additional content when project limit is reached
Control access to a plan’s resources to authorized users
Glossary
Content: User-contributed files, datasets. (as opposed to Data which is too ambiguous)
Download: Download is defined as a request for a pre-signed URL.
Limits: The restriction on data, egress, or other process, count etct.
For example, an Organization may have a limit on storage, egress, and/or other limits we may define in the future.
Organization: An object to encapsulate the collection of users, project, teams and storage. An Organization is defined by a Plan and enforces the Plan’s Limits.
Plan (Managed Plan, plans): A plan is the construct to map accounting and limits to an Organization. All users of Synapse get a (free) Basic plan (and associated Organization). Users who wish to purchase a paid plan (Self-Managed or DCC) will need to contact Sage to create the Organization associated with the Plan. The Plan accounts for the limits and length of the agreement, the Organization implements the Plan with Synapse.
Project: A Synapse project. Definition of a project is not changed for this discussion.
Storage (buckets): Storage can be one of the following types:
Shared Storage: Shared storage is any storage managed by Sage on behalf of the user. Shared storage MAY HAVE many Organizations' data included in it. Storage and Egress is limited by the Organization’s Plan. Sage’s shared storage bucket proddata is an example of shared storage.
Custom Storage: Storage managed by the user. Users control and manage the storage. Storage and egress costs are paid by the user and no limits are imposed by Synapse. Data from custom storage does not count against the plan limits.
Private Storage: (previously BaaS) Storage managed by Sage on behalf of the Organization in the form of an S3 bucket that contains only that Organization’s content. Storage and Egress is limited by the Organization’s Plan.
User Classes
...
Downloader Only - User who does not need a plan - only wishes to browse and access data.
...
Basic Plan User (aka: Independent Researchers) - User who wants a Basic Plan for publishing content. Each Certified User account has a single Default Organization as part of a Basic Plan.
...
Multi-Plan User - User who has one Basic Plan, and a member of one or more Self-Managed Plans or DCC Plans.
...
AWS Open Data is an example of private storage.
Storage Limit: The upper allowed amount of storage (in GB) for the total amount of data in the organization.
Egress Limit: The upper limit of egress of data from within projects in the organization (measured in GB/Year).
User Classes
Downloader Only - User who does not need a plan - only wishes to browse and access data.
Basic Plan User (aka: Independent Researchers) - User who wants a Basic Plan for publishing content. Each Certified User account has a single Default Organization as part of a Basic Plan- only wishes to browse and access data and is included as an uploader on one or more plans owned by others..
Multi-Plan User - User who has one Basic Plan, and a member of one or more Self-Managed Plans or DCC Plans.
Legacy Plan User - User who has uploaded content into Synapse previous to the plan rollout date (##/##/2023) or is a member of a Legacy Plan.
Functional Design
Organization Properties
Organizations have the following properties:
...
PropertyName
...
Type
...
Source
...
Default
...
Definition
...
OrganizationID
...
number
...
Synapse
...
Unique Synapse ID of the Organization
...
PlanID
...
number
...
Plan System
...
Unique Synapse ID of the corresponding Plan. Each Organization MUST HAVE a corresponding Plan.
...
Organization name
...
String
...
User Defined
...
UserName&”Organization” for basic plans
...
Friendly Name of the Organization
...
CreatedOn
...
datetime
...
Synapse
...
Datetime the Organization was created
...
CreatedBy
...
number
Synapse
...
Unique UserID of the User that created the Organization
...
OwnerID
...
number
...
Plan System
...
Plan Owner's SynID
...
UserID of the Synapse User who owns /manages the Organization.
...
StorageLimit
...
number
...
Plan System
...
Dependent on Plan
...
Total Limit of storage (in GB) allowed in Shared & Private Storage
...
EgressLimit
...
number
...
Plan System
...
Dependent on Plan
...
Total Egress (GB/Time) allowed for data within the Organization’s boundaries from within Shared & Private storage.
...
Default Storage
...
Default storage bucket created at the Organization creation time (can be overridden at the project level for a project)
...
CertifiedUserLimit
...
Number
...
1 for Default organization (basic plans), null for all others
Project Information
Synapse MUST store and maintain project and organization relationships.
Hierarchy:
Projects to Organizations (Many to One)
A Project MUST BE a member of one (and only one) Organization
An Organization MAY HAVE multiple member Projects
User Accounts to Organizations (Many to Many)
An Organization MUST HAVE one or more member User Accounts
A User Account MAY BE a member of one or more Organizations
Storage to Organizations (Many to Many)
An Organization MUST HAVE one default storage.
A Storage MAY BE allocated to multiple Organizations
Organization User Privileges
Outside the privileges described here, there are no privileges or access required by users. Users WILL BE able to access data using the existing ACL and AR methods, irrespective of Organizations.
Organization Administrators
Organization Administrators have privileges spanning all Organizations. They can administer the Organization system. All system Administrators SHOULD BE Organization Administrators
Organization Administrators MAY perform the following tasks:
Create, Update or Delete Organizations
Edit Properties (including Limits) of an Organization
Set values of properties (including override of default values)
Assign users as Managers and Members of the organization
All of the functions Organization Managers can perform
Organization Managers
Organization managers (those with Manage privileges on the Organization) MAY perform the following tasks on the Organization they have management privileges on
Add or remove Users, Teams as Organization Members
Rename the Organization
All of the functions Organization Members can perform
Organization Members
Organization Members are granted Create Project within the organization, equivalent to that of users within Synapse pre-organization.
Users MUST BE Organization Members in order to create projects within an Organization. Once they create the project, behavior within the project is the same as pre-organization Synapse.
Use Cases
Creating an Organization
Organizations are created by the following actions:
When a Certified User without a default Organization creates their first project , this creates a Basic Plan and a Default Organization - The Organization should have a default name of the username & ”org” or username & ”Organization”
A user with a default organization is an Organization Member, not a Organziation Manager.
An Organization Admin creates an Organization from the Plan Definition
Once a plan is created, an Organization Admin can create the organization, assign the Organization Managers according to the Plan Specification. This includes legacy plans.
The Plan System creates the Organization automatically based on the Plan Definition. (FUTURE)
Renaming an Organization
An Organization can be renamed by an Organization Manager or Organization Administrator
Adding Users, Teams to an Organization
Users and Teams can be added to an existing Organization by a user with Organization Management privileges. Being added to an Organization makes the users or users in the team Organization Members.
(note: Default Organizations have no Organization Manager, and cannot add additional members)
Organization Privileges do not overwrite Project or entity privileges. For example, an Organization Member may be granted access to a project (such as a private project). This user will have access to the project, but WILL NOT have access to other projects within the Organization without further action.
Removing Users, Teams from an Organization
Users and teams can be removed from an Organization by a User with Organization Management privilege.
Users MAY remove themselves from an Organization, unless it is their Default Organization.
Teams may be removed from an Organization by user with Organization Management privilege, or by a Team Owner.
Creating Projects in an “Organized” Synapse
Once Organizations are implemented in Synapse, existing users MUST choose the Organization the project belongs to at creation time. Failing to indicate an Organization MUST result in an error and the project is not created.
A user creating a project MUST BE an Organization Member to add a project to that organization.
Moving Data Between Organizations
Moving data between organizations is accomplished just as a user would do today. Users MUST HAVE permissions to access the data at the source project and target project.
Managing Storage in an “Organized” Synapse
Organizations MUST HAVE a default storage selected when the organization is created.
Two or more organizations MAY NOT utilize the same private storage.
Two or more organizations MAY utilize the same shared storage.
Two or more organizations MAY utilize the same custom storage.
Projects MAY choose separate storage from their organization, but must follow these rules:
A project MUST NOT reference another organization’s private storage
A project MAY reference another organization’s custom storage (dependent on access privileges)
Implementing Plan Limits with Organizations in Synapse
Organizations utilize the following behaviors when implementing Plan limits.
Storage Limits
Calculating Storage Used
Storage limits are calculated against any content in shared or private storage. Content stored in Custom Storage is not counted against the limit. For example, take a simple organization with a Self-Managed plan (100 GB limit) with 3 projects, (Projects A, B, & C) Project A (30GB) has its storage in shared storage. Project B (40GB) has its storage in private Storage. Project C (700GB) has its storage in custom storage.
The Organization has a total storage of 770GB with a Storage Limit of 100GB.
The Organization is 30GB under its limit.
We calculate the total stored in shared and private storage
Project A: 30GB + Project B: 40GB = 70GB
Calculating Egress Used
Egress limits are calculated against any content in shared or private storage (except that stored in AWS Open Data). Content egressed from Custom Storage is not counted against the limit. For example, take a simple organization with a Self-Managed plan (100 GB limit) with 3 projects, (Projects A, B, & C) Project A (30GB) has its storage in shared storage. Project B (40GB) has its storage in private Storage. Project C (700GB) has its storage in custom storage.
The Organization has a total storage of 770GB with a Storage Limit of 100GB.
The Organization is 30GB under its limit.
We calculate the total stored in shared and private storage
Project A: 30GB + Project B: 40GB = 70GB
Reaching Storage Limit
When an organization reaches its Storage Limit, Members are prevented from uploading additional content.
Should the Organization need to utilize more than 100% of the plan’s storage limit, the Organization Manager MAY choose to acquire additional storage limits (TBD).
Once the storage limit is reached, Organization Members MAY perform the following
Delete data
Migrate one or more project to custom storage
Add new projects with Custom Storage
Reaching Egress Limit
When an organization reaches its Egress Limit, users are prevented from egressing data from shared or private storage. Users are still able to shift data within the AWS region the shared or private storage.
NOTE: Content stored in the AWS Open Data bucket is not counted against the Egress Limit
Should the Egress Limit be reached, Organization Members MAY perform the following to enable egress:
Migrate one or more projects to Custom Storage
Wait for the Egress limit to reset.
Storage Limit Notifications
When an organization is at > = NN% of the Storage Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.
Egress Limit Notifications
When an organization is at > = NN% of the Storage Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.
Users MUST get a notification in the Download cart when the contents of their cart will exceed the Egress Limit with directions on how to avoid the limit.
Migrating Existing Data to an Organized Synapse
...
Use Cases & Scenarios
Actors:
Data Contributors: Data contributors are those individuals who upload data. They must be Certified (in order to upload) AND must be a member of the organization to create a project, upload data into an Organization.
Data Users (Downloaders): Data users need to have no relationship to an Organization. They can access data from any Organization using the governance and controls that exist today.
Organization Adminstrators
Organization Managers (may or may not be Data Contributors): The Organization Manager(s) are responsible for controlling membership to the organization, setting up the default storage location.
Organization Members:
System (Synapse):
Scenario 1: New user to the system
Dr Fauci is a new cancer researcher with a grant for his new vaccine. He is researching appropriate repositories for his lab’s work. He finds Synapse.org, creates an account and Certifies his account. He now navigates to his Dashboard which lists an empty list of projects. He clicks on “Create New Project” and is offered to name a new Organization and Project. He’s not sure what an Organization is, but assumes its similar to his GitHub experience. Nonetheless, he clicks on the ? link to go to help and reads about Organizations and plans. He names his Organization “CancerOrg123” and his new project, “ProjectCancer1”. He now has a project and an Organization. After several days of use and kicking the tires, he decides that Synapse is his choice for his work, and he wants his research associates, Thing1 and Thing2 to upload data to his project. He goes back to his Synapse Organization page, and chooses “Invite users”, where he invites Thing1, (already a Synapse user) by his user name, and Thing2 (not a Synapse user) by email.
Thing1 and Thing2 create new projects and start uploading data into Dr Fauci’s org. After a few days, he sees that he is close to his 100GB limit of storage, he clicks on the Organizations menu on the left rail of Synapse and is shown his one organization “CancerOrg123” from there, he can view the Organization page, which describes his org as a Basic Plan and leads to the SYNSD Managed Plan form. He fills out the form to the best of his ability and clicks “Send”. He receives an email response from Sage, forms are flown, signatures are signed and a few days later, “CancerOrg123” is created as a Managed Plan with 500GB and he receives an email from Ann at Sage offering to help him configure the governance for his data.
As a Project owner, (necessarily an Organization Member), I must be able to see the storage amount used for my project and the limit and storage used for the organization in order to make decisions on how to control my data usage.
As a System, Synapse needs to track a collection of projects and storage locations associated with those projects. (Organizations)
As a System, Synapse can limit storage based on Organization in order to limit storage costs for Sage.
As a System, Synapse can limit egress based on Organization for Shared and Private storage locations in order to limit Egress costs for Sage.
As an Organization Manager, I want to invite other Synapse users to my Organization so that they create projects and upload data as part of my Organization.
As an Organization Manager, I want to invite other non-Synapse users via email to my Organization so that I can invite users who are not yet a user of Synapse.
As an Organization Manager, I want to remove a user from my organization maintaining the data they contributed. (Organization is the data owner, but data contributor just uploads data).
As an Organization Manager, I want Synapse to limit the size of data from uploaded into my Custom Storage from Synapse.
As an Organization Manager, I want to be able to rename my organization.
As an Organization Manager, I want to merge two organizations that I manage into a single organization.
As an Organization Manager, I want to remove empty project(s) from my organization, which reverts them to the project owner’s Basic Plan organization.
As an Organization Manager, I want to see usage against my organizations' storage limits broken out by project, storage and by contributor over time so that I can make decisions about my organization (like when to purchase more storage, etc).
As a System, Synapse SHALL NOT have projects outside of o
Functional Design
Organization Properties
Organizations have the following properties:
PropertyName | Type | Source | Default | Definition |
OrganizationID | number | Synapse | Unique Synapse ID of the Organization | |
PlanID | number | Plan System | Unique Synapse ID of the corresponding Plan. Each Organization MUST HAVE a corresponding Plan. | |
Organization name | String | User Defined | UserName&”Organization” for basic plans | Friendly Name of the Organization |
CreatedOn | datetime | Synapse | Datetime the Organization was created | |
CreatedBy | number | Synapse | Unique UserID of the User that created the Organization | |
OwnerID | number | Plan System | Plan Owner's SynID | UserID of the Synapse User who owns /manages the Organization. |
StorageLimit | number | Plan System | Dependent on Plan | Total Limit of storage (in GB) allowed in Shared & Private Storage. Null indicates unlimited. |
EgressLimit | number | Plan System | Dependent on Plan | Total Egress (GB/Time) allowed for data within the Organization’s boundaries from within Shared & Private storage. Null indicates unlimited |
Default Storage | Default storage bucket defined at the Organization creation time (can be overridden at the project level for a project) | |||
CertifiedUserLimit | Number | 1 for Default organization (basic plans), null for all others | Null indicates unlimited. This will be used for Legacy Basic Plans. |
Project Information
Synapse MUST store and maintain project and organization relationships.
Synapse Projects MUST be a member of one (and only one) organization.
Projects MAY BE migrated between two organizations,
Hierarchy:
Projects to Organizations (Many to One)
A Project MUST BE a member of one (and only one) Organization
An Organization MAY HAVE multiple member Projects
User Accounts to Organizations (Many to Many)
An Organization MUST HAVE one or more member User Accounts
A User Account MAY BE a member of one or more Organizations
Storage to Organizations (Many to Many)
An Organization MUST HAVE one default storage.
A Storage MAY BE allocated to multiple Organizations
Organizations MAY HAVE more than one storage.
Organization User Privileges
Outside the privileges described here, there are no privileges or access required by users. Users WILL BE able to access data using the existing ACL and AR methods, irrespective of Organizations.
Organization Administrators
Organization Administrators have privileges spanning all Organizations. They can administer the Organization system. All system Administrators SHOULD BE Organization Administrators
Organization Administrators MAY perform the following tasks:
Create, Update or Delete Organizations
Edit Properties (including Limits) of an Organization
Set values of properties (including override of default values)
Assign users as Managers and Members of the organization
All of the functions Organization Managers can perform
Organization Managers
Organization managers (those with Manage privileges on the Organization) MAY perform the following tasks on the Organization they have management privileges on
Add or remove Users, Teams as Organization Members
Rename the Organization
All of the functions Organization Members can perform
Organization Members
Organization Members are granted Create Project within the organization, equivalent to that of users within Synapse pre-organization.
Users MUST BE Organization Members in order to create projects within an Organization. Once they create the project, behavior within the project is the same as pre-organization Synapse.
Actions
Default Organization Creation (Basic Plan)
Organizations are created by the following actions:
When auser without a default Organization creates their first project, this creates a Basic Plan and a Default Organization - The Organization should have a default name of the username & ”org” or username & ”Organization”
A user with a default organization is an Organization Member, not an Organization Manager.
Organization Creation by Admin
An Organization Admin creates an Organization from the Plan Definition
Once a plan is created, an Organization Admin can create the organization, assign the Organization Managers according to the Plan Specification. This includes legacy plans.
Info |
---|
The Plan System creates the Organization automatically based on the Plan Definition. (FUTURE) |
Renaming an Organization
An Organization can be renamed by an Organization Manager or Organization Administrator
Adding Users, Teams to an Organization
Users and Teams can be added to an existing Organization by a user with Organization Management privileges. Being added to an Organization makes the users or users in the team Organization Members.
(note: Default Organizations have no Organization Manager, and cannot add additional members)
Organization Privileges do not overwrite Project or entity privileges. For example, an Organization Member may be granted access to a project (such as a private project). This user will have access to the project, but WILL NOT have access to other projects within the Organization without further action.
Removing Users, Teams from an Organization
Users and teams can be removed from an Organization by a User with Organization Management privilege.
Users MAY remove themselves from an Organization, unless it is their Default Organization.
Teams may be removed from an Organization by user with Organization Management privilege, or by a Team Owner.
Creating Projects in an “Organized” Synapse
Once Organizations are implemented in Synapse, existing users MUST choose the Organization the project belongs to at creation time. Failing to indicate an Organization MUST result in an error and the project is not created.
A user creating a project MUST BE an Organization Member to add a project to that organization or add data to an existing project.
Moving Data Between Organizations
Moving data between organizations is accomplished just as a user would do today. Users MUST HAVE permissions to access the data at the source project and target project.
Managing Storage in an “Organized” Synapse
Organizations MUST HAVE a default storage selected when the organization is created.
Two or more organizations MAY NOT utilize the same private storage.
With the exception of Default Organizations, two or more organizations MAY NOT utilize the same shared storage.
Two or more organizations MAY utilize the same custom storage.
Projects MAY choose separate storage from their organization, but must follow these rules:
A project MUST NOT reference another organization’s private storage
A project MAY reference another organization’s custom storage (dependent on access privileges)
Error Cases
Setup a project in an organization using non-organization storage
Implementing Plan Limits with Organizations in Synapse
Organizations utilize the following behaviors when implementing Plan limits.
Storage Limits
Calculating Storage Used
Storage limits are calculated against any file handle in shared or private storage. Content stored in Custom Storage is not counted against the limit. For example, take a simple organization with a Self-Managed plan (100 GB limit) with 3 projects, (Projects A, B, & C) Project A (30GB) has its storage in private storage A. Project B (40GB) has its storage in private Storage B. Project C (700GB) has its storage in custom storage.
The Organization has a total storage of 770GB with a Storage Limit of 100GB.
The Organization is 30GB under its limit.
We calculate the total stored in shared and private storage
Project A: 30GB + Project B: 40GB = 70GB
Reaching Storage Limit
When an organization reaches its Storage Limit, Members are prevented from uploading additional content.
Should the Organization need to utilize more than 100% of the plan’s storage limit, the Organization Manager MAY choose to acquire additional storage limits (TBD).
Once the storage limit is reached, Organization Members MAY perform the following
Delete data
Migrate one or more project to custom storage
Add new projects with Custom Storage
Storage Limit Notifications
When an organization is at > = NN% of the Storage Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.
Egress Limits
Egress limit
Calculating Egress Used
Egress limits are calculated against any content in shared or private storage (except that stored in AWS Open Data). Content transferred from Custom Storage is not counted against the limit.
Egress limits are defined by TB/Year of egress, starting with start date of the plan.
Egress MUST BE calculated for all organizations in order to report usage stats to users and stakeholders.
Synapse WILL NOT enforce egress limits from the following:
Custom storage
Storage within AWS Open Data
Any future private or shared storage that designated to be “unlimited”
Info |
---|
Egress WILL BE calculated on a daily basis, not realtime. |
Reaching Egress Limit
When an organization reaches its Egress Limit, users are prevented from transferring data from shared or private storage. Users are still able to shift data within the AWS region the shared or private storage.
NOTE: Content stored in the AWS Open Data bucket is not counted against the Egress Limit
Should the Egress Limit be reached, Organization Members MAY perform the following to enable egress:
Migrate one or more projects to Custom Storage
Wait for the Egress limit to reset.
Egress Limit Notifications
When an organization is at > = NN% of the egress Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.
Users MUST get a notification in the Download cart when the contents of their cart will exceed the Egress Limit with directions on how to avoid the limit.
User Limit Notification
When a downloader attempts to download, Synapse will be required to check the Egress limit. The user should receive an error that the download cannot proceed.
Organization Managers should be notified that users cannot further download.
Egress Yearly Reset
On the yearly mark (annually from the start date of the active plan), the egress limit will reset (as long as the plan is still active) with no rollover of unused credits.
Updating Limits
Organization Admins can update the limits of an organization:
Notifications
Transitioning Existing Data to an Organized Synapse
Data and content exist in pre-organization Synapse today, this “Legacy “ content MUST BE allocated to appropriate new Organizations. This will require some data analysis.
Basic Plan Requirements
Qualified Synapse users MUST NOT BE required to contact Sage in order to create and begin utilizing a Basic Plan.
User Experience for users migrating from a Basic Plan to a Self-Managed Plan SOULD BE easy and understandable.
Open Questions
Please put in any open questions you have and I will address - Kevin
FAQs
Can I have a user without an Organization?
...
Can I move a project from one org to another?Yes, if you have appropriate access to the source project and are a member of the target organization
No.
Info |
---|
May consider this if both organizations are using the same storage and the user is a member of each and we get requests. |
Can a user be a member of multiple Organizations?
...