Introduction

Synapse Managed Plans require a new object to encapsulate the definition of the plan (the limits (storage, egress, others) to manage data and governance across projects. Organizations are the proposed solution.  Organizations are a paradigm that is used by other applications (GitHub, HuggingFace).  Organizations allow us to support users with complex data curation needs that span projects.  

What are Organizations? Organizations are objects that enforce Plan limits and encapsulate a collection of projects, storage locations or both.

Users may not desire Organizations (or managed plans as usage of Synapse was previously unrestrained). But once Organizations are deployed, they represent substantial assets that users will want to protect from misuse.

In order to protect Sage from runaway or unmanaged costs of data egress or storage, we need to implement controls on the system to prevent users from uploading too much data, egressing data at a rate faster than we can financially manage and support the managed plans through enforcement.

• Eliminate unlimited free storage and egress by Synapse users.

• Configurable manage total size of contents in Synapse to meet requirements of managed plans

• Display size limits and current size in understandable and actionable by users in SWC

• Prevent users from uploading additional content when project limit is reached

• Control access to a plan’s resources to authorized users

Glossary

Content: User-contributed files, datasets. (as opposed to Data which is too ambiguous)

Download: Download is defined as a request for a pre-signed URL.

Limits:  The restriction on data, egress, or other process, count etct.

For example, an Organization may have a limit on storage, egress, and/or other limits we may define in the future. 

Organization: An object to encapsulate the collection of users, project, teams and storage.  An Organization is defined by a Plan and enforces the Plan’s Limits.

Plan (Managed Plan, plans): A plan is the construct to map accounting and limits to an Organization.  All users of Synapse get a (free) Basic plan (and associated Organization). Users who wish to purchase a paid plan (Self-Managed or DCC) will need to contact Sage to create the Organization associated with the Plan.  The Plan accounts for the limits and length of the agreement, the Organization implements the Plan with Synapse.

Project: A Synapse project.  Definition of a project is not changed for this discussion.

Storage (buckets): Storage can be one of the following types:

Shared Storage: Shared storage is any storage managed by Sage on behalf of the user.  Shared storage MAY HAVE many Organizations' data included in it. Storage and Egress is limited by the Organization’s Plan. Sage’s shared storage bucket proddata is an example of shared storage.

Custom Storage: Storage managed by the user. Users control and manage the storage. Storage and egress costs are paid by the user and no limits are imposed by Synapse.  Data from custom storage does not count against the plan limits.

Private Storage: (previously BaaS) Storage managed by Sage on behalf of the Organization in the form of an S3 bucket that contains only that Organization’s content.  Storage and Egress is limited by the Organization’s Plan.  AWS Open Data is an example of private storage.

Storage Limit: The upper allowed amount of storage (in GB) for the total amount of data in the organization.

Egress Limit: The upper limit of egress of data from within projects in the organization (measured in GB/Year).

User Classes

1. Downloader Only - User who does not need a plan - only wishes to browse and access data.

2. Basic Plan User (aka: Independent Researchers) - User who wants a Basic Plan for publishing content. Each Certified User account has a single Default Organization as part of a Basic Plan. 

3. Multi-Plan User - User who has one Basic Plan, and a member of one or more Self-Managed Plans or DCC Plans. 

4. Legacy Plan User - User who has uploaded content into Synapse previous to the plan rollout date (##/##/2023) or is a member of a Legacy Plan.

Use Cases & Scenarios

Actors:

Data Contributors: Data contributors are those individuals who upload data. They must be Certified (in order to upload) AND must be a member of the organization to create a project, upload data into an Organization.

Data Users (Downloaders): Data users need to have no relationship to an Organization. They can access data from any Organization using the governance and controls that exist today.

Organization Adminstrators

Organization Managers (may or may not be Data Contributors): The Organization Manager(s) are responsible for controlling membership to the organization, setting up the default storage location.

Organization Members:

System (Synapse):

Scenario 1: New user to the system

Dr Fauci is a new cancer researcher with a grant for his new vaccine. He is researching appropriate repositories for his lab’s work. He finds Synapse.org, creates an account and Certifies his account. He now navigates to his Dashboard which lists an empty list of projects. He clicks on “Create New Project” and is offered to name a new Organization and Project. He’s not sure what an Organization is, but assumes its similar to his GitHub experience. Nonetheless, he clicks on the ? link to go to help and reads about Organizations and plans. He names his Organization “CancerOrg123” and his new project, “ProjectCancer1”. He now has a project and an Organization. After several days of use and kicking the tires, he decides that Synapse is his choice for his work, and he wants his research associates, Thing1 and Thing2 to upload data to his project. He goes back to his Synapse Organization page, and chooses “Invite users”, where he invites Thing1, (already a Synapse user) by his user name, and Thing2 (not a Synapse user) by email.

Thing1 and Thing2 create new projects and start uploading data into Dr Fauci’s org. After a few days, he sees that he is close to his 100GB limit of storage, he clicks on the Organizations menu on the left rail of Synapse and is shown his one organization “CancerOrg123” from there, he can view the Organization page, which describes his org as a Basic Plan and leads to the SYNSD Managed Plan form. He fills out the form to the best of his ability and clicks “Send”. He receives an email response from Sage, forms are flown, signatures are signed and a few days later, “CancerOrg123” is created as a Managed Plan with 500GB and he receives an email from Ann at Sage offering to help him configure the governance for his data.

As a Project owner, (necessarily an Organization Member), I must be able to see the storage amount used for my project and the limit and storage used for the organization in order to make decisions on how to control my data usage.

As a System, Synapse needs to track a collection of projects and storage locations associated with those projects. (Organizations)

As a System, Synapse can limit storage based on Organization in order to limit storage costs for Sage.

As a System, Synapse can limit egress based on Organization for Shared and Private storage locations in order to limit Egress costs for Sage.

As an Organization Manager, I want to invite other Synapse users to my Organization so that they create projects and upload data as part of my Organization.

As an Organization Manager, I want to invite other non-Synapse users via email to my Organization so that I can invite users who are not yet a user of Synapse.

As an Organization Manager, I want to remove a user from my organization maintaining the data they contributed. (Organization is the data owner, but data contributor just uploads data).

As an Organization Manager, I want Synapse to limit the size of data from uploaded into my Custom Storage from Synapse.

As an Organization Manager, I want to be able to rename my organization.

As an Organization Manager, I want to merge two organizations that I manage into a single organization.

As an Organization Manager, I want to remove empty project(s) from my organization, which reverts them to the project owner’s Basic Plan organization.

As an Organization Manager, I want to see usage against my organizations' storage limits broken out by project, storage and by contributor over time so that I can make decisions about my organization (like when to purchase more storage, etc).

As a System, Synapse SHALL NOT have projects outside of o

Functional Design

Organization Properties

Organizations have the following properties:

Project Information

Synapse MUST store and maintain project and organization relationships.

Synapse Projects MUST be a member of one (and only one) organization.

Projects MAY BE migrated between two organizations,

Hierarchy:

Projects to Organizations (Many to One)

A Project MUST BE a member of one (and only one) Organization

An Organization MAY HAVE multiple member Projects

User Accounts to Organizations (Many to Many)

An Organization MUST HAVE one or more member User Accounts

A User Account MAY BE a member of one or more Organizations

Storage to Organizations (Many to Many)

An Organization MUST HAVE one default storage.

A Storage MAY BE allocated to multiple Organizations

Organizations MAY HAVE more than one storage.

Organization User Privileges

Outside the privileges described here, there are no privileges or access required by users.  Users WILL BE able to access data using the existing ACL and AR methods, irrespective of Organizations. 

Organization Administrators

Organization Administrators have privileges spanning all Organizations.  They can administer the Organization system. All system Administrators SHOULD BE Organization Administrators

Organization Administrators MAY perform the following tasks:

• Create, Update or Delete Organizations

• Edit Properties (including Limits) of an Organization 

• Set values of properties (including override of default values)

• Assign users as Managers and Members of the organization

• All of the functions Organization Managers can perform

Organization Managers

Organization managers (those with Manage privileges on the Organization) MAY perform the following tasks on the Organization they have management privileges on

• Add or remove Users, Teams as Organization Members

• Rename the Organization 

• All of the functions Organization Members can perform

Organization Members

Organization Members are granted Create Project within the organization, equivalent to that of users within Synapse pre-organization. 

Users MUST BE Organization Members in order to create projects within an Organization. Once they create the project, behavior within the project is the same as pre-organization Synapse. 

Actions

Default Organization Creation (Basic Plan)

Organizations are created by the following actions:

• When a user without a default Organization creates their first project, this creates a Basic Plan and a Default Organization - The Organization should have a default name of the username & ”org” or username & ”Organization”

• A user with a default organization is an Organization Member, not an Organization Manager. 

Organization Creation by Admin

• An Organization Admin creates an Organization from the Plan Definition

Once a plan is created, an Organization Admin can create the organization, assign the Organization Managers according to the Plan Specification. This includes legacy plans. 

Renaming an Organization

An Organization can be renamed by an Organization Manager or Organization Administrator 

Adding Users, Teams to an Organization

Users and Teams can be added to an existing Organization by a user with Organization Management privileges. Being added to an Organization makes the users or users in the team Organization Members. 

(note: Default Organizations have no Organization Manager, and cannot add additional members)

Organization Privileges do not overwrite Project or entity privileges.  For example, an Organization Member may be granted access to a project (such as a private project). This user will have access to the project, but WILL NOT have access to other projects within the Organization without further action. 

Removing Users, Teams from an Organization 

Users and teams can be removed from an Organization by a User with Organization Management privilege. 

Users MAY remove themselves from an Organization, unless it is their Default Organization.

Teams may be removed from an Organization by user with Organization Management privilege, or by a Team Owner. 

Creating Projects in an “Organized” Synapse

Once Organizations are implemented in Synapse, existing users MUST choose the Organization the project belongs to at creation time.  Failing to indicate an Organization MUST result in an error and the project is not created. 

A user creating a project MUST BE an Organization Member to add a project to that organization or add data to an existing project.

Moving Data Between Organizations

Moving data between organizations is accomplished just as a user would do today. Users MUST HAVE permissions to access the data at the source project and target project. 

Managing Storage in an “Organized” Synapse

Organizations MUST HAVE a default storage selected when the organization is created. 

• Two or more organizations MAY NOT utilize the same private storage.

• With the exception of Default Organizations, two or more organizations MAY NOT utilize the same shared storage.

• Two or more organizations MAY utilize the same custom storage.

Projects MAY choose separate storage from their organization, but must follow these rules:

A project MUST NOT reference another organization’s private storage

A project MAY reference another organization’s custom storage (dependent on access privileges)

Error Cases

Setup a project in an organization using non-organization storage

Implementing Plan Limits with Organizations in Synapse

Organizations utilize the following behaviors when implementing Plan limits.

Storage Limits

Calculating Storage Used

Storage limits are calculated against any file handle in shared or private storage.  Content stored in Custom Storage is not counted against the limit. For example, take a simple organization with a Self-Managed plan (100 GB limit) with 3 projects, (Projects A, B, & C)  Project A (30GB) has its storage in private storage A.  Project B (40GB) has its storage in private Storage B.  Project C (700GB) has its storage in custom storage.

The Organization has a total storage of 770GB with a Storage Limit of 100GB.

The Organization is 30GB under its limit.

We calculate the total stored in shared and private storage 

Project A: 30GB + Project B: 40GB = 70GB

Reaching Storage Limit 

When an organization reaches its Storage Limit, Members are prevented from uploading additional content. 

Should the Organization need to utilize more than 100% of the plan’s storage limit, the Organization Manager MAY choose to acquire additional storage limits (TBD).

Once the storage limit is reached, Organization Members MAY perform the following 

• Delete data 

• Migrate one or more project to custom storage

• Add new projects with Custom Storage

Storage Limit Notifications

When an organization is at > = NN% of the Storage Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.  

Egress Limits

Egress limit

Calculating Egress Used 

Egress limits are calculated against any content in shared or private storage (except that stored in AWS Open Data).  Content transferred from Custom Storage is not counted against the limit.

Egress limits are defined by TB/Year of egress, starting with start date of the plan.

Egress MUST BE calculated for all organizations in order to report usage stats to users and stakeholders.

Synapse WILL NOT enforce egress limits from the following:

• Custom storage

• Storage within AWS Open Data

• Any future private or shared storage that designated to be “unlimited”

Reaching Egress Limit

When an organization reaches its Egress Limit, users are prevented from transferring data from shared or private storage.  Users are still able to shift data within the AWS region the shared or private storage.

NOTE: Content stored in the AWS Open Data bucket is not counted against the Egress Limit 

Should the Egress Limit be reached, Organization Members MAY perform the following to enable egress:

• Migrate one or more projects to Custom Storage

• Wait for the Egress limit to reset.

Egress Limit Notifications

When an organization is at > = NN% of the egress Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.  

Users MUST get a notification in the Download cart when the contents of their cart will exceed the Egress Limit with directions on how to avoid the limit.

User Limit Notification

When a downloader attempts to download, Synapse will be required to check the Egress limit. The user should receive an error that the download cannot proceed.

Organization Managers should be notified that users cannot further download.

Egress Yearly Reset

On the yearly mark (annually from the start date of the active plan), the egress limit will reset (as long as the plan is still active) with no rollover of unused credits.

Updating Limits

Organization Admins can update the limits of an organization:

Notifications

Transitioning Existing Data to an Organized Synapse

Data and content exist in pre-organization Synapse today, this “Legacy “ content MUST BE allocated to appropriate new Organizations.  This will require some data analysis.

Basic Plan Requirements

Qualified Synapse users MUST NOT BE required to contact Sage in order to create and begin utilizing a Basic Plan.

User Experience for users migrating from a Basic Plan to a Self-Managed Plan SOULD BE easy and understandable.

Open Questions

Please put in any open questions you have and I will address - Kevin

FAQs

Can I have a user without an Organization?

Yes, creating a user does not require an organization (many users may be download only).

Can I move a project from one org to another?

No.

Can a user be a member of multiple Organizations?

Yes, so can a Team.

Does this mean I have to be a member of an Organization to access data stored within?

No, ARs and ACLs remain on Projects, entities, folders, tables, etc. 

Can I create a project without an Organization?

No.

Do I need a plan to create an Organization?

Every Certified User gets a default Organization with a Basic Plan.  Beyond this plan, yes, users will need a plan to have an Organization.