Introduction

Synapse Managed Plans require a new object to encapsulate the definition of the plan (the limits (storage, egress, others) to manage data and users across projects. Organizations are the proposed solution.  Organizations are a paradigm that is used by other applications (GitHub, HuggingFace).  Organizations allow us to support users with complex data curation needs that span projects.  

What are Organizations? Organizations are objects that encapsulate Projects, Users, Teams and Storage in order to enforce Plans.  Organizations serve no other purpose, they are not intended for governance or access (other than project creation as part of plan enforcement), data curation or dissemination. 

Glossary

Content: User-contributed files, datasets. (as opposed to Data which is too ambiguous)

Limit:  The upper allowed amount of storage (in GB), or egress (in GB/Month) or projects.

For example, an Organization may have a limit on storage, egress, certified users, and/or other limits. 

Organization: An object to encapsulate the collection of users, project, teams and storage.  An Organization is defined by a Plan and enforces the Plan’s Limits.

Plan (Managed Plan, plans): A plan is the construct to map accounting and limits to an Organization.  All users of Synapse can create and access a (free) Basic plan (and associated Organization). Users who wish to purchase a paid plan (Self-Managed or DCC) will need to contact Sage to create the Organization associated with the Plan.  The Plan accounts for the limits and length of the agreement, the Organization implements the Plan with Synapse.

Project: A Synapse project.  Definition of a project is not changed for this discussion.

Storage (buckets): Storage can be one of the following types:

Shared Storage: Shared storage is any storage managed by Sage on behalf of the user.  Shared storage MAY HAVE many Organizations' data included in it. Storage and Egress is limited by the Organization’s Plan. Sage’s shared storage bucket proddata is an example of shared storage.

Custom Storage: Storage managed by the user. Users control and manage the storage. Storage and egress costs are paid by the user and no limits are imposed by Synapse.  Data from custom storage does not count against the plan limits.

Private Storage: Storage managed by Sage on behalf of the Organization in the form of an S3 bucket that contains only that Organization’s content.  Storage and Egress is limited by the Organization’s Plan.  AWS Open Data is an example of private storage.

User Classes

1. Downloader Only - User who does not need a plan - only wishes to browse and access data.

2. Basic Plan User (aka: Independent Researchers) - User who wants a Basic Plan for publishing content. Each Certified User account has a single Default Organization as part of a Basic Plan. 

3. Multi-Plan User - User who has one Basic Plan, and a member of one or more Self-Managed Plans or DCC Plans. 

4. Managed Plan Only User - A User who does not create their own Organization as part of a Basic Plan - only wishes to browse and access data and is included as an uploader on one or more plans owned by others.

5. Legacy Plan User - User who has uploaded content into Synapse previous to the plan rollout date (##/##/2023) or is a member of a Legacy Plan.

Functional Design

Organization Properties

Organizations have the following properties:

Project Information

Synapse MUST store and maintain project and organization relationships.

Hierarchy:

Projects to Organizations (Many to One)

A Project MUST BE a member of one (and only one) Organization

An Organization MAY HAVE multiple member Projects

User Accounts to Organizations (Many to Many)

An Organization MUST HAVE one or more member User Accounts

A User Account MAY BE a member of one or more Organizations

Storage to Organizations (Many to Many)

An Organization MUST HAVE one default storage.

A Storage MAY BE allocated to multiple Organizations

Organizations MAY HAVE more than one storage.

Organization User Privileges

Outside the privileges described here, there are no privileges or access required by users.  Users WILL BE able to access data using the existing ACL and AR methods, irrespective of Organizations. 

Organization Administrators

Organization Administrators have privileges spanning all Organizations.  They can administer the Organization system. All system Administrators SHOULD BE Organization Administrators

Organization Administrators MAY perform the following tasks:

• Create, Update or Delete Organizations

• Edit Properties (including Limits) of an Organization 

• Set values of properties (including override of default values)

• Assign users as Managers and Members of the organization

• All of the functions Organization Managers can perform

Organization Managers

Organization managers (those with Manage privileges on the Organization) MAY perform the following tasks on the Organization they have management privileges on

• Add or remove Users, Teams as Organization Members

• Rename the Organization 

• All of the functions Organization Members can perform

Organization Members

Organization Members are granted Create Project within the organization, equivalent to that of users within Synapse pre-organization. 

Users MUST BE Organization Members in order to create projects within an Organization. Once they create the project, behavior within the project is the same as pre-organization Synapse. 

Use Cases

Default Organization Creation (Basic Plan)

Organizations are created by the following actions:

• When a Certified User without a default Organization creates their first project , this creates a Basic Plan and a Default Organization - The Organization should have a default name of the username & ”org” or username & ”Organization”

• A user with a default organization is an Organization Member, not an Organization Manager. 

Organization Creation by Admin

• An Organization Admin creates an Organization from the Plan Definition

Once a plan is created, an Organization Admin can create the organization, assign the Organization Managers according to the Plan Specification. This includes legacy plans. 

• The Plan System creates the Organization automatically based on the Plan Definition.  (FUTURE)

Renaming an Organization

An Organization can be renamed by an Organization Manager or Organization Administrator 

Adding Users, Teams to an Organization

Users and Teams can be added to an existing Organization by a user with Organization Management privileges. Being added to an Organization makes the users or users in the team Organization Members. 

(note: Default Organizations have no Organization Manager, and cannot add additional members)

Organization Privileges do not overwrite Project or entity privileges.  For example, an Organization Member may be granted access to a project (such as a private project). This user will have access to the project, but WILL NOT have access to other projects within the Organization without further action. 

Removing Users, Teams from an Organization 

Users and teams can be removed from an Organization by a User with Organization Management privilege. 

Users MAY remove themselves from an Organization, unless it is their Default Organization.

Teams may be removed from an Organization by user with Organization Management privilege, or by a Team Owner. 

Creating Projects in an “Organized” Synapse

Once Organizations are implemented in Synapse, existing users MUST choose the Organization the project belongs to at creation time.  Failing to indicate an Organization MUST result in an error and the project is not created. 

A user creating a project MUST BE an Organization Member to add a project to that organization or add data to an existing project.

Moving Data Between Organizations

Moving data between organizations is accomplished just as a user would do today. Users MUST HAVE permissions to access the data at the source project and target project. 

Managing Storage in an “Organized” Synapse

Organizations MUST HAVE a default storage selected when the organization is created. 

• Two or more organizations MAY NOT utilize the same private storage.

• With the exception of Default Organizations, two or more organizations MAY NOT utilize the same shared storage.

• Two or more organizations MAY utilize the same custom storage.

Projects MAY choose separate storage from their organization, but must follow these rules:

A project MUST NOT reference another organization’s private storage

A project MAY reference another organization’s custom storage (dependent on access privileges)

Error Cases

Setup a project in an organization using non-organization storage

Implementing Plan Limits with Organizations in Synapse

Organizations utilize the following behaviors when implementing Plan limits.

Storage Limits

Calculating Storage Used

Storage limits are calculated against any content in shared or private storage.  Content stored in Custom Storage is not counted against the limit. For example, take a simple organization with a Self-Managed plan (100 GB limit) with 3 projects, (Projects A, B, & C)  Project A (30GB) has its storage in shared storage.  Project B (40GB) has its storage in private Storage.  Project C (700GB) has its storage in custom storage.

The Organization has a total storage of 770GB with a Storage Limit of 100GB.

The Organization is 30GB under its limit.

We calculate the total stored in shared and private storage 

Project A: 30GB + Project B: 40GB = 70GB

Reaching Storage Limit 

When an organization reaches its Storage Limit, Members are prevented from uploading additional content. 

Should the Organization need to utilize more than 100% of the plan’s storage limit, the Organization Manager MAY choose to acquire additional storage limits (TBD).

Once the storage limit is reached, Organization Members MAY perform the following 

• Delete data 

• Migrate one or more project to custom storage

• Add new projects with Custom Storage

Storage Limit Notifications

When an organization is at > = NN% of the Storage Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.  

Egress Limits

Calculating Egress Used 

Egress limits are calculated against any content in shared or private storage (except that stored in AWS Open Data).  Content transferred from Custom Storage is not counted against the limit.

Egress limits are defined by TB/Year of egress, starting with start date of the plan.

Egress MUST BE calculated for all organizations in order to report usage stats to users and stakeholders.

Synapse will ignore egress limits from the following:

• Custom storage

• Storage within AWS Open Data

• Any future private or shared storage that designated to be “unlimited”

Reaching Egress Limit

When an organization reaches its Egress Limit, users are prevented from transferring data from shared or private storage.  Users are still able to shift data within the AWS region the shared or private storage.

NOTE: Content stored in the AWS Open Data bucket is not counted against the Egress Limit 

Should the Egress Limit be reached, Organization Members MAY perform the following to enable egress:

• Migrate one or more projects to Custom Storage

• Wait for the Egress limit to reset.

Egress Limit Notifications

When an organization is at > = NN% of the egress Limit, the Organization Managers MUST BE notified via Email and via a UI indicator in SWC with directions (or links to directions in help) to avoid hitting the limit.  

Users MUST get a notification in the Download cart when the contents of their cart will exceed the Egress Limit with directions on how to avoid the limit.

User Limit Notification

When a downloader attempts to download, Synapse will be required to check the Egress limit. The user should receive an error that the download cannot proceed.

Organization Managers should be notified that users cannot further download.

Egress Yearly Reset

On the yearly mark (annually from the start date of the active plan), the egress limit will reset (as long as the plan is still active) with no rollover of unused credits.

Updating Limits

Organization Admins can update the limits of an organization:

Notifications

Migrating Existing Data to an Organized Synapse

Data and content exist in pre-organization Synapse today, this “Legacy “ content MUST BE allocated to appropriate new Organizations.  This will require some data analysis.

Open Questions

Please put in any open questions you have and I will address - Kevin

FAQs

Can I have a user without an Organization?

Yes, creating a user does not require an organization (many users may be download only).

Can I move a project from one org to another?

Yes, if you have appropriate access to the source project and are a member of the target organization.

Can a user be a member of multiple Organizations?

Yes, so can a Team.

Does this mean I have to be a member of an Organization to access data stored within?

No, ARs and ACLs remain on Projects, entities, folders, tables, etc. 

Can I create a project without an Organization?

No.

Do I need a plan to create an Organization?

Every Certified User gets a default Organization with a Basic Plan.  Beyond this plan, yes, users will need a plan to have an Organization.