Trash Can

Main Features

1. A new API to delete an entity into a trash can. Once an entity is in the trash can, it becomes invisible from the normal CRUD operations. The current delete API will be kept as an alternative to permanently delete an entity.
2. Each user has a dedicated trash can. The user can view deleted entities in his/her trash can.  (Note: The current trash folder approach is in fact a global trash can.)
3. The user can restore deleted entities in the trash can.
4. The user can purge the whole trash can or purge individual entities.
5. A worker periodically scans trash cans and purges entities that are in the trash can for more than a month.

Challenges

1. Need to define a reasonable boundary as to what data should go to the trash can so that it can be restored as completely as possible. The boundary obviously goes beyond the entity/node. We need to also trash the revisions (which has the annotations, references, version numbers), the ACLs, together with entity. What else? When deciding what data to trash, also keep in mind the impact on maintenance and future development.
2. Need to handle hierarchies. Especially each entity has a parent and a benefactor. Deleting a node has the cascading effect of also deleting the descendants and the beneficiaries. We must also trash the dependents so that they can be restored together the deleting node. This requirement can hit the performance badly if we end up deleting a large tree of nodes. One approach to this challenge is to set a limit to the number of entities to move to the trash can. If we count the number of entities to be more than 100, for example, we throw an exception ("Too large to fit into the trash can.") and prompt the user to permanently delete the entities instead.
3. Need to cope with changes. Once the entity is in trash can, it is frozen from changes. In the meantime, the data in Synapse keeps changing. When we restore an entity from the trash can, its surroundings may have already changed. For example, its parent does not exist any more, or the access requirement has changed. Not only data. Schemas change too. Besides, the schemas are not standardized, not versioned, not persisted, and not very well detached from the code logic. That said, there exists the risk that the items you put in a trash can today may fail to restore a month later due to incompatible schemas. We should at least be able to detect such conflicts and fail the restore with exceptions. Or perhaps better, cut off the conflicting parts and let the user restore manually.

Proposed Solutions

The boolean flag approach

Add a boolean column IS_DELETED. Data is not deleted but is flagged as "deleted". Relatively simple to implement trash can using this strategy. But it requires setting up filtering on all the non-trash-can queries.

The backup approach

Delete by backing up an entity to a different place. As the trash data is backed up in a different place, all the existing, non-trashcan queries are not affected.

One approach is to move deleted data to separate tables of the same schema. For example, move an entry in the table REVISION to REVISION_TRASH of the same schema.

Another perhaps easier approach is to use the exiting backup mechanisms. The main advantage of this approach is that challenge 1 is already taken care of – necessary backup data are already wrapped into NodeBackup and NodeRevisionBackup. NodeBackupManager.getNode() and NodeBackupManager.getNodeRevision() create the backup objects. NodeBackupDriver.writeBackup() serializes the backup objects to XML files. It also recursively write the whole tree. However, a limit need to be inserted here for the trash can feature. Backup/retore can afford running for hours while moving entities into a trash can must finish quickly. The trash can in this approach use Amazon S3 to store the trash data.

The folder approach

Move the entities being deleted to a trash folder. Every entity within the trash folder will have its benefactor set to the trash folder. If we only allow administrators to access the trash folder, it essentially hides the trash can from normal users. We then use an additional table to track individual trash entities (i.e. who deleted what and when) so that a user can view only items deleted by him/her and restore if needed. This approach fits naturally our current design around folders and files. It requires minimal amount of work comparing with the other two approaches. The main disadvantage is the loss of the original ACLs during restore. Once an entity is moved to the trash folder, the ACLs of its descendants are set to inherit that of the trash folder and the original ACLs are lost. When the tree rooted at the entity is restored, the ACL of every node is reset to the new parent.

The Trash Folder Approach in Details

Use case diagram

Sequence diagram

Proposed Rest APIs

For authenticated users:

1. Move an entity to the trash can (PLFM-1688)
2. View entities in the trash can (PLFM-1688)
3. Restore an entity in the trash can (PLFM-1688)
4. Purge an entity in the trash can (PLFM-1700)
5. Purge the trash can (PLFM-1700)

For administrators:

1. View entities in the trash can (PLFM-1698)
2. Restore an entity in the trash can (PLFM-1698)
3. Purge an entity in the trash can (PLFM-1698)

For daemon workers:

1. Purge entities that have been in the trash can for more than 1 month (PLFM-1699)