Page Comparison

Jira Epic: 

...

1. Create a metadata File record;
2. Create a FileRevision record, which returns a pre-signed URL to upload a file;
3. Upload that file;
4. S3 upload even triggers lambda (probably) that calls Bridge as a worker and finishes the revision, marking it as availableputs a work item on a queue for the BridgeWorkerPlatform to mark the item as available (see error handling below);

Presigned URLs can be used more than once. If this happens, we should update the revision record.

When retrieving a revision record, if it is marked pending and the presigned URL is expired, we should update it with a new presigned URL and return that. Basically a pending revision record will always give a way to upload a file, until it's deleted. So the expiration period on this presigned URL can be very long. Once the revision record has been marked available, it can't be deleted.

If the worker process fails to mark a pending record as available, we can check the status of the file when an individual revision is requested through the API (GET /v3/files/<guid>/revisions/<createdOn>) by looking for the file and if it exists, marking the record as available. Until we see how often this fails, this may be sufficient for admin users to clean up orphaned records (they can also delete them and try over).

Changelog

CREATE TABLE IF NOT EXISTS `Files` (
    `studyId` varchar(255) NOT NULL,
    `guid` varchar(60) NOT NULL,
    `name` varchar(255) DEFAULT NULL,
    `description` text,
    `mimeType` varchar(255) DEFAULT NULL,
    `deleted` tinyint(1) NOT NULL DEFAULT '0',
    `version` int(10) unsigned NOT NULL, // optimistic lock
    PRIMARY KEY (`guid`),
    KEY `Studies_idx` (`studyId`), // retrieve all for a study
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;


CREATE TABLE IF NOT EXISTS `FileRevisions` (
    `fileGuid` varchar(60) NOT NULL, 
    `createdOn` BIGINT UNSIGNED NOT NULL,
    `description` text,
    `uploadURL` VARCHAR(512) DEFAULT NULL,
    `status` ENUM('PENDING','AVAILABLE') NOT NULL
    PRIMARY KEY (`fileGuid`, `createdOn`),
    CONSTRAINT `Files-Guid-Constraint` FOREIGN KEY (`fileGuid`) REFERENCES `Files` (`guid`) ON DELETE CASCADE
) CHARACTER SET utf8 COLLATE utf8_unicode_ci;

...

publicinterface File {

    String getName();

    void setName(String name);

    String getGuid();

    void setGuid(String guid);

    String getDescription();

    void setDescription(String description);

    /** Probably optional since S3 autodetects, but some files (e.g. without extensions) may need it. */

    String getMimeType();

    void setMimeType(String mimeType);

    /** Files support logical deletion. Revisions remain accessible on S3 for released clients and configurations. */

    boolean isDeleted();

    void setDeleted(boolean deleted);

}

...

public interface FileRevision {

    String getFileGuid();

    void setFileGuid(String fileGuid);

    /** To handle upload fails and upload URL expiration, we use the creation time of this record, not any timestamp of the S3 file itself. */

    DateTime getCreatedOn();

    void setCreatedOn(DateTime createdOn);

    String getDescription();

    void setDescription(String description);

    /** This is the presigned S3 URL to upload contents. Will be cleared when revision is moved to "available." */

    String getUploadURL();

    void setUploadURL(String uploadURL);

    FileStatus getFileStatus();

    void setFileStatus(FileStatus status);

    /** Synthetically derived from other data as: http://docs.sagebridge.org/<studyId>/<guid>/<createdOn> */

    String getDownloadURL();

    void setDownloadURL(String downloadURL);

}

...

public enum FileStatus {

    PENDING,

    AVAILABLE

...