/
Bridge on Google Cloud

Bridge on Google Cloud

May 31, 2019

Overview

Phase 1: Bridge Export to Google Cloud Instead of Synapse (18 story points, 2.6 dev weeks)

Phase 2: Bridge Running on Google Cloud Instead of AWS (38 story points, 7.6 dev weeks)

Phase 3: Bridge Stores Data in Google Cloud Instead of AWS (Stretch Goal) (69 story points, 13.8 dev weeks)

Estimates assume a velocity of 5 story points per dev week.

Phase 1: Bridge Export to Google Cloud Instead of Synapse

Subtasks:

  • Refactor SynapseHelper to be a generic Export Adapter with pluggable implementations for Synapse and Google Cloud (5 story points)

  • Define Google Cloud storage format (1 story points)

  • Define Google Cloud permissions model (1 story points)

  • Implement Google Cloud Export Adapter (5 story points)

  • Define and implement Bridge Study config to select between Synapse and Google Cloud (3 story points)

  • Implement selector in Exporter to select between Synapse and Google Cloud (3 story points)

Total: 18 story points (2.6 dev weeks)

Phase 2: Bridge Running on Google Cloud Instead of AWS

Total: 38 story points (7.6 dev weeks)

Phase 2A: Bridge Server on Google Cloud

Subtasks:

  • Launch Spring MVC Service on Google Cloud (8 story points)

  • Set up DNS and SSL (2 story points)

  • Set up secure credentials (1 story point)

  • Log aggregation (1 story point)

  • Dashboards, monitoring, and alarms (1 story point)

  • Grant access to AWS Elasticache and RDS from Google Cloud OR partition and create a Redis and SQL instance in Google Cloud (3 story points)

Total: 16 story points (3.2 dev weeks)

Phase 2B: Bridge Exporter on Google Cloud

Subtasks:

  • Launch Spring Boot Worker on Google Cloud (8 story points)

  • Set up secure credentials (1 story point)

  • Log aggregation (1 story point)

  • Dashboards, monitoring, and alarms (1 story point)

Total: 11 story points (2.2 dev weeks)

Phase 2C: Bridge Worker on Google Cloud

Subtasks:

  • Launch Spring Boot Worker on Google Cloud (8 story points)

  • Set up secure credentials (1 story point)

  • Log aggregation (1 story point)

  • Dashboards, monitoring, and alarms (1 story point)

Total: 11 story points (2.2 dev weeks)

Phase 3: Bridge Stores Data in Google Cloud Instead of AWS (Stretch Goal)

Relevant docs:

AWS Services that need to be replaced:

  • DynamoDB → BigTable? Firestore? (53 story points)

  • Lambda (Cron scheduler) → Cloud Scheduler (3 story points)

  • RDS (MySQL) → Cloud SQL (1 story point)

  • S3 → Cloud Storage (5 story points)

  • S3 Presigned URLs → Cloud Storage Signed URLs (3 story points)

  • S3 Public Webhosting → Cloud Storage? (1 story points)

  • SES (Simple Email Service) → NO EQUIVALENT

  • SNS Push Notifications → Firebase Cloud Message MAY REQUIRE CLIENT APP CHANGES

  • SNS SMS → NO EQUIVALENT

  • SQS → PubSub (3 story points)

Total: 69 story points (13.8 dev weeks)

DynamoDB Migration

Subtasks (53 story points)

  • Bridge Server (40 story points)

    • 25 object/dao pairs (+ 2 survey element classes) (30 story points)

    • DynamoInitializer or equivalent (3 story points)

    • 9 data type marshallers (5 story points)

    • 3 backfill classes, to be ported or removed (2 story points)

  • Exporter accesses the following tables directly (5 story points)

    • ExportTime

    • HealthDataAttachment

    • HealthDataRecord

    • Study

    • SynapseMetaTables

    • SynapseTables

  • Worker accesses the following tables directly (5 story points)

    • FitBitTables

    • NotificationConfig

    • NotificationLog

    • Study

    • SynapseSurveyTables

    • SynapseTables

    • UploadSchema

    • WorkerLog

  • Master Scheduler (3 story points)

    • Config

    • Status

Note: Spring Migration required us to migrate ~56 classes, which took about 3 weeks (15 days), which suggests a velocity of ~3.75 classes per day. However, these were mostly trivial copy-pastes with minimal transformation. Re-writing all the DynamoDB classes would be signficantly more work. We'll base our estimates around 1.8 classes per day.