Document toolboxDocument toolbox

Bridge on Google Cloud

Owned by Sonia Carlson
Jun 21, 2024

Overview

Phase 1: Bridge Export to Google Cloud Instead of Synapse (18 story points, 2.6 dev weeks)

Phase 2: Bridge Running on Google Cloud Instead of AWS (38 story points, 7.6 dev weeks)

Phase 3: Bridge Stores Data in Google Cloud Instead of AWS (Stretch Goal) (69 story points, 13.8 dev weeks)

Estimates assume a velocity of 5 story points per dev week.

Phase 1: Bridge Export to Google Cloud Instead of Synapse

Subtasks:

  • Refactor SynapseHelper to be a generic Export Adapter with pluggable implementations for Synapse and Google Cloud (5 story points)
  • Define Google Cloud storage format (1 story points)
  • Define Google Cloud permissions model (1 story points)
  • Implement Google Cloud Export Adapter (5 story points)
  • Define and implement Bridge Study config to select between Synapse and Google Cloud (3 story points)
  • Implement selector in Exporter to select between Synapse and Google Cloud (3 story points)

Total: 18 story points (2.6 dev weeks)

Phase 2: Bridge Running on Google Cloud Instead of AWS

Total: 38 story points (7.6 dev weeks)

Phase 2A: Bridge Server on Google Cloud

Subtasks:

  • Launch Spring MVC Service on Google Cloud (8 story points)
  • Set up DNS and SSL (2 story points)
  • Set up secure credentials (1 story point)
  • Log aggregation (1 story point)
  • Dashboards, monitoring, and alarms (1 story point)
  • Grant access to AWS Elasticache and RDS from Google Cloud OR partition and create a Redis and SQL instance in Google Cloud (3 story points)

Total: 16 story points (3.2 dev weeks)

Phase 2B: Bridge Exporter on Google Cloud

Subtasks:

  • Launch Spring Boot Worker on Google Cloud (8 story points)
  • Set up secure credentials (1 story point)
  • Log aggregation (1 story point)
  • Dashboards, monitoring, and alarms (1 story point)

Total: 11 story points (2.2 dev weeks)

Phase 2C: Bridge Worker on Google Cloud

Subtasks:

  • Launch Spring Boot Worker on Google Cloud (8 story points)
  • Set up secure credentials (1 story point)
  • Log aggregation (1 story point)
  • Dashboards, monitoring, and alarms (1 story point)

Total: 11 story points (2.2 dev weeks)

Phase 3: Bridge Stores Data in Google Cloud Instead of AWS (Stretch Goal)

Relevant docs:

AWS Services that need to be replaced:

  • DynamoDB → BigTable? Firestore? (53 story points)
  • Lambda (Cron scheduler) → Cloud Scheduler (3 story points)
  • RDS (MySQL) → Cloud SQL (1 story point)
  • S3 → Cloud Storage (5 story points)
  • S3 Presigned URLs → Cloud Storage Signed URLs (3 story points)
  • S3 Public Webhosting → Cloud Storage? (1 story points)
  • SES (Simple Email Service) → NO EQUIVALENT
  • SNS Push Notifications → Firebase Cloud Message MAY REQUIRE CLIENT APP CHANGES
  • SNS SMS → NO EQUIVALENT
  • SQS → PubSub (3 story points)

Total: 69 story points (13.8 dev weeks)

DynamoDB Migration

Subtasks (53 story points)

  • Bridge Server (40 story points)
    • 25 object/dao pairs (+ 2 survey element classes) (30 story points)
    • DynamoInitializer or equivalent (3 story points)
    • 9 data type marshallers (5 story points)
    • 3 backfill classes, to be ported or removed (2 story points)
  • Exporter accesses the following tables directly (5 story points)
    • ExportTime
    • HealthDataAttachment
    • HealthDataRecord
    • Study
    • SynapseMetaTables
    • SynapseTables
  • Worker accesses the following tables directly (5 story points)
    • FitBitTables
    • NotificationConfig
    • NotificationLog
    • Study
    • SynapseSurveyTables
    • SynapseTables
    • UploadSchema
    • WorkerLog
  • Master Scheduler (3 story points)
    • Config
    • Status

Note: Spring Migration required us to migrate ~56 classes, which took about 3 weeks (15 days), which suggests a velocity of ~3.75 classes per day. However, these were mostly trivial copy-pastes with minimal transformation. Re-writing all the DynamoDB classes would be signficantly more work. We'll base our estimates around 1.8 classes per day.