Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Current »

Overview

Phase 1: Bridge Export to Google Cloud Instead of Synapse

Phase 2: Bridge Running on Google Cloud Instead of AWS

Phase 3: Bridge Stores Data in Google Cloud Instead of AWS (Stretch Goal)

Phase 1: Bridge Export to Google Cloud Instead of Synapse

Subtasks:

  • Refactor SynapseHelper to be a generic Export Adapter with pluggable implementations for Synapse and Google Cloud
  • Define Google Cloud storage format
  • Define Google Cloud permissions model
  • Implement Google Cloud Export Adapter
  • Define and implement Bridge Study config to select between Synapse and Google Cloud
  • Implement selector in Exporter to select between Synapse and Google Cloud

Phase 2: Bridge Running on Google Cloud Instead of AWS

Phase 2A: Bridge Server on Google Cloud

Subtasks:

  • Launch Spring MVC Service on Google Cloud
  • Set up DNS and SSL
  • Set up secure credentials
  • Log aggregation
  • Dashboards, monitoring, and alarms
  • Grant access to AWS Elasticache and RDS from Google Cloud OR partition and create a Redis and SQL instance in Google Cloud

Phase 2B: Bridge Exporter on Google Cloud

Subtasks:

  • Launch Spring Boot Worker on Google Cloud
  • Set up secure credentials
  • Log aggregation
  • Dashboards, monitoring, and alarms

Phase 2C: Bridge Worker on Google Cloud

Subtasks:

  • Launch Spring Boot Worker on Google Cloud
  • Set up secure credentials
  • Log aggregation
  • Dashboards, monitoring, and alarms

Phase 3: Bridge Stores Data in Google Cloud Instead of AWS (Stretch Goal)

Relevant docs:

AWS Services that need to be replaced:

  • DynamoDB → BigTable? Firestore?
  • Lambda (Cron scheduler) → Cloud Scheduler
  • RDS (MySQL) → Cloud SQL
  • S3 → Cloud Storage
  • S3 Presigned URLs → Cloud Storage Signed URLs
  • S3 Public Webhosting → Cloud Storage?
  • SES (Simple Email Service) → NO EQUIVALENT
  • SNS Push Notifications → Firebase Cloud Message MAY REQUIRE CLIENT APP CHANGES
  • SNS SMS → NO EQUIVALENT
  • SQS → PubSub

DynamoDB Migration

Subtasks

  • Bridge Server
    • 25 object/dao pairs (+ 2 survey element classes) (30 story points)
    • DynamoInitializer or equivalent TODO
    • 9 data type marshallers
    • 3 backfill classes, to be ported or removed
  • Exporter accesses the following tables directly
    • ExportTime
    • HealthDataAttachment
    • HealthDataRecord
    • Study
    • SynapseMetaTables
    • SynapseTables
  • Worker accesses the following tables directly
    • FitBitTables
    • NotificationConfig
    • NotificationLog
    • Study
    • SynapseSurveyTables
    • SynapseTables
    • UploadSchema
    • WorkerLog
  • Master Scheduler
    • Config
    • Status

Note: Spring Migration required us to migrate ~56 classes, which took about 3 weeks (15 days), which suggests a velocity of ~3.75 classes per day. However, these were mostly trivial copy-pastes with minimal transformation. Re-writing all the DynamoDB classes would be signficantly more work. We'll base our estimates around 1.8 classes per day.

  • No labels