...
How to handle prod vs staging? Isn’t there the issue that an email could be sent by staging after a first migration if some notification were not processed already (We probably have this problem already?): Below are some solutions discussed with the engineering team, for the first iteration of the implementation solution 5. below is the easiest and more flexible to implement.
A solution that comes to mind is that we implement a service to disable certain features, similar to the read-only service (could be part of the same API call) and when go live we reenable them. The flow might be: We prepare the staging stack → put in read-only mode + disable the email notifications (e.g. the worker that sends emails asynchronously) → we perform migration → we put the system back in read-write mode → testing → read-only mode → final migration → re-enable all features → release. Note however that this would disable all the email notifications that are processed asynchronously by a worker (e.g. messages to users).
An alternative might be to configure staging to redirect all the messages processed by the worker to a single recipient that we control (this way we can actually check the emails that are sent without spamming users)?
Alternative solutions proposed during the design review:The staging stack could detect that it’s “staging” and make decisions based on that: the drawback is that this introduces (as the solutions proposed above) different behavior between prod/staging.
A service that is separate from the stack and deployed separately could be created to process the emails to be sent, the service could interrogate a given stack only (e.g. prod only) and ask for the emails to be sent. The same service could be deployed for staging but configured differently (e.g. instead of sending the emails it could save them to S3). In this way the prod/staging stacks would remain the same there is no need to switch configurations. The drawback is that we need to build additional infrastructure.
Another interesting solution: We can implement a new (administrative) web service that allows to push messages for example on a queue. A timer worker will periodically send a request to this service to trigger another worker (that polls the given queue and is setup as any other MessageDriven worker). The target worker is the one that actually process the scheduled messages to send, so we can basically externalize the trigger of the worker that process the messages that are scheduled. In this way the timer worker can be instructed to always trigger the request to prod and at the same time we can reuse the same web service on staging for testing. This solution has the advantage that it reuses the current infrastructure for timers and workers and it’s a bit more generic and can be reused for other purposes.
Would it be better to store the submission id in the notification table instead of the ar/submitter? It seems that an access approval can be created without a submission (See https://rest-docs.synapse.org/rest/POST/accessApproval.html)? How that ties together with the accessor group?