Flexpa - Notice history

This incident has been resolved.

Why did it happen?
The new parameter in Flexpa Link was checked if it was defined, and so the word "undefined" was being serialized into the Flexpa Link application state, and the application was crashing on this unexpected value.

How do we prevent it in the future?

• Include Flexpa Link (package) in our CICD test suites.
• Implement a UX Monitoring alarm on MyFlexpa to test that Link is loading successful

We have resolved the issue with our API.

The database cache our apps connect to was destroyed in the course of a configuration change today.

This prevented a service from being online.

That service being offline meant that every new Patient Access Token transitioned to an error state. An error state was shown visibly to patients / our customers and Links do not occur successfully.

This impacted all customers (excepting those opted out from our data cache) because every new Patient Access Token requires this step to successfully complete.

We completed resolution of this event once we had all applications pointed at the new data cache.

Flexpa takes downtime such as this event seriously. We have completed a post-mortem procedure and will be introducing changes to our process to prevent this issue from re-occurring.

This incident has been resolved. Post mortem summary:

What happened?
api.flexpa.com was unavailable and 503s were returning from the load balancer.

Root cause analysis
We tried to deploy a Task Definition that was pointing to secrets that were not in the parameter store, because those secrets were removed after the deploy was initiated.

Action items to prevent its reoccurrence
Ensure that removing a secret from Doppler should regenerate a task definition and redeploy.