Flexpa - Notice history

This incident has been resolved. Post mortem summary:

What happened?
api.flexpa.com was unavailable and 503s were returning from the load balancer.

Root cause analysis
We tried to deploy a Task Definition that was pointing to secrets that were not in the parameter store, because those secrets were removed after the deploy was initiated.

Action items to prevent its reoccurrence
Ensure that removing a secret from Doppler should regenerate a task definition and redeploy.

This incident has been resolved. We believe this was an incorrect alarm.

This incident has been resolved.

We discovered an uncaught error in responses to our /fhir endpoint and implemented a fix.

We have resolved the issue with our API.

We have resolved the issue with our API.

Post Incident Summary

Availability of Flexpa is critically important to our customers. As part of our normal incident response, we have conducted a post-incident summary as to the source of this incident.

On June, we deployed new logging mechanisms in order to more effectively troubleshoot customer issues. The deployment was completed successfully and internal testing showed systems were operationally normal.

On June 6, Flexpa experienced three brief API outages cumulatively lasting 6 minutes. During these outages, Flexpa was completely unavailable as services restarted

At 6:31 PM EST, it was observed that the MyFlexpa application was behaving erratically.

At 7:22 PM EST, a crash of Flexpa applications was observed and noted internally.

At 7:23 PM EST, Flexpa's applications had automatically restarted and recovered.

At 8:05 PM EST, a root cause had been identified and a fix was deployed to update logging to prevent future crashes. We have been operationally normal since this time.

At 8:26 PM EST, an update was applied to ensure future alerting and logging was routed to the appropriate location.

Moving forward from this incident we are:

• Improving testing environment for logging updates to detect issues in advance of production deployment
• Continue investigation into logging libraries Flexpa depends on in order to understand failure more deeply and prevent similar issues in the future.
• Re-affirming our commitment to operational excellence

We have resolved the issue with our API.

We have resolved the issue with our API.

We have resolved the issue with our API.

We have resolved the issue with our API.