CMS and feeds unavailable
Incident Report for Megaphone
Postmortem

What happened?

On Thursday, September 19 Megaphone experienced degraded performance within the CMS (platform application) and associated URLs from approximately 7:00am EST to 9:25am EST. The outage impacted access to the CMS and associated links. All podcast audio, ad serving, podcast metadata and metrics were fully functional and listeners were not disrupted during the incident.

The incident was caused by an unusual surge in traffic hitting the CMS domains which caused timeouts and rendered the application unavailable.

What was the solution?

We worked to divert some of the traffic temporarily to allow CMS to recover. This quickly reduced error rates and restored full service.

How we are preventing this in the future?

As stated above, the impacted endpoint was moved to higher capacity infrastructure to reduce likelihood of a similar incident in the future. Additionally, we will be doing some enhancements to better distribute unusual loads should they arise.

If you have any questions, please reach out to us via Megaphone Live Chat or email support@megaphone.fm for more information.

Posted Sep 19, 2019 - 17:06 EDT

Resolved
This incident has been fully resolved. We will be posting a post-mortem later today with additional details. Thanks again for your patience this morning!
Posted Sep 19, 2019 - 10:00 EDT
Monitoring
Megaphone is slowly recovering from this incident. Our engineers are continuing to work on it until we are fully recovered. Please keep us posted on any issues you’re encountering by email at support@megaphone.fm, and we will continue to provide updates as they become available. Thanks again for your patience!
Posted Sep 19, 2019 - 09:25 EDT
Investigating
The CMS application (megaphone.fm platform) and RSS feeds are currently unavailable. The team is investigating, and will post an update shortly. You can reach our support team at support@megaphone.fm in the interim. Thank you in advance for your patience.
Posted Sep 19, 2019 - 08:21 EDT
This incident affected: Megaphone CMS and RSS Feeds.