Incorrect content returned on menu pages
Incident Report for BentoBox
Postmortem

At approximately 5:21 PM EST on Friday May 20th, 2022 our customer success reported that menu pages on certain sites were serving content from the wrong website. The url /menus/ on certain BentoBox websites was incorrectly loading the content directly from a single BentoBox website. Our support team immediately received a large amount of support tickets from concerned customers. Our incident response process was triggered starting at 5:30pm and our production support engineering team was notified via PagerDuty. Investigation began immediately. The root cause appeared to be related to site caching but it was unclear whether it was due to something on the BentoBox application or our edge caching provider. Earlier that day, due to an expected flood of traffic on a single site, a caching feature had been enabled on a single site in order to better handle additional load. Our production support team immediately started work to disable the caching feature for this website. At approximately 6pm, using our feature flag functionality, a caching layer was turned off which appeared to resolve the issue for most affected customers. Remediation worked was paused. Shortly thereafter more reports started coming in, including more affected urls (menus, locations, about). Remediation worked was resumed. At 6:25pm the caching feature was fully turned off and the cache cleared with our content distribution network provider. At 6:30pm the issue was declared resolved and no further issues were detected.

Posted May 24, 2022 - 09:55 EDT

Resolved
The /menus/ url on BentoBox customer websites is returning content from a single, incorrect, BentoBox customer.
Posted May 20, 2022 - 17:30 EDT