Disruption to departmental and module mailing lists

Incident Report for Prifysgol Aberystwyth University

Postmortem

Summary

During routine system maintenance, the MySQL database that stores our mailing list data was shut down and upgraded. At the same time, the automated process responsible for updating the membership of each mailing list was still running. Because the database became unavailable during this process, the resulting data became inconsistent.

To ensure accuracy and reliability, the entire mailing lists database was rebuilt using our automated scripts.

What Happened

  • Scheduled maintenance required shutting down and upgrading the MySQL database.
  • The process that updates mailing list membership was running at the time.
  • With the database unavailable mid‑update, the data was left in an inconsistent state.
  • Once identified, a full database rebuild was initiated to restore correct membership information.

Resolution

  • A complete rebuild of the mailing lists database was carried out.
  • Post‑rebuild checks confirmed that all mailing lists and their memberships were accurate and up to date.
  • No further issues were detected after validation.

Next Steps

We will review the coordination between maintenance activities and background processes to reduce the risk of similar inconsistencies during future maintenance periods.

Posted Mar 26, 2026 - 07:30 GMT

Resolved

This incident has been resolved.
Posted Mar 04, 2026 - 10:46 GMT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Feb 26, 2026 - 19:44 GMT

Identified

We have confirmed that the issue only affects users being added to or removed from automatically generated module and departmental mailing lists. This issue should only affect a very small number of users while we work to implement a solution.
Posted Feb 26, 2026 - 14:13 GMT

Investigating

We are investigating an issue affecting email lists. Updates will be posted as soon as they are available.
Posted Feb 26, 2026 - 12:09 GMT
This incident affected: Microsoft 365.