Confidential Information, Disclaimer and Trade Marks
This document serves as a Root Cause Analysis for the Primo Central service interruption experienced by Ex Libris customers on October 18,2016.
The goal of this document is to share our findings regarding the event, specify the root cause analysis, outline actions to be taken to solve the downtime event, as well as preventive measures Ex Libris is taking to avoid similar cases in future.
Service interruption was experienced by Ex Libris customers served by the Primo Central instance at the North America Data Center during the following hours:
October 18, 2016 from 8:54 AM until 9:30 AM Chicago time zone
During the event, the service was unavailable for the environment.
Root Cause Analysis
Ex Libris Engineers investigated this event to determine the root cause analysis with the following results:
An ongoing repeating process had been identified to cause an abnormal load on the system, causing a complete system down.
The process is running on an ongoing basis and its purpose is rebalancing of the information between the multiple database components.
We were able to identify that the job is causing a load on the system, that in specific situations may cause a complete service disruption.
Technical Action Items and Preventive Measures
Ex Libris has taken the following action and preventive measures to avoid such an occurrence in future:
- Once Identified, the process had been stopped and the system returned to it's normal behavior.
- Following the system disruption, the ongoing process had been analyzed and had been updated to run in a manner that will have no impact on the system.
ExLibris is committed to providing customers with prompt and ongoing updates during Cloud events. Ongoing and prompt updates on service interruptions appear in the system status portal at this address: http://status.exlibrisgroup.com/