CAUSES OF THE OUTAGES
Investigations into the incidents showed that the main leads to were failures associated with hardware devices within data centres, stated Dr Puthucheary.
In the times leading up to the Aug 27 incident, two nodes – hardware devices which run in tandem — in firewall areas at data centres failed.
The system usually provides other nodes to handle the load of data traffic to continue service operations if one fails. On August 27, as technical engineers tried to restore the 2 failed nodes, the operation failed. This caused the cluster of firewall nodes to malfunction, which subsequently caused the outage.
Dr Puthucheary said that the engineers worked well to reset the particular systems to the before state without the perform of the two impacted nodes and company was progressively restored.
“The failure of the nodes was caused by bugs in the firmware of the devices. They have since been identified by the manufacturer, CISCO, and the devices have been patched, ” he added.
The Sep 5 incident had been caused by “the simultaneous failure of 2 further nodes, once again from the same manufacturer, and of the same model”, noted Dr Puthucheary.
The way the failure occurred has been different from the previous incident and more time was needed to restore procedures. The cause of this failure is still under investigation.
“There was a suggestion in one associated with questions from associates that the failures may be due to the lack of manpower of IHiS. IHiS has a headcount of 3, 500 workers, they have a lot to accomplish and will always encouraged more manpower, but a lack of manpower is not the cause of these problems, ” said the particular Senior Minister associated with State.
He added there were no indications of security give up to the affected systems based on investigations.
In response to Doctor Tan’s question upon whether systems had been benchmarked against “best-in-class” systems elsewhere, Doctor Puthucheary said: “There are service level agreements about the uptime availability as well as the kind of user interface usability for your products that IHiS manages, and these are usually indeed benchmarked against best in class around the world. ”
MP He Ting Ru (WP-Sengkang GRC) asked what help was given to affected frontline staff, and if they were provided with training to deal with such conditions.
In reply, Dr Puthucheary declared that support from the Ministry of Health and IHiS centred on “communication and providing obvious information about what has happened, what are the anticipated steps taken to regain functionality, (and) how much time will be required”.
The healthcare institutions also mobilised staff to deal with the outages and the additional processes involved, mentioned Dr Puthucheary.
“Is schooling provided? Yes, business continuity plans… catastrophe recovery plans are drilled regularly in the healthcare units and teams are component of standard training for all of the healthcare workers that are in our public medical care sector. And the instruction is updated regularly.
“This is not something that is standard in every team and in every unit across the healthcare ecosystem because are peculiar towards the operations and moves within each clinical team. ”