Optus has revealed that a ‘routine software upgrade’ triggered the outage that crippled the nation last week, claiming they have taken steps to ensure it won’t happen again.
The meltdown affected up to 10 million Australians and over 400,000 business who were left in the dark for up to 12 hours after their internet and phone services were cut off.
Optus told its disgruntled customers on Monday afternoon it had been trying to discover what went wrong and insisted they had ‘taken steps to ensure it will not happen again’.
‘We apologise sincerely for letting our customers down and the inconvenience it caused,’ the statement said.
‘At around 4.05am Wednesday morning, the Optus network received changes to routing information from an international peering network following a routine software upgrade,’ the statement explained.
‘These routing information changes propagated through multiple layers in our network and exceeded preset safety levels on key routers which could not handle these.’
The statement said the action caused many routers to automatically disconnect from the Optus IP Core network to protect themselves.
This resulted in a large-scale logistical effort to reconnect or reboot the routers physically, requiring ‘the dispatch of people across a number of sites in Australia’.
The embattled telco said this was the reason why some people were able to connect earlier than others last Wednesday.
‘Given the widespread impact of the outage. investigations into the issue took longer than we would have liked as we examined several different paths to restoration,’ the statement said.
The blackout prevented millions from making and receiving calls and completing transactions, with CEO Kelly Bayer Rosmarin not publicly appearing until several hours into the drama, leaving the communications minister to front cameras.
Thousands of frustrated Optus customers flocked to stores across the country, demanding answers from besieged staff members who have been forced to draft in extra security.
A staff member at an Optus branch on George Street in Sydney’s CBD told Daily Mail Australia that police had to be called twice to calm the furious mob.
‘Angry boomers were just flooding in – I couldn’t count them there were so many,’ he said.
‘The police were here for half an hour: a couple of paddy wagons and a couple of cruisers because there were so many people involved.’
Daily Mail Australia revealed that on the day of the crisis, Ms Rosmarin’s $15miilion Sydney home was the scene of an elaborate photoshoot.
Customers were offered an extra 200GB of data for their ‘patience and loyalty’.
But Federal Labor minister Bill Shorten said on Friday the extra data wouldn’t ‘touch the sides’ of customer frustration and encouraged small businesses to speak with their account managers.
‘The telecommunications industry ombudsman can assist small businesses who are dissatisfied with the responses, I would encourage those customers to keep records, to document the impacts of the outage on them, but it was a nightmare for everyone,’ he told Sydney radio 2GB.
It is the second debacle for the company in a little over 12 months after a huge hack in September 2022 saw the information of 10million customers compromised – more than a third of the Australian population.