Information Technology Services

Network outage across multiple campuses

Summary

Network outage reported across: SBY, MSP, NHM and SYD

Start Date

25th Feb 2017 10:30am

End Date

2nd Mar 2017 10:00am

Status

Resolved

Type

Unplanned Outage

Location

All Campuses

We have received many reports of network access outages from within multiple UTAS campuses. This is affecting Sandy Bay, Medical Sciences Precinct, Newnham and Sydney campuses. No external internet or local network resources are available from these campuses. Many Utas services appear to remain available off-campus.
The ITS Networks team is attending to this as a high priority.

UPDATE 11:45am Saturday 25-Feb-2017: Network services have returned to some locations in Sandy Bay, the Networks team continue to work on this issue.

UPDATE 12:40pm Saturday 25-Feb-2017: Network services are again offline at Sandy Bay and other campuses, the Networks team continue to work on this issue. Access to VPN services is also offline. Access to UTAS staff email via owa.utas.edu.au is available.

UPDATE 11:30am Sunday 26-Feb-2017: Firewall related issues have been rectified, however problems with network access persist intermittently with the UTAS website also temporarily inaccessible. The Networks team continue to work on this issue.

UPDATE 1:00pm Sunday 26-Feb-2017: Network services, VPN and webmail are currently available. The underlying issue is still being investigated.

UPDATE 9:15am Monday 27-Feb-2017: Between approximately 8:30am and 9:00am we again experienced a major network outage. The underlying Firewall issue appears to be ongoing despite major changes and upgrades over the weekend with the Firewall Vendor (Palo Alto). The underlying issue remains under urgent investigation with the Vendor as our highest priority.

UPDATE 4:05pm Monday 27-Feb-2017: A further short major network outage took place from approximately 3:50pm to 4:00pm. The cause relates to short massive loads on our external facing firewall infrastructure. Firewall rectification work is ongoing with possible major changes to take place after business hours tonight.

UPDATE 5:20pm Monday 27-Feb-2017: A further short major network outage took place from approximately 5:00pm to 5:10pm. The Firewall Vendor Palo Alto is assisting ITS with our rectification.

UPDATE 6:10pm Monday 27-Feb-2017: The Palo Alto Firewall appliances have been successfully updated as directed by the Vendor. The Network team are actively monitoring performance to ascertain if problems persist.

UPDATE 8:15am Tuesday 28-Feb-2017: The vendor was able to provide new software yesterday evening. That update was applied, but unfortunately it did not mitigate the fault as a further brief interruption was experienced at approximately 8:30pm for sub-five minutes, with four additional brief interruptions occurring between that time and 6:45am this morning. The network team are continuing to work with the Vendor.

UPDATE 1:50pm Tuesday 28-Feb-2017: Further investigations as to the root cause are ongoing by 3 Senior Vendor Engineers working with the UTAS Networks team. The Vendor has made available a spare appliance, the existing units have been running smoothly since early this morning and are being actively monitored.

UPDATE 3:40pm Tuesday 28-Feb-2017: IT Services staff have been focused on diagnosing the cause and putting in place a permanent fix. After significant diagnostic effort the vendor has confirmed that multiple external distributed denial of service (DDOS) attacks have been the root cause of the intermittent outages. IT Service staff will continue to work with the vendor to implement additional defences to minimise the impact of future attacks.

UPDATE 9:00am Wednesday 1-Mar-2017: IT Services staff have worked with the vendor overnight on further diagnosis and have put in place appropriate changes that minimise the likelihood of the intermittent network issues experienced reoccurring. They will continue to work with the vendor over coming days to implement permanent changes that will ensure the uninterrupted delivery of online services.

UPDATE 10:00am Thursday 2-Mar-2017: Changes put in place on Tuesday night (28-Feb) have returned us to a stable network. Further improvements are being planned to ensure this continues.