Monitoring - We have seen services restored; and have received more details about the outage.

We have been advised that earlier this morning, Telstra had an issue and they needed to perform an Emergency change to upgrade the software. They anticipated an outage of up to 60 Minutes, and an issue has occurred whilst doing this. While the reboot of the device had been successful in clearing the errors, they have experienced a card failure. This is impacting many customers throughout NSW. They are planning to replace the card tonight.
May 19, 13:44 AEST
Identified - We are experiencing an issue with a small number of NSW based Telstra ADSL services. Telstra have confirmed a fault exists within their network this morning. Some services will not authenticate.
May 19, 10:17 AEST
Update - AAPT have further advised today that they expect the rebuild of core infrastructure at the exchange to require a further two weeks, at which point AAPT can commence restoring services.
May 19, 09:29 AEST
Update - Telstra's service status has updated to indicate service restoration between Friday 19th May and Monday 29th May, however these dates are subject to change. AAPT have advised they expect to be restoring services next week, however this is dependent on site cleanup.
May 17, 09:48 AEST
Update - Telstra have advised that some services have been restored over the weekend, however we anticipate that issues relating to this outage will continue for several weeks due to the complexity of the event and the complete destruction of the exchange caused by the fire.
May 15, 10:18 AEST
Update - Telstra are still waiting for the arrival of the MEOW. We have also been advised of a substantial amount of toxic materials on site which need to be removed before service restoration can occur.
May 10, 14:03 AEST
Update - Telstra are deploying the Mobile Exchange on Wheels (MEOW) from Victoria and it should arrive today.
May 10, 11:07 AEST
Update - Some photos of the external damage: https://twitter.com/big_gaffa/status/861778669324382208
May 9, 14:16 AEST
Update - Site investigations have been completed, and Telstra has determined they are deploying a mobile exchange to the site to begin to restore services. There is no Estimated Time for Restoration at this stage.
May 9, 13:11 AEST
Identified - Telstra has advised of a fire in the Parklea Exchange overnight. The exchange is currently inoperable. Services fed from this exchange are offline at present with no ETR.
May 9, 09:10 AEST
Network Services Operational
IP Transit Operational
Ethernet services Operational
NBN Operational
ADSL Services Operational
Fixed Wireless Ethernet Operational
Intercapital Capacity Operational
3G/4G Data Services Operational
Private WAN (MPLS VPN) Operational
Managed VPN Operational
Voice Services Operational
Voice trunking Operational
Hosted voice Operational
ISDN Services Operational
Telstra Rebill Operational
Email Services Operational
SpamCloud Operational
Hosted Exchange Operational
POP/IMAP Legacy Email Operational
Hosting and Data Centre Operational
Virtual Machine Hosting Operational
Co-location Operational
Web Hosting Operational
Managed Backup (Dell Rapid Recovery) Operational
Managed Backup (Veeam) Operational
Billing Systems Operational
Helpdesk Services Operational
AWS cloudFront Operational
AWS ec2-ap-southeast-2 Operational
AWS ec2-ap-southeast-1 Operational
Xero Ltd - API Accounting API Operational
Xero Ltd - API Files API Operational
Xero Ltd - API Payroll API Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
Network Availability (syd01 Data Centre)
Fetching
Network Availability (syd02 Data Centre)
Fetching
Network Availability (mel01 Data Centre)
Fetching
SIP Trunking Availability
Fetching
Hosted Voice Availability
Fetching
Past Incidents
May 24, 2017

No incidents reported today.

May 23, 2017

No incidents reported.

May 22, 2017

No incidents reported.

May 21, 2017

No incidents reported.

May 20, 2017

No incidents reported.

May 18, 2017

No incidents reported.

May 17, 2017
Resolved - This incident has been resolved.
May 17, 09:47 AEST
Monitoring - Engineers identified three circuits affected by a configuration issue which has now been resolved.
May 17, 09:14 AEST
Investigating - We are investigating reports of issues with some of our AAPT MBE services.
May 17, 08:41 AEST
Completed - The scheduled maintenance has been completed.
May 17, 03:00 AEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 17, 01:01 AEST
Update - Due to unforeseen circumstances this work has been pushed forward 24 hours.
May 16, 00:57 AEST
Update - Just to clarify this work on a single router at our NextDC M1 POP in Melbourne, VIC.
May 12, 19:19 AEST
Scheduled - We have identified a service effecting issue with the software version one of our edge routers at NextDC M1 is currently running. Our vendor has advised the only work around is to move to another software version. We expect the outage duration to be around 10 minutes for most services in Melbourne. Engineers will be performing this work in the low traffic period late at night. If there is any issue with this we can pre-emptively move services to another edge router within the POP. For enquiries regarding this please submit a ticket with our help desk.
May 12, 19:18 AEST
May 16, 2017

No incidents reported.

May 15, 2017
Resolved - We are now seeing two way call flow for all services.
May 15, 10:55 AEST
Monitoring - We have identified a further issue following the outage this morning and have implemented a further configuration change to resolve call flow.

The subsequent issue was due to an incorrect authentication setting being used following the changes made to resolve the earlier issue between media nodes. This has been rectified and now full communication between nodes has been restored.
May 15, 10:39 AEST
Identified - Engineers are continuing to identify a subsequent issue affecting call flow
May 15, 10:27 AEST
Monitoring - Due to an earlier system failure, some inbound calls failed to our vpbx platform this morning. Engineers have identified the affected component and restarted the service. In addition they have identified the root cause of the failure as a legacy configuration element, which they have removed. The failure scenario has since been tested.
May 15, 10:21 AEST
Resolved - This incident has been resolved.
May 15, 10:19 AEST
Identified - We are seeing failures with the legacy CDN based clouddns.vc name servers this morning. We are working with the provider to identify and restore the services associated with this domain.
May 10, 11:10 AEST
May 14, 2017

No incidents reported.

May 13, 2017

No incidents reported.

May 12, 2017

No incidents reported.

May 11, 2017

No incidents reported.