tag:status.rwts.com.au,2005:/historyReal World Technology Solutions Status - Incident History2024-03-29T14:54:47+11:00Real World Technology Solutionstag:status.rwts.com.au,2005:Incident/201632052024-04-06T10:00:00+11:002024-03-05T16:19:57+11:00Planned maintenance affecting services handed off in NextDC C1 (Canberra)<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>6</var>, <var data-var='time'>10:00</var> - <var data-var='time'>14:00</var> AEDT</strong></p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>16:19</var> AEDT</small><br><strong>Scheduled</strong> - Engineers will be relocating network equipment at NextDC C1. Services will experience an outage of up to 1 hour during the maintenance window.</p>tag:status.rwts.com.au,2005:Incident/203476712024-03-25T18:54:36+11:002024-03-25T18:54:36+11:00Partial outage for Darwin services<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>18:54</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>17:07</var> AEDT</small><br><strong>Monitoring</strong> - All services should be restored. Please contact support if there is any further impact.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>16:51</var> AEDT</small><br><strong>Update</strong> - Some services have been restored. Engineers are working to restore remaining services.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>16:26</var> AEDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>16:25</var> AEDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>16:24</var> AEDT</small><br><strong>Investigating</strong> - We've identified an outage for services out of our Darwin POP. This is affecting a number of services. Engineers are investigating.</p>tag:status.rwts.com.au,2005:Incident/201608812024-03-06T10:48:27+11:002024-03-06T10:48:29+11:00CloudPBX Desktop Licensing Issues<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:48</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>09:19</var> AEDT</small><br><strong>Identified</strong> - Users of CloudPBX Desktop who have not updated their client since October 2023 may receive a notice that their license key has been revoked.<br /><br />This is due to a change in the licensing system used by CounterPath/Allianza, our software vendor who manufactures this application.<br /><br />We have become aware that the auto update mechanism that was integrated into CloudPBX Desktop may have not correctly worked for all users.<br /><br />Users who receive a message that their licence has expired can download the updated client at https://rwts.com.au/cpbxdesktop/ and sign in with their credentials. If you do not know these or need any assistance please contact our Helpdesk.</p>tag:status.rwts.com.au,2005:Incident/199433212024-02-09T10:44:52+11:002024-02-09T10:44:52+11:00Managed VPN Emergency Maintenance<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:44</var> AEDT</small><br><strong>Resolved</strong> - All patches have been applied, and the emergency maintenance has been completed. All services are now restored.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>09:43</var> AEDT</small><br><strong>Identified</strong> - Our managed VPN service (Pulse/Invanti Secure Access) is currently unavailable while we are performing emergency maintenance. The vendor has released patches for new security vulnerabilities, which must be applied immediately. We expect the service to be unavailable for no more than 1 hour.</p>tag:status.rwts.com.au,2005:Incident/199180692024-02-06T14:39:32+11:002024-02-06T14:39:32+11:00CloudPBX Web Interface Authentication Issues<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:39</var> AEDT</small><br><strong>Resolved</strong> - Engineers have resolved an issue affecting login services for the CloudPBX Web Administration Portal, and all services are now restored.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:03</var> AEDT</small><br><strong>Identified</strong> - Our team are currently resolving an issue affecting authentication in the CloudPBX Web Interface. No other components of CloudPBX are affected by this incident.</p>tag:status.rwts.com.au,2005:Incident/199085752024-02-05T14:32:27+11:002024-02-05T14:32:39+11:00Managed VPN Emergency Maintenane<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>14:32</var> AEDT</small><br><strong>Resolved</strong> - Emergency maintenance has been completed, and all services are now restored.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>12:39</var> AEDT</small><br><strong>Identified</strong> - Our managed VPN service (Pulse/Invanti Secure Access) is currently unavailable while we are performing emergency maintenance.</p>tag:status.rwts.com.au,2005:Incident/199088112024-02-05T13:43:28+11:002024-02-05T13:44:17+11:00CloudPBX Home Server unavailability<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>13:43</var> AEDT</small><br><strong>Resolved</strong> - Our team have resolved an issue affecting CloudPBX Home Servers 08 and 09. All services are now restored.</p>tag:status.rwts.com.au,2005:Incident/198504442024-02-05T13:19:08+11:002024-02-05T13:19:08+11:00Storage Fabric Availability in syd02<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>13:19</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>13:18</var> AEDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>23:53</var> AEDT</small><br><strong>Monitoring</strong> - Engineers have corrected the hardware fault and restored all ancillary services. Any customers who continue to have any service issues should contact our help desk.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>23:16</var> AEDT</small><br><strong>Update</strong> - Engineers have arrived on site.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>22:40</var> AEDT</small><br><strong>Identified</strong> - Engineers have identified a fault condition in a redundant storage fabric where controller B has failed, but some volumes have not successfully migrated due to a dirty cache operation. Engineers are liaising with vendor technicians to identify next steps.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>22:37</var> AEDT</small><br><strong>Investigating</strong> - Engineers are investigating an issue affecting a storage fabric in our syd02 data centre. Currently multiple services are impacted by a suspected hardware failure.</p>tag:status.rwts.com.au,2005:Incident/197207912024-01-16T09:30:00+11:002024-01-16T18:00:48+11:00Packet loss for internationall traffic<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>09:30</var> AEDT</small><br><strong>Resolved</strong> - At approximately 9:30am this morning our team identified an issue affecting some international traffic. The incident was caused due to a packet storm on the NYIIX fabric which we are part of, which impacted a number of providers (including Real World). We disabled the affected interfaces and re-routed traffic, restoring services.</p>tag:status.rwts.com.au,2005:Incident/194607002023-12-17T14:57:02+11:002023-12-17T14:57:02+11:00CloudPBX voicemail outage<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>14:57</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>14:56</var> AEDT</small><br><strong>Monitoring</strong> - Engineers have investigated reports of users being unable to leave messages on their CloudPBX voicemail system. We have identified the cause and applied a fix to restore full functionality.</p>tag:status.rwts.com.au,2005:Incident/193852772023-12-10T09:22:05+11:002023-12-10T09:22:06+11:00NBN outage affecting services in NT and SA<p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>09:22</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>23:10</var> AEDT</small><br><strong>Monitoring</strong> - A fault was identified with a fibre transceiver. This has been replaced and services have been restored.<br /><br />If you continue to have issues, please restart your NBN modem.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>22:55</var> AEDT</small><br><strong>Investigating</strong> - We have seen NBN services in NT and SA go offline. Engineers are working on identifying the cause.</p>tag:status.rwts.com.au,2005:Incident/191768762023-12-05T01:07:06+11:002023-12-05T01:07:07+11:00Scheduled fibre relocation works affecting some services in NSW and ACT<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>01:07</var> AEDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>00:28</var> AEDT</small><br><strong>Verifying</strong> - Services are back online. Final validation work is now taking place.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>22:00</var> AEDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>18:30</var> AEDT</small><br><strong>Scheduled</strong> - Technicians will be performing fibre relocation works that will impact a number of Ethernet and IP transit services.<br />This work will affect some Ethernet and IP transit services handed off in NextDC S1 and services at 43 Eastlake Pde, Kingston ACT. <br />The outage duration is expected to be up to 3 hours.</p>tag:status.rwts.com.au,2005:Incident/191023192023-11-27T22:08:40+11:002023-11-27T22:08:43+11:00Planned maintenance affecting services in Kingston ACT<p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>22:08</var> AEDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>27</var>, <var data-var='time'>22:00</var> AEDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>10:42</var> AEDT</small><br><strong>Scheduled</strong> - Planned maintenance will affect all services at 43 Eastlake Pde, Kingston ACT.<br />After the maintenance window, if services are offline please power cycle NTUs and customer routers.</p>tag:status.rwts.com.au,2005:Incident/191018472023-11-14T08:38:01+11:002023-11-14T08:38:02+11:00Telstra outage in NT<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>08:38</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>08:37</var> AEDT</small><br><strong>Update</strong> - All services have been restored.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>16:05</var> AEDT</small><br><strong>Update</strong> - We have seen most services restored, however we are aware of at least one service still affected.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>14:07</var> AEDT</small><br><strong>Update</strong> - Technicians have almost finished hauling a new cable, one end is being prepped for splicing, anticipated service restoration in the next couple of hours.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>09:44</var> AEDT</small><br><strong>Update</strong> - Telstra technicians are on-site working to repair fibres. There is no ETR available at the moment.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>09:16</var> AEDT</small><br><strong>Identified</strong> - We've seen some Telstra services drop in NT. There is a fibre break near Noonamah.<br />Telstra have repaired some services overnight, and are continuing to work on restoring remaining services.<br />On-net services to Darwin are running over an alternate path and may see some change in latency.</p>tag:status.rwts.com.au,2005:Incident/190507442023-11-08T21:46:38+11:002023-11-08T21:46:38+11:00Optus outage<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>21:46</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>16:32</var> AEDT</small><br><strong>Monitoring</strong> - Services appear to be back online.<br /><br />If you are still experiencing an outage, please reboot your LTE modem.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:16</var> AEDT</small><br><strong>Update</strong> - We are starting to see a few services come back online.<br /><br />However there has not yet been official word that services have been restored.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:37</var> AEDT</small><br><strong>Identified</strong> - Optus are experiencing a national outage. This is affecting all mobile SIMs on the Optus network. <br /><br />There is no restoration ETA at this stage.</p>tag:status.rwts.com.au,2005:Incident/184872072023-09-13T18:32:12+10:002023-09-13T18:32:12+10:00Data centre power issues effecting services<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>18:32</var> AEST</small><br><strong>Resolved</strong> - Investigations by our staff and data centre technicians have found a possibly faulty device that cause circuit breakers upstream of the rack to trip. We do not anticipate any further service issues in relation to this.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>16:27</var> AEST</small><br><strong>Monitoring</strong> - Power is restored and we are working with the data centre staff at present.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>16:25</var> AEST</small><br><strong>Investigating</strong> - We are currently investigating alerts from our monitoring systems regarding loss of power to some equipment in our data centre location NextDC S1. There were power works expected in some areas but we have seen a power loss in racks not covered by the providers powers works.</p>tag:status.rwts.com.au,2005:Incident/183033802023-08-30T17:03:28+10:002023-08-30T17:03:28+10:00Optus 4G data outage<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>17:03</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>08:50</var> AEST</small><br><strong>Update</strong> - We are continuing to see services come back online. Some modems/routers may need a reboot to re-establish connectivity.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>20:25</var> AEST</small><br><strong>Update</strong> - Please reboot your modem/router if your 4G data service has not come back online.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>17:49</var> AEST</small><br><strong>Monitoring</strong> - The issue has been identified and a fix implemented. Services should come back online shortly.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>16:50</var> AEST</small><br><strong>Investigating</strong> - We have seen a number of 4G LTE services go offline that are supplied via the Optus network. We are investigating the issue with our upstream provider.</p>tag:status.rwts.com.au,2005:Incident/175663322023-06-26T01:00:05+10:002023-06-26T01:00:06+10:00Firewall maintenance<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>01:00</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>21:00</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>09:42</var> AEST</small><br><strong>Scheduled</strong> - Engineers will be performing maintenance on central firewall platforms. Some services will experience an outage of up to 30 minutes during the maintenance window while the work is performed.</p>tag:status.rwts.com.au,2005:Incident/176511562023-06-25T06:30:11+10:002023-06-25T06:30:12+10:00Emergency firewall reboot<p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>06:30</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>05:30</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>17:29</var> AEST</small><br><strong>Scheduled</strong> - A central firewall requires a reboot to clear a fault condition.</p>tag:status.rwts.com.au,2005:Incident/175663142023-06-19T00:19:44+10:002023-06-19T00:19:45+10:00Firewall maintenance<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>00:19</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>21:00</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>09:40</var> AEST</small><br><strong>Scheduled</strong> - Engineers will be performing maintenance on central firewall platforms. Some services will experience an outage of up to 30 minutes during the maintenance window while the work is performed.</p>tag:status.rwts.com.au,2005:Incident/173382942023-05-29T00:00:09+10:002023-05-29T00:00:09+10:00Service migrations<p><small>May <var data-var='date'>29</var>, <var data-var='time'>00:00</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>28</var>, <var data-var='time'>22:01</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>17:45</var> AEST</small><br><strong>Scheduled</strong> - Some customer internet services terminated in NextDC S1 will be migrated onto new hardware.<br />In most cases services will experience an outage of no more than a minute.</p>tag:status.rwts.com.au,2005:Incident/171163932023-05-22T00:04:08+10:002023-05-22T00:04:10+10:00Ethernet port migrations in NextDC S1<p><small>May <var data-var='date'>22</var>, <var data-var='time'>00:04</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>20:00</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>17:00</var> AEST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>16:51</var> AEST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>14:58</var> AEST</small><br><strong>Scheduled</strong> - Engineers will be migrating some physical ports onto new hardware for customer services that are handed off in NextDC S1.<br />There will be an outage of up to 30 minutes for each service during the maintenance window.</p>tag:status.rwts.com.au,2005:Incident/172550762023-05-16T18:00:09+10:002023-05-16T18:00:09+10:00Telstra Transmission Fault - syd02 (NEXTDC S1)<p><small>May <var data-var='date'>16</var>, <var data-var='time'>18:00</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>14:32</var> AEST</small><br><strong>Monitoring</strong> - Telstra fibre crew has repaired the damaged cable and services have been restroed.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>10:11</var> AEST</small><br><strong>Update</strong> - Telstra technicians are on site repairing the fibres.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>07:13</var> AEST</small><br><strong>Update</strong> - Telstra have identified a fibre break and have dispatched field technicians.</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>23:42</var> AEST</small><br><strong>Identified</strong> - We have identified a transmission fault within Telstra's network affecting some Ethernet Access services terminating in NEXTDC S1 (our syd02 data centre). This is also affecting some Sydney Telstra based transit.<br /><br />Our NOC have spoken with Telstra Engineers, who are aware of the fault and are working to resolve.</p>tag:status.rwts.com.au,2005:Incident/172525932023-05-15T23:00:27+10:002023-05-15T23:00:28+10:00pe01.adl01 and pe01.drw01 emergency reboot<p><small>May <var data-var='date'>15</var>, <var data-var='time'>23:00</var> AEST</small><br><strong>Resolved</strong> - We further identified an issue caused by a third party route leak that was also affecting both pe01.adl01 and pe01.drw01 and have completed a configuration change to resolve the impact. All services are now restored.</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>19:27</var> AEST</small><br><strong>Monitoring</strong> - The reboot has been completed and engineers are verifying network behaviour has returned to normal.</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>19:14</var> AEST</small><br><strong>Identified</strong> - Engineers have identified a fault condition requiring emergency reboot on this device. This will commence shortly.</p>tag:status.rwts.com.au,2005:Incident/172488012023-05-15T22:59:12+10:002023-05-15T22:59:14+10:00SSL VPN Currently Unavailable<p><small>May <var data-var='date'>15</var>, <var data-var='time'>22:59</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>10:43</var> AEST</small><br><strong>Monitoring</strong> - Services have been restored and are being monitored</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>09:35</var> AEST</small><br><strong>Identified</strong> - We are currently investigating an issue impacting the availability of our SSL VPN via Pulse. Next update 12pm AEST.</p>