tag:status.driveyello.com,2005:/historyDrive Yello Status - Incident History2024-03-29T16:22:44+11:00Drive Yellotag:status.driveyello.com,2005:Incident/163866662023-03-08T19:15:26+11:002023-03-08T19:29:49+11:00Elevated public API errors<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>19:15</var> AEDT</small><br><strong>Resolved</strong> - Today, the 8th of March 2023 at approximately 3:30pm AEST, we started seeing an elevated level of API errors with our public facing services.<br /><br />All public services were partially impacted, with the quote api hit the hardest. Clients would have been receiving a high number of error responses from this time. Order creation was also impacted at this time with an elevated error response being returned and also resulted in a small number of partially created orders.<br /><br />Upon investigation, there was an issue with a common downstream dependency, a shared distributed cache (redis). The poor performance of the cache due to a large and sustained spike in memory requirements, had knock on effects with multiple services which led to the incident.<br /><br />Upon finding the root cause, the team immediately remedied the situation and all services returned to normal at approximately 4:06pm.<br /><br />The system is being closely monitored and partially created orders have been rectified. A further investigation into future preventative measures is under way.</p>tag:status.driveyello.com,2005:Incident/68492312021-04-27T14:22:25+10:002021-04-27T14:22:25+10:00Here Maps Outage<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>14:22</var> AEST</small><br><strong>Resolved</strong> - The third-party mapping service (Here Maps) are still experiencing the outage, however, the impact has been minimised. The traffic is fully routed to another mapping service and all Yello services reporting healthy since. We will continue to monitor the relative services.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>12:15</var> AEST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>12:00</var> AEST</small><br><strong>Monitoring</strong> - Most of Yello services have been automatically rolled back to another third-party mapping service. Public Quote API and Public Orders API performance has been impacted due to the third party outage, and we have experienced a higher 400 HTTP error rate than normal in Quote API. We have switched the primary third party mapping in the affected APIs to another service. Errors have been resolved for the past hour, we will continue monitoring the performance and failures.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>10:00</var> AEST</small><br><strong>Investigating</strong> - We’re experiencing errors with some of the APIs that have dependencies on the Mapping service due to one of our third-party Mapping outage. Our team is actively working to minimise the impact on our services. We will provide more details as soon as more information becomes available.</p>tag:status.driveyello.com,2005:Incident/51722702020-09-25T12:07:19+10:002020-09-25T12:09:28+10:00We are investigating the increase error rate on Google Maps API<p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>12:07</var> AEST</small><br><strong>Resolved</strong> - The issue is fully resolved and all APIs are operational for the past 35 minutes. We have started receiving errors from Google Maps geocoding from 11 AM - 11:22 AM. This caused performance degradation and an increase of 400 HTTP errors on Quote API when using geocoding functionality. The outage of the Firebase short link API did not have a major effect on the Yello consumer tracking service as it was automatically switched to an internal link. We will implement further failure handling functionality in the mentioned API for better handling of third party incidents. Thank you</p><p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>11:46</var> AEST</small><br><strong>Monitoring</strong> - Google Maps API error rate has dropped and seems to be stable for the past few minutes, affected Yello APIs are fully operational since. The main affected Google Maps APIs that the Yello platform uses are geocoding and Firebase short links API. We will keep monitoring and update this incident with more details.</p><p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>11:28</var> AEST</small><br><strong>Investigating</strong> - We are investigating the increased error rate on Google Maps API, seems Google Maps Geocoding API is affected the most.</p>tag:status.driveyello.com,2005:Incident/46345822020-07-17T13:28:09+10:002020-07-17T13:28:09+10:00Consumer Tracking Short URL error<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>13:28</var> AEST</small><br><strong>Resolved</strong> - We have disabled consumer tracking URL shortener in all the relative systems including the webhook events for integrations. Now the provided link for consumer tracking should directly point to Yello Tracking Service which is fully operational. We will enable the short link once the third-party service issue is resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>11:45</var> AEST</small><br><strong>Update</strong> - We have released a change to bypass the URL shortener in the SMS tracking and consumers tracking links should be working fine.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>11:20</var> AEST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>11:00</var> AEST</small><br><strong>Investigating</strong> - We have identified an error in the third party URL shortener service that is used in the SMS tracking. The tracking short URL is returning a not found error and we are currently investigating the problem. Yello consumer tracking service is fully operational and we will be working on a release to not use the URL shortener in the meanwhile.</p>tag:status.driveyello.com,2005:Incident/36229272020-02-23T14:30:30+11:002020-02-28T10:41:53+11:00Courier app login problem<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>14:30</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>14:10</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>13:45</var> AEDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>12:50</var> AEDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.driveyello.com,2005:Incident/34344242019-10-25T13:15:45+11:002020-02-28T10:02:31+11:00Experiencing downtime on API<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>13:15</var> AEDT</small><br><strong>Resolved</strong> - The system is fully recovered and all APIs are operating normally</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>12:05</var> AEDT</small><br><strong>Monitoring</strong> - Continuing on recovering the system</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>11:55</var> AEDT</small><br><strong>Identified</strong> - Public API integration and ordering system are experiencing downtime</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>11:45</var> AEDT</small><br><strong>Investigating</strong> - Receiving errors on API integrations</p>