Google Cloud Problem

Incident Report for AfterShip Shipping

Postmortem

With regard to disruptions of our services on 2017-11-30, namely * www.aftership.com * www.postmen.com * secure.aftership.com * secure.postmen.com

which were caused by issues with Google Cloud Networking, we had been informed by Google:

"A system that is responsible for throttling individual instances in cases where there is networking congestion received data from one of its data sources that were implying that there are multiple choke points on our network. This caused throttling for a few VMs while the network had no real choke points in the path of these instances. There are enough action items on our internal post mortem to detect similar issues faster and not have this exact issue happening again."

We will follow instructions from Google to minimize future recurrence.

Posted Dec 17, 2017 - 20:08 EST

Resolved

The outage was caused by an issue with Google Cloud Networking. It had been resolved by Google at 18:46 HKT. We will work with Google to prevent or minimize future recurrence.

Posted Nov 30, 2017 - 07:53 EST

Update

We have not received any status updates from Google Cloud by 18:30 HKT. We continue to monitor the situation.

Posted Nov 30, 2017 - 06:00 EST

Monitoring

Between 2017-11-30 09:50 UTC and 2017-11-30 10:20 UTC the secure web portal is down.
Now the incident has been resolved and the service is fully operational.
We are keeping monitoring the status.

Posted Nov 30, 2017 - 05:22 EST

Update

The issue is caused by Google Cloud Network Problem in US/Pacific. The incident is still occurring.
We are contacting Google Cloud Support to resolve it.

Posted Nov 30, 2017 - 05:06 EST

Identified

Between 2017-11-30 09:50 UTC and 2017-11-30 10:00 UTC the secure web portal is down.
Now the incident is still occurring.

Posted Nov 30, 2017 - 05:01 EST