Slow Environment Provisioning
Incident Report for CloudShare
Postmortem

Slow Environment Provisioning - 10/27/2014, all times PDT

• 4:30am: detected intermittent connectivity on one of our storage servers that manages virtual machines’ RAM state

• 4:40am: Engineering initiated fallback procedures to a backup storage server and environments regained access to their RAM state, however some environments still failed to load or had very long preparation times

• 4:45am: CloudShare restarted a core component of its infrastructure

• 5:10am: detected similar intermittent connectivity on the backup storage server and fell back to a secondary system. Verified that the issue was with the OS of the storage device, rather than the device itself. RAM storage device rebooted

• 8:00am: RAM storage device is functioning normally, however there is a large backlog of environments that need to complete preparing

This was a very rare error with one of our vendor’s devices. CloudShare has put monitors in place to detect this type of event in the future so that the resolution will be quick and seamless.

Posted Oct 29, 2014 - 23:53 UTC

Resolved
This incident has been resolved.
Posted Oct 27, 2014 - 19:22 UTC
Update
Environments are beginning to prepare normally however we continue to monitor the issue closely.
Posted Oct 27, 2014 - 16:09 UTC
Update
Still sporadic cases of long time to prepare.
Should be loading soon.
Posted Oct 27, 2014 - 14:52 UTC
Update
Some users may experience slow load times for their environment.
We have identified the problem and it should be resolved in 10 minutes.
Posted Oct 27, 2014 - 13:53 UTC
Monitoring
Issue has been resolved
Posted Oct 27, 2014 - 12:36 UTC
Identified
Some users may experience slow load times for their environment.
We have identified the problem and it should be resolved in 10 minutes.
Posted Oct 27, 2014 - 11:55 UTC