A bad Day?
Last week, we (Alcatel-Lucent)were one of the companies impacted by the Jive Software Outage (@JiveSoftware). Our platform Engage runs on Jive S.B.S. At first, it seemed like it was going to be a long day. Working for a European company, this hit when they were all working (11am), and most of us in the US were still in bed. There was very little (no) communications to indicate that Jive was aware of the problem, let alone working on it. It was starting to get a bit frustrating entering hour 2 of a global outage.
By all account, it was through Facebook that I actually learned that other companies were impacted, not just us. Ted Hopton (@Ted_Hopton), one of my peers from the 2.0 Adoption Council noted, “does not like bad news when he first checks the Blackberry in the morning.”
followed by a comment when queried by another of his connections, “No, company website I run crashed.”
It was then that I realized that we were not alone. That single piece of information actually made me feel better. Why? Because since multiple customers were impacted, I was sure that it was getting attention.
What was starting off not so good, actually turned around and was almost magic. After being beaten up by us (and I’m sure a bunch of other companies about the poor communications), Jive really rose to the occasion and opened a Webex session to communicate to everyone. This information was communicated back to the Alcatel-Lucent team through their support site. This was not just “we’re doing everything possible restore service“, but to a level of root cause analysis, identifying the hardware that failed, etc.
What they had done is reduce their effort to communicate by consolidating what must have been 90+ conversations, down to one. During the Webex, they provided updates every 15 minutes, even if just to say that they were still on track with the previous update. This was fantastic, now as business owners (and other related roles), we now had information that could help us make decisions for our respected companies that we could act upon.
The Day at Alcatel-Lucent
Internally, we used Yammer to relay that message to our community. I must say that Kevin Joyce (@cNet2u0) did a fantastic job of keeping the community informed. Not only did Kevin provide updates, but also took the time to answer questions from the community. Additionally, after the service was restored, Kevin wrote a blog indicating a timeline and actions was shared with everyone. A stellar job of working transparently inside the company. His efforts were openly appreciated by many of the community.
The feedback included statements like, “Thanks for providing this excellent transparency into what happened. Really appreciated.” and “Kudos for making this info ‘public’”
What I really love is that by taking this proactive approach to communicating that everyone felt that they were informed and didn’t have to get a ton of phone calls, emails, etc. to feel that they knew exactly what was going on throughout the process.
By using Yammer, the Yammer community could be kept informed without having to go out and look for status updates. While everyone was frustrated, the information was there for all to see.
What still needs work
In hindsight, while members of the Yammer community were well informed, people who were not members, were not as well informed. What this indicates to me is that we need to work harder to solidify the communications channels so that every member of the community knows exactly where to go to stay informed.
This really makes me wonder if the platform approach is the “best” approach. If you have an outage, you are totally out of the water. Even with fault tolerance and redundancy, things go wrong. If not for having a separate channel for communication (in our case Yammer). We would have had to rely on our Corporate Internet (which requires a rather complicated approval process) or e-mail to communicate.
Overall, I think that by sharing the details of a crisis that it gives insight into the challenges that we face. It also helps to build a trust that you are not going to just share the good stuff. Finally, it provides visibility into the decision making process. I am very proud of the team that supports our Engage platform. Their openness about the details (good and bad) regarding the outage is being the change that we hope to see in a large company.
My hats also off to Jive for taking this very open approach with their customers. Being a company that sells software to provide transparency, sets expectations very high that they indeed will be transparent. Jive has committed to sharing a retrospect on the event. I am very confident that the challenges that started the day will improve and that fail-over testing will become part of their standard operating procedures.
What started off to be the makings of a very bad day, turned out to be a mostly positive experience and exemplifies the value of transparency in this highly connected, “social” world we live in.