Cereal Killer App

Remember Sugar Bear? He was the coolest of the early 70s cartoon cereal ad mascots, way cooler than the Trix rabbit or the Fruit Loops toucan. At CISCO TAC Bootcamps we adopted him as our guru and I code-named this one project Super Sugar Crisp, mainly so I could hear serious network engineers refer to the thing by that name.

The Problem

in 2012 CISCO’s Technical Assistance Center (TAC) was a huge operation contributing to the Service and Support organization accounting for 1/3 of CISCO’s overall revenue. As with most call centers, support calls are initially handled by Level 1 engineers, and escalated to Level 2 if they aren’t able to resolve the issue. L2 escalations were carefully tracked and cost-accounted and at CISCO scale, any reduction in the number of escalations would have a huge bottom-line impact.

TAC engineers had access to a massive amount of support and training content, but it was distributed across a number of legacy sites managed by a variety of teams. This included support content like user and administrator guides, training content, data sheets, New Product Introduction (NPI) videos and presentations, release notes, software version lookups and more.

Finding the appropriate content in a timely manner while a customer was on the line was nearly impossible as all of these sites were organized for browsing rather than searching.

The Solution

Instead of trying to re-aggregate all this content under a single system, I designed and developed what amounted to a federated search engine tied to a metadata database with URL references to each content resource, document or file. Working primarily with Josh McCarty under Steve Roche and his boss John Rouleau, we scraped 15 or so source repositories and built the metadata database using MySQL. Metadata included summary description, URL, publish date, product and product family, version numbers, content type and more.

Then we built the back end application using an object-oriented, MVC-like model, with PHP Zend and PEAR frameworks and Zend Lucene as the search engine. Lucene is incredibly powerful for an easy-to-implement, open source search engine. We crawled the database and source repositories weekly to keep the data current.

The front end used the Domo javascript framework to display results (JQuery was just coming into fashion) as this allowed us to easily integrate into the CISCO intranet. We used a simple RESTful API and returned results in JSON format that was easily parsed into Domo widgets.

The Results

Within the first six months of deployment we found the system in widespread use by the TAC and L2 escalations being reduced in some areas by as much as 40%. Steve ran the numbers and found we were saving the overall TAC around $20M on an annualized basis. He submitted the project for a CISCO Pinnacle Award from Ana Pinczuk’s TAC organization under Joe Pinto, and we won in early 2013.