Wrangling Enterprise Data


In our previous blog, The Myth of One Golden Informational Warehouse, we described the ideal informational warehouse where all relevant corporate information resides in an easily searchable, coherent, and up-to-date form. We also discussed how individual preferences, habits and organizational culture make it hard to achieve such a warehouse.

In this article, we will try to analyze this problem from the “systems” perspective. Systems, in this case, consist of tools, processes and the data that they hold and operate.

Data and Tools


The above diagram describes various types of data and the tools and services a typical organization uses. For the sake of simplicity, this diagram only includes systems that are pervasive and used frequently by employees.

As you can see, the tools on the left tend to be “systems of engagement” – natural, conversational and dynamic; while tools on the right tend to be “systems of record” – highly structured, curated and managed. Moreover, communication mediums like E-mail and IM are noisier (e.g. due to language ambiguities) from the information perspective compared to structurally curated information repositories like CRM.

Unfortunately, internal knowledge organizations in enterprises have this unenviable task of wrangling all these sources of information into a cohesive, searchable, navigable solution that is non-disruptive, secure, and easy to use.

The Problem/Solution Gap

It is quite easy to see why knowledge or IT organizations prefer stricter, more structured services (right side of the spectrum). For example, extracting a customer name or an employee assigned to a task is much easier from a CRM or PM tool where specific fields capture that information. However, it is much harder to extract such information from Slack messages or E-mail conversations.

On the other hand, employees hate to curate information for the system. They have a job to do after all, be it development, sales or marketing. Documenting something is just an overhead they would rather avoid.

And therein lies the dilemma. Employees are communicating through IM or E-mail and generating tons of useful information in a dynamic work environment – and almost none of it makes it to the system of record. Meanwhile, the more structured sources that contain no IM or E-mail are the places from where business intelligence is derived.

Needless to say, this deep rooted and pervasive disconnect has created a gap in the way employees access corporate information relevant to their jobs.

Enterprise Wiki: Shifting to the Right

A good example of the tension described above is Enterprise Wiki. A few years ago when wikis were all the rage, there was a massive effort to shift collaboration from chat and E-mail to wiki.


As demonstrated in the diagram, many forward looking organizations thought that this new initiative would make their data easier to manage by shifting to a more structured, organized, and system ready service.

Unfortunately, this shift required a change in behavior from employees as they had to learn a new user interface and process which was less natural than before. While some appreciated the change, others were not quite so enthusiastic. For example, Product Marketing may love wiki pages as they are well organized and easy to manage. However, for the Sales team (supposedly the benefactor of this content), such a solution would be a burden as they may not have access to corporate network all the time and may want a mobile friendly solution. They just need answers, not documents. They would naturally stick with E-mail or chat.

Indeed, research from MITRE has suggested that people have resisted putting information onto wikis because:

First, we uncovered a reluctance to share specific information due to a perceived extra cost, the nature of the information, the desire to share only “finished” content, and sensitivities to the openness of the sharing environment. Second, we discovered a heavy reliance on other, non-wiki tools based on a variety of factors including work practice, lack of guidelines, and cultural sensitivities.

In other words, the failed adoption of wikis was part of the systematic difference in expectations between the consumer and the producer of the information.

The Ideal Solution

With organizations wanting more clean and structured data that is easier to slice and dice for business intelligence and employees wishing for more conversational style mediums, an ideal solution would have to fill the gap of expectations.

In an ideal world, employees would be able to converse in a free-flowing manner using whatever means their group feels is best, while innovative technologies and products would analyze that content and extract valuable business information to form a structured database. Thankfully, traditional search technology along with linguistics and machine learning applied on very specific areas like sales and support can make this problem tractable.

We at Nimeyo are working on technology where employees can continue use E-mail, chat, SharePoint, Salesforce or anything else they desire, while our algorithms build the knowledge meta-layer on top of that.

With our “pods and bots” approach, our pods aggregate unstructured information and add a layer of intelligence, while our bots deliver answers right into your workflow. Essentially, we require no change in user behavior while still tapping into one of the richest sets of corporate information.
Drop us a note if you would like to learn more.

Can Enterprise Search effectively serve employees’ needs?

My toddler has an impressive collection of toys. He tries to keep his favorite ones somewhere “safe” but then he cannot find them when he wants to actually play with them. While trying to find it, we both know that the one that we are looking for is somewhere in the house and yet it remains alluringly out-of-reach since the exact places we determine to look for never have it. It’s frustrating for him to have this happen on regular basis.

Unfortunately, millions of enterprise employees feel similar frustration when they can’t find the information that they know is around them in various forms. They could be in mailing lists or in SharePoint or Box repositories, or in internal chat rooms, or on some internal wiki pages.

Realizing the potential value of unearthing information that employees need to do their jobs, companies – particularly large ones – employ enterprise search. Unfortunately, most of those engines remain poorly deployed and minimally used by the end users.

imageSo the question is why do enterprise search engines do such a poor job at engaging users whereas search outside of the corporate firewalls are part of our daily lives?
Although there are number of technical reasons, we believe the key problem is a “lack of user-orientation”. In other words, these solutions are neither attuned to the actual needs of the end users nor they understand the data itself in a meaningful way to be able to serve it in a meaningful way.

Let’s take few examples –

  1. Content and Users: Search engines’ key strength is in indexing wide ranging data types – web pages, documents, CRM systems etc. So when user searches with few keywords, search engines define success by uncovering wide-ranging data in a sequential manner based on some ranking criteria.However, not all data types are created equal. For instance, email communications are a lot more meta-data rich and time-relevant compared to documents. If used intelligently, such data specific analysis can be immensely useful in understanding how row communication relates to the end user needs.
  1. User interface:   Needless to say, we all are used to typing a simple search query (or question where there is a unique answer – e.g. “Father’s day 2015”) and expecting search engine to return “satisfactory” results. However, in a corporate context, this model is highly limiting as any one article or document is unlikely to provide a comprehensive answer in most real world scenarios.For example, when a sales rep gets into a competitive situation, a query like “MyCompany vs MyCompetitor” should surface variety of information including product differentiation, pricing, and nuggets from other similar situations. All these pieces are equally important to put together a competitive tactic to win the opportunity. A linear listing of results based on uniform ranking criteria does not do justice to needs of that sales rep.A UI that allows users to navigate through these various pieces in a consistent manner and “assist” in creating a cohesive picture would be much more effective in creating engaging user experience.And finally,
  1. User preferences and behaviors: In most of our enterprise search experience, presentation of results is “black magic”. Let’s say you searched for some information today, worked through the hits and were fortunate to find what you were looking for at the 20th You had to put the effort but you found what you were looking for!Unfortunately, say a month from now, if you are looking for the similar information through a similar query, you would still find the information deep down in the ranking.If solutions allowed ways to capture user preferences – expressed implicitly and explicitly – it would be able to return results that are a lot more aligned with what the end user needs are.

We at Nimeyo believe that to achieve the true potential of enterprise search, we need to stop viewing it through the prism of consumer search technologies.

To fulfill the promise, enterprise search products must understand not just who the user is and what role she performs in the organization but also identify optimal mechanism through which to deliver knowledge to the employee.