Why is that? Well it does make it easer for us mortals to grasp the vast amount of data that is around us. Think about the early days of Yahoo! that used a directory approach and manual work to build a whole categorised database of web links.

But then cracks start appearing in our approach. Familiar with the “in which folder am I going to put this email?”-problem? First of all you start to have a gazillion folders and then the problem pops up that an email can belong to two different folders. After a while you start to wonder “where the hell is that message that I filed?”.

Social Networks and Cloud driving the change

As a technologist, I also understand the underlying technical reasons why we want to normalise and categorise data. You first start with “objectifying” your view of the world in an object-oriented language like Java so it is easier to program and maintain those systems. This is persisted in classic row-column structure in a database where you create different tables for different data concepts. You have the student object with its properties stored in the Student table and course object with its properties in the Course table. Now you can create meaningful relations between the two and develop your software accordingly.

When your application starts to get a huge load, you will either spend an awful lot of money on expensive Oracle cluster software or you will use concepts like sharding to split databases to keep it performing. But still, you stick to your normalized Student and Course table structures.

An approach that started to get traction in some of the successful internet startups was the concept of key/value pairs. While not a novel concept at all, it got more popular because we started to get more and more websites that couldn’t cope with the amount of users. The idea is that you don’t normalise data in Course and Student tables, but you just have a key and an associated value combination in one huge table. This approach scales extremely well because it can be easily spread over a cluster of machines. Google’s BigTable uses this as well as Facebook’s Cassandra. It became more “mainstream” with Amazon’s introduction of their SimpleDB that has this key/value concept, as well as Microsoft’s key/value storage in Azure.

SSD discs to the rescue!

The only problem is… that developers have been so brainwashed with the relational row/column concept that the average corporate developer had a hard time grasping this key/value concept and building meaningful apps on top of it.

Both Amazon and Microsoft have realized that and have added a relational layer on top of their key/value data storage engine. This means that the vendors can use key/value for achieving extreme scalability, but they let developers interact through a relational model interface. (It does create a performance penalty, but negligible for most users I think).

But still, even this key/value approach requires that you are modelling your apps and data in a particular way. What if we didn’t have to do ANY effort at all? What if we could just take the vast amount of data we have in our company, dump it on a disc and just being able to find meaningful information AS IS? Yes, we do have the Google Appliance for that, or Microsoft Enterprise Search, but they are merely indexing in a batch process and storing the results in a cache. What if we want a real-time extremely fast search that gives you on the spot information regardless of the data structures?

That’s exactly the gap Oracle wants to fill with their Exadata v2 server. Consider it as a beast of a server, crammed with more than a terabyte of Solid-State Drive (flash) discs that has your data fully stored, it’s fairly comparable with having ALL your data hot in memory, ready to be queried.

Can you grasp the opportunity this presents us? We can achieve real-time search inside our enterprise on vast amounts of unstructured data!

Be like the kitteh

This is again an example that we shouldn’t limit ourselves by the current state of technology. In the past 10 years I’ve seen again and again that eventually, technology will catch up to make things possible that we couldn’t do before.

Most people will quote Einstein and say “imagination is more important than knowledge”, I’d rather tell you to be more like the kitteh at the right. I’m pretty sure that one day, he/she will be able to fax itself to the Burger King to get some delicious “cheezburgerz“.

Impact on Enterprise 2.0

So how does this relate to Enterprise 2.0 I hear you asking. Well, I argue that we shouldn’t lose too much time and effort on meticulously categorising and tagging information and knowledge across all the different systems we have. This is because I believe that it is almost a lost cause anyway: it doesn’t scale and it’s often not the quality you want.

My point is that the tools and technology we get to our disposal are advancing at such a rapid speed, that we slowly start to reach the point where we can rely on software and hardware to do this boring heavy lifting for us.

I shared my view in a previous blog post that knowledge should be a happy by-product of your daily work, however this creates such a vast amount of information that this overload creates a problem on itself. Having intelligent software that can make sense of this, automatically tag it and categorise it, automatically make relationships and assumptions, would dramatically increase our efficiency.

At Headshift we’re doing a lot of research on helping companies staying ahead of the pack and we’ve been looking at software solutions to more intelligently and automatically manage your knowledge as described. It seems that now we finally have the right hardware innovations to make it even juicier…

This post originally appeared on the Headshift blog.

“Content and data are everywhere. People are creating and curating content like never before. As data storage becomes cheaper, businesses are storing, archiving, and mining more data than previously possible. The increasing openness of APIs and data portability make more enterprise data available for both consumers and employees to consume. Free flow of data also allows business partner relationships to be readily analyzed and optimized.” (Emerging Opportunities in Social Business Design)

Filter Failure

With large corporations storing more and more data (be it for compliance, regulatory or internal mining purposes) in their Enterprise 2.0 (and overall IT) systems, we have the danger of getting big data silos or disparate solutions. To make matters worse, they are often stored locally in systems that are owned by different business units with different purposes. So, imagine that you have invested a lot of money and effort in a knowledge management system, just to realize after 3 years that it does not suit your needs anymore and you need something else? If you have a couple of thousands of files, it’s still quite manageable. However, if you work in a very knowledge-intensive organisation, three years of data might have accumulated into several hundreds of gigabytes of data. Good luck with that migration.

Then go through a merger with your competitor or launch a whole load of new products or services and try to gain consensus and consistency across these disparate solutions.

With the increasing importance (and increasing amount) of data floating around your organisation, it becomes more and more important to think about open standards for data interoperability. Accept the reality of the day that a lot of your data is stored in silos. What we need to think of now is how we are going to make this data step-by-step accessible so that we don’t need to do tedious and error-prone data migrations when the system doesn’t cope with our demands anymore.

Perhaps to your surprise, I’d argue that the data silo lock-in is not your biggest problem. No, the inability to intelligently manage and reuse this volume of content in a meaningful way is a much bigger danger that has a direct impact on your business. Filter failure arises when individuals are unable to synthesize and understand the vast amounts of information being generated by an organisation.

Where the problem used to be getting enough information, now it’s being able to make sense of it all. So in addition to filtering the underlying plethora of data and subsequent applications, you also have to be an inline translator. For anyone dealing with end users directly (e.g. a front line customer agent dealing with lots of applications), they will always speak in their own language and never that of your systems, applications or processes. More importantly, they have no reason to.

The interface is the product

But what exactly are we trying to solve here? Why would we even care about this problem? Just a Bunch of Stuff That Happens perfectly coined it in the following cartoon:

Even this is being kind – the average knowledge worker will use between 6 and 15 of these apps, we have experienced people using upwards of 30 because of these data silos. The typical enterprise application looks much like “Your company’s app” as shown in the cartoon. There is such a vast amount of data flowing around your company that you often end up with these kind of user interfaces. Instead of achieving the goal of bringing powerful information to the fingertips of the business end-user, it just confuses people. It just makes people unhappy and unproductive. For every new channel, (email, web, social channels, …) and for every new product, the quick answer is often to bring in a new additional application. This all adds to the complexity and mess on the knowledge workers desktop.

And just in case you would forget, an IBM Design tweet nailed it:

The end-user of your product doesn’t care what kind of data silos are laying underneath your IT system. They just want the information they need to do their work, but very importantly: taking in account the context of the work! Put yourself in the consumer’s shoes, your customer doesn’t talk to you in silos and certainly doesn’t want to be treated in silos. Have you ever called your bank, only to be passed to several different departments? We know already that Interactions aren’t Connected and ultimately we are still running processes as if we are in the industrial age – an assembly line of handoff after handoff.

This is the same as going to a McDonalds and asking for a ‘Happy Meal’ but being told to get a drink from one counter, a sandwich from another, fries from a third and the toy will be sent directly from Mattel – oh and you want a straw, napkins and sauce – there’s self service for that. More of a Meal than Happy! Not quite so fast or convenient food. What happens when the meal then changes, there’s a new toy, you add something else to the box – how is the existing process able to cope with changes easily?

Impact of data ubiquity

We have now identified the impact of data ubiquity:

1. the corporate IT department being challenged by huge data silos (lock-in) and disparate solutions with complex processes that rely on humans to be the integration layer
2. the business end-user dealing with too many different and complex applications and not being able to make sense of all the data (filter failure), subsequently not being able to deliver the process

In many businesses the delivery of an end to end business process relies on users accessing multiple software applications that combine to deliver the complete process. The result of this can be disjointed processes, mistakes, slow access to required information, no single customer view and ultimately a dysfunctional customer experience that the business users can’t impact.

So, the goal we’re trying to achieve is to provide business end-users Enterprise 2.0 systems with meaningful contextual information in a simple and elegant way – we like to call this fit for purpose!

Mashing it together

As a technologist, the first reaction would be to try to solve this data silo problem. (Let’s ignore for a second the old approach to create a monolithic repository, called the black hole, where we dump everything in.) How can we make it more accessible, can we wrap a web services around it, can we apply on a large scale the principles of Service-Oriented Architecture and Model-View-Controller, can we add an open data API interface to it, etc. That will most likely keep us busy for the next coming years. Wake up call: your customers are not going to wait two years till you have your internal issues solved. This approach also often instigates new shadow projects that proclaim to deliver tactical, quick win solutions whilst waiting for the ‘nirvana’.

As a business end-user, you’re faced with a proliferation of applications. A lot of time and money have been invested in building these, so… why not starting by reusing the useful bits of the existing apps to get immediate value?

This is the approach pioneered by Corizon. A user-centered focus approach, starting with the end users and working down – understanding the process in which they go through to complete a task, be it solve a customer enquiry in the front office or manage work in the back office – referred to as the user process. All too often, technology’s answer to a problem is upgrade to the latest version as it has all these new features. The problem with this is that this doesn’t necessarily resolve the original problems, complex process, too many applications.

Once the user process is clearly defined, we then (or can in parallel) look at working up – understanding what data & applications we need access to effectively and efficiently complete the defined user processes. We will now understand the pattern, what gets the most use, by who, how. This firmly puts the cross hairs on which applications to enable for reuse. Unlike traditional data re-use approaches, this approach enables the useful & required bits of applications, but also defines reusable UI and stores these in a library of reusable services.

Now we have two key elements defined, a clear user process and a set of reusable UI services. Corizon’s solution then allows you to mashup these to create the optimal interface, be it a standalone UI or consumed as a widget in your Enterprise 2.0 application. Adding new, or removing legacy applications becomes a much less complex task – the UI Services approach allows you to interchange these without affecting the interface.

Good is good enough

At first sight, this approach might sound a bit unconventional, but we’d like to invite you to an excellent post by Peter Evans-Greenwood (@pevansgreenwood) that talks about “The Price of Regret“.

Building the big, scalable perfect solution in the first place might be more efficient from an engineering point of view. However, if we make the delivery effort so large that we miss the window of opportunity, then we’ve just killed any chance of helping the business to capitalise on the opportunity. … Size the solution to match the business opportunity, and accept that there may need to be some rework in the future. Make the potential need for rework clear to the business so that there are no surprises. Don’t use potential rework in the future as a reason to do nothing. Or to force approval of a strategic infrastructure project which will deliver sometime in the distant future, a future which may never come.

One thing we’ve learned in this consulting business is that most of the times, good is good enough since perfection takes an eternity.