Friday, February 28, 2014

Data Pragmatism

In any kind of business pragmatism demands a balance between quality cost and time. This applies to data management as much as it does to any other type of effort.

When you need statistical significance and to understand trends – it seems almost intuitive that you do not need the highest quality of data. However, one has to be careful. Some dimensions of the quality will be critical to your usage, others would need to comply to some boundary rules and other dimensions might be completely irrelevant. It all depends on the impact, or sensitivity of your measure to those dimensions.

So to be pragmatic, you need to consider these sensitivities and decide what you can, or should accept from a practical perspective. I know that ideally, having beautiful, defect-free data sounds like a dream come true (sorry, but I am a data geek!), but you may have certain objectives which may not care about being data-perfect at all.

For example, consider a sample of data depicting the volume of traffic on a road at various days and times in the week. The precise number of cars may be irrelevant if you are evaluating how well the roads are designed to handle the traffic, or it may be critical, if the road is a toll road and you need to bill the road users. Even then, it may be worth investing in cheaper license plate recognition technology which will help you identify 90% of the road users, than buying and maintaining a system that gives you 99% success rate but costing you a lot more on the long run. It all depends on the practical constraints of your operation.

The larger the data, the more important are the boundaries (or thresholds), rather than the exact value, while for small data, every bit counts (literally).

At the end of the day, what matters is what you do with the data. As the old saying goes: knowledge is power, but how you apply it makes all the difference.

Saturday, February 15, 2014

Why Lack of Data Management is So Pervasive

People invest their time in what they believe is worthwhile. This is an overloaded statement. What is the definition of worthwhile? What is affecting the belief that a certain outcomes will transpire?

A business, or any organization for that matter, is driven to realize a certain vision. When you look at what companies’ state as their mission, you find things like: to be the best..; to serve our clients...; to create...; to change...; it is all about making a difference and effectively dominate the servicing of a need. This is no secret - fulfill a certain need or want and be rewarded in return. This is basic principles of trade.

So then we ask why care about data? Even customer experience is heavily skewed by emotions and perceptions. So how much does the control of the data really matter? The answer is - as much as it affects the fulfillment of the mission.

Usually, new businesses capitalize on being rewarded for delivering to a need that has not been serviced before, or that is in great demand. Therefore the ability to deliver overrides the efficiencies and even quality (to an extent).

As the needs market mature, the organization finds itself crossing in to a different realm where efficiencies and quality becomes more important. New competitors arrive; the market becomes more demanding and so forth.

By the time this happens, the servicing organization has grown to focus mostly on delivering on speed and volume. The size of the organization has increased and priorities have started to shift. This movement increases the entropy of the organization and in most cases the impact on data management is ignored. Attention to management and operational efficiencies enjoy all the attention and poor data becomes an unfortunate reality.

This might be very difficult to avoid, and that is why the lack of data management is so pervasive to begin with.

But why does it linger?

We realize we have a need to improve the data, and we start talking about a data strategy. In the common and unfortunate cases, organizations fall into believing you can fix the data by upgrading systems and cleaning the data. If you've been in the data management business for a bit of time - you know this is not enough.

So what's next? Data governance. Ah..... 99% of your organization runs away. More admin? More boring meetings? We don't have time to talk about the bad data - we just need to fix it.

So here starts the true challenge of data management: Fix the data now, but make it last forever. With no commitments to co-stewardship, make sure it lives across the entire value chain. This probably sounds familiar, but do not despair.

Realization and appropriate investment should prevail. With the growing maturity in data management and in the worthwhileness of common information consciousness, I am confident that the sensitivity and naturalization of data stewardship will prevail.

In the meantime, keep your eyes on the goal but make sure to stay pragmatic so that you remain relevant.