Sunday, June 30, 2013

Don’t forget the data scars

Ever sat in a coffee shop, and looked around you at the people sitting at other tables, or perhaps at those who are walking by? You might, in fact, be doing that right now. You might be at your work desk, walking somewhere, or sitting on a bus or train.

What do you notice? You probably take a quick measure of people’s physical build, their likely state of mind (happy, upset, stressed or annoyed) and maybe take notice of their choice in fashion. You might go as far as trying to guess whether they are well-off, whether they are local or tourists, or even how different or similar their journey to this point in time and space might be to yours.

One lesson I have learned in life, is that you can never know where a person has come from, and where their bound. To illustrate this point, just imagine that person you are looking at, is some undercover secret agent, who is in the middle of a secret mission to save the world…

But there is more. What I really would like to draw your attention is to the signature that time bestows upon us. In simple terms – the scars. You might notice a scar on someone’s arm which is a memory of a physical injury or streaks of white hair or wrinkles that are the scars of time on people’s worldly vessel. This would probably make you think of the person’s longer, and significant history in life. Not only what their immediate journey might be, but what hardships and joys they might have experienced during their life.

These scars are a form of metadata about people. Did you ever stop to think about that? What about the fact that data has scars of its own history? Sure, data cannot be joyous or experience pain, but the way it is handled can affect its appearance, much like the scars in people. Take for example, two reports that show financial numbers. One report has many suffixes such as 8.99 or 16.99 etc. while the other set of numbers are fully rounded (e.g. 9.00 and 17.00). Now if I told you one report is in aid of budget planning, and the other is for product price tagging – chances are you will know which list is which. You will still need to verify it – but you can advance an assumption out of the data scar. When all the financial numbers are rounded in a certain pattern – it gives you a clue. When correlation is apparent between parameters of values – there is a historical event in the life of the data that caused it to appear the way it does.

When you are looking at the data, and you cannot find the answers you are looking for. Try looking deeper, in to the scars of data, therein may lies the story which can give you the answers you are looking for. It’s like reading between the lines of a letter, or a note. Don’t get discouraged by massive amount of seemingly meaningless data. If there is a story behind those numbers, the scars will help you find it.

So good luck with your data analysis, and don’t forget the data scars.