Data Non-Sense

There is a certain false premise in artificial intelligence, business intelligence, big data and smart technologies. We want to believe that these will lessen our burden, maybe even our responsibilities. We will develop algorithms that will solve all our problems and technology will take care of everything for us. That is true to an extent, but we have to be very very careful.

I'm sure you sent an email with an embarrassing auto correction. Mobile devices are known to lead to funny ones in peculiarity (in particular). The point is that while a wrong auto correction might often lead to nothing more than a smile, it's a whole different story when we talk about decision making that impact money, health and life.

It's the age old story of would we let robots choose who will live and who will die, but there's a long gray area in between pen and paper and machine world domination. It boils down to assumptions, ability to adapt to change and nurturing insight from data. None of which depends on technology, but rather on human ingenuity.

Some people say we should fear that machines will take over our jobs, that humans will be perceived as a waste of precious natural resources. But machines cannot harness inspiration. They cannot fault through passion to reveal unknowns and power discovery. Behind every breakthrough and significant leap, there is a human mind that by intention or by mistake, and through a maze of circumstances, broke a path to something truly unique.

This is the romantic tale of brain over processor, emotion over logic, and human over machine.

While technology can do good and bring harm, data itself has no sense. Context, passion, and sparks of ingenuity are what will propel us forward.

Where are my Keys?

We all apply a different degree of focus depending on our priorities. We come home in a rush, put down our keys while speaking on the phone and twenty minutes later we can't find them anymore.

This is typical, but can be avoided. There are things you can get and do to avoid or find your keys more easily. You can buy a key ring that when you whistle it makes a sound. You can train your brain to always put your keys in the same place. You can even keep your house so neat that the keys would stick out as out of place. The common theme though - is that they require planning and effort to achieve.

The same idea applies to your information. From: where is that bill from the Dr? to: who did I send a copy of my ID to? Without a system to keep track and stay organized - chances are that things get lost. This is why they send you reminders, and this is why fraud is more prevalent than it could be.

I'm not saying that you should stop losing your keys or never ask to be reminded of a bill, but rather ask yourself: what are my priorities? What about the different type of information would you like to manage better? You might be happy the ways things are, or you might sigh at the thought misplaced information.

Either way, I recommend that you: list the important types of information; plan how you are going to manage it; discuss this with those who you partner to manage the information; and make sure you review how it success over time so you can tweak and improve. This way you will only lose your keys when your careless unreliable cousin asks to borrow your car...

10 years since I wrote my article providing a blueprint for the role of a Chief Data Officer

Find the link on my tweet:

https://twitter.com/gmarcan/status/1038086264292106241

Schema of Life

We communicate using semantics, in other words - a language. It has repeating patterns and rules which enable us to create familiarity that translates to meaning and leads to understanding. We learn languages by exposure to the patterns and rules and experimenting until the useful level of understanding is achieved. I am not a linguistic expert of any sort, and I am sure my statement is not exactly what the textbooks would use to define a language. I apologize for that. Nonetheless, I would say it is a reasonable framing of what constitutes a means of communication.


Now without language, there is no communication. Without communication there is isolation and disconnect. We spend a great deal of resources to develop, compare and interpret information. However, when we exchange structured information, we often spend very little resources on ensuring the information we exchange is well understood and exists within a clearly understood contract.


When we sign up to exchange data, we often focus on the channel rather than the semantics. We will agree when to deliver information, where and how. We will also agree on the scope. For example, every Friday, we will get a export of all the new customers in a spreadsheet delivered to this xyz server. We will agree on the format, which will allow us to extract meaning from the data and also on the usage, provided there is sufficient risk or value embedded in the data. Notice however, that semantics has a limited presence in this definition.


Things get a bit better when the provider gives you a set of definitions, which is a portion of their semantic definition of the data. However, this definition is often riddled in several ways. Firstly, it is likely that it was partially defined, since the context of the contract is limited to a certain business activity. It would be rare for an organization to provide a definition based on a mature internal semantics language. Secondly, the domain of your business is at least slightly different from your provider and your definition of a product is not necessarily the same as your provider (and so are many more definitions). Lastly, your business is not likely to hold a mature semantics language for the same reasons your provider does not (there is little direct profit value out of such activities).


Now, this is the pit fall. When the semantics differ, and dissemination of the semantics is poor, you end up with augmented meanings and improper use of the data. It takes teams of knowledge workers to daily address complaints and quality issues, which we often blame on our provider or "bugs" in our systems.


To be clear, I am not pointing to any specific organization I have worked with, or for before. There is not a single institution I have ever come across which does not exhibit this phenomena. Nonetheless, this is simply a clear indication of a low maturity of data management.


So, what do we do?


Well, everything is driven by value or perceived value. There is a price tag to these activities. You would need to imagine a world without these issues first, and imagine the efficiency and the opportunity that come along with effective data management. Then you need to draw attention to the vision. In other words - market it. Finally, you need to work with the leaders of the organization to consciously integrate design and operational behavior changes to reduce ambiguity and create better semantics harmony. Internally and across organizations.


Ultimately, this will lead to a data exchange language which will allow us all to communicate and respond more effectively. So instead of having to ask questions about people meant by "date originated" or "number of irregular accounts", we can focus on value enhancements and product development rather than reactive and corrective behavior.

The Return to Diversity: The Unexpected Outcome of Global Data Evolution

Back in the days, information was scarce. It took a long time for information to travel from its source to its consumers. Over time we have seen technological breakthroughs, starting from the print press and moving on to digital media to the internet and social media. With these advancements our ability to access information has increased. Information travels much faster and the number of sources available has become abundant.

However, it has also become increasingly difficult to verify the sources of information. It has become much cheaper to create content and to publish it to the outside world. Therefore, anyone with any intention can find innovative ways to publish their content in a convincing manner. As a result,  it is much harder to be certain that the information you are consuming is in fact the whole truth and in fact the perspective you are looking for.

For this reason, we are now in an era where people are starting to diverge to "content groups" based on the concentration of specific sources aligned with a particular view of the world. While you will always have people who crossover between varying sources of information, the majority of people will stick to a set of sources that align with their education, background and experiences. So Instead of the free flow of information bringing us closer together, it is actually creating a wider reap and division between groups. This will deepen gaps between geographical areas, ethnic groups, languages and more.

This will eventually result in an increase in the diversity of perspectives, culture and behavior across the globe. However, while there are negative consequences in diversity, there are also benefits. One major advantage is the fact that with diversity comes strength. More perspectives means more and varied ideas and approaches to solve problems and innovate.

We must therefore challenge ourselves to firstly recognize that this pattern in the evolution of information commodity is a reality. Secondly we should find ways to use this phenomenon to help us maximize the return of our goals and objectives. You only need to look at recent politics in the U.S. to realize how politicians and corporations use this to their advantage.

The other question remaining is: where will this lead us? Are we going to see more diversity and more drift between groups in human society? Are we going to see a deliberate growth in diversity with an underline common set of core values?

How we respond to this change will drive and determine the evolution of mankind. There is little that an individual or a small group can do to control this. However, as a society as a whole, we can and should develop a global data governance framework that will look at data holistically across all domains and across all social and economic activity. This framework will enable us to drive this diversity to ensure a common set of values are protected. These core shared values will help support the ultimate goal of sustainability and evolution of our specie. Otherwise, we are basically taking a potentially irreversible chance with the future of mankind.

Implementing Data Sharing in a Multi-Stakeholder Ecosystem

Information flow is complex, but it does not seem so for most of the stakeholders involved. At a high level, data flows from the originator and passes through various data handlers and eventually reaches the data consumers. Now this would have been complex enough had the data remained in its original packaging, but we know, data is re-organized, filtered, translated and aggregated. This affects the roles and responsibilities of each of the links in the flow of data. Therefore, it is crucial to understand the implication of these processing points and to amend the contract that is attached to the data being processed.

This contract needs to define the meaning of the data, its origination, the constraints imposed by its originator (which need to include the data owners' rights) and the scope, or conditions, which apply to the data. This could be implemented in various ways, but should not be locked into a single medium or format, since most data can be transported over various mechanism and the contract would be relevant regardless of the mode of storage or representation.

To provide an optimal control over data, you need to consider several elements:

1. Holistic flow chart: starting from origination and extending through the data flow's life cycle as far as possible from a practical and risk/value proposition perspective.
2. Governance body: together with the stakeholders who manage the links in the data flow, determine the policy and processes to follow to ensure initiation, use and retirement of data. This would include everything from quality control, issue resolution, data life management and related responsibilities.
3.  Internal governance controls: develop measures and processes to ensure compliance with the data ecosystem policy, while ensuring compliance with related policies around the internal business components which handle the data. For example: while you need to ensure you keep customer data for as long as it legally permissible, you also need to consider whether keeping it for that long serves a purpose and value to the business (as well a cost of maintenance and prolonging of handling risk)

The point is that there is an important thread for information handling, which is often ignored, and is often the source of risk exposure, conflicts, misunderstanding and a barrier for value enhancement. This thread is the need to consider data in an EXTERNAL ecosystem. Most data is not isolated to your business. It co-exists with customers, vendors, policy makers and others. To succeed in this challenge, one needs to stitch a business vertical ecosystem with a horizontal data life ecosystem. A significant portion of this horizontal ecosystem exists outside your business and control, and the challenge is to accept this, identify the risk and opportunities within this fluid position and create and govern the right mechanism to maximize the benefits (short and long term) for your business. 

True Homogeneous Data Management

The universe is bound by laws which create harmony and enable us to innovate and create more complex and advanced structures. Those in turn, allow us to explore more, and to create new and better experiences. I believe information has a similar potential, that if harmonized and trivialized - would lead to products and services we could barely start to imagine today.

Imagine a world where access to precise and complete information is the basic premise of innovation. We no longer try to improve the data quality, but rather focus on new ways to create new products and services that allow us to protect and enhance our world, knowing that all the information we need will be available when we need it. Every piece of data that is generated is naturally appended to a global system that allows instant collaboration according to the rules governed by the same leadership that looks after our social footprint.

The challenge is no small feat. The information era in which we live today, is blinding with an assortment of sources, frameworks and consumers. Like the story of the tower of babel: we have limited our own capacity to truly harness global information - since everyone is doing things their own way. All these different semantics, regulations and technologies result in obscure harmonization.

We only need to look to recent years to see how governance and ethical issues, ranging from misuse to misdirection, lead to outcries and painful changes in global maturity in data management.

We have an incredible ability to transform information into real-time, space-independent commodity. We can create models and harness technology to store, access, analyze and present information in any imaginable way.

Yet it seems we lack focus in aligning ourselves towards a framework that would lead us to the dawn on universal laws for information handling. I believe it is possible to carve the path towards transforming information handling into a global enabler that will allow us to explore more, and to create new and better experiences.

To achieve this goal would require intentional effort from organizations and bodies that have a significant influential role in managing data globally as well as social and regional leadership buy-ins. Think of the OSI model that defines the way the Internet works. We need almost something similar to enable global information management, while not crippling freedom of expression and governance.

I see this more of a responsibility, rather than an opportunity. While there are obvious economic opportunities here too, the true value is in the enabling a new kind of environment for information currency.

We need to create a trusted and resilient entity that can prioritize and drive the realization of this vision. We would need to consider regulations, disruptors, economics, complexity and many other factors. Therefore a blue print and road map is needed. Starting from a manifesto and concluding with a realistic plan to lead this shift to fruition.

Why is Information Handled the Wrong Way

Information Technology is a field of applying the science of information handling to man-made tools to ease and enhance the life of people.

However, somehow, it seems we are missing the mark. While there are great achievements and value-add through information technology, there are concerns on its true value due to risks stemming from fundamental flaws in the practice of handling information.

It ranges from risks of inaccuracies to information abuse seen as privacy and ownership concerns to the public's eye. These are due to either ignorance or malicious intent. Whether it is over exposing protected data or creating false representation on reality. Either way, this rough use of technology is decaying, rather then enhancing, people's quality of life.

We can, and should, be appreciative and grateful for the excellent abilities and tools we have today. Some of which we take for granted. But we have lost direction as indicated by lower trust levels and stronger cries for legislation.

The fundamental problem is that the vision for these technological innovations does not focus on benefiting human life as much as they are about demonstrating stronger capabilities to eliminate effort and control of people over information. This has the additional unsettling consequence of a narrowing set of entities which are able to control and possibly manipulate information.

In order to move in the right direction, every technology built must consider, at design time, the requirements and implications to ALL stakeholders of the information being handled by the technology.

To discern the issue of poor information handing we must adopt an open and inclusive framework which, by design, shifts and retains the power of decisions to the entities who should own and impact the information based on ethical and moral principles.

This is not only the right thing to do, but also the best long term economic strategy for information technology!

The Data Economy of the Future

Data handling today is fundamentally flawed. Information is traded as a commodity and not as an asset which belongs to its owner. This leads to a basic erosion of its value and to the ability to handle data effectively.

How is data value being eroded today?

Data is traded today as a finite commodity which can be used as raw material to develop products and services which provides added-value to end users. Whether it is the aggregation of sources against a single entity to create in-depth perspectives, or the aggregation of data to get insight into a specific population. The data is acquired, cleansed / prepared , matched / assembled , enriched, analyzed and reported / crafted to meet the needs of customers.

However, access and use of the data is riddled with rules and regulations. There are liabilities associated with sensitive data, restrictions linked to privacy and terms of use of almost any data. Every product, depending on the data classification and the purpose of use - needs to be carefully placed in an ecosystem of custodians and controls. This puts a heavy toll on managing data, and despite of the best efforts - it does not give the data owners the sense of comfort that their data is under their control. Why? because it is NOT under their control. This is the erosion of data value. 

Since the owner has limited and sometimes no control over their data - they are inhibited from sharing data, and only allow the data to be used if they believe the value they would gain from sharing the data is greater than the risk they are taking in sharing it. This effectively reduces the ability of businesses to access, learn, innovate and generate greater value out of information, that is generated all the time.

What is the right way to harness data?

The cause of the risk/value imbalance is the fundamental in the premise that data may be governed by others, and that your interest must be taken into account but only as an afterthought. Through time, law and regulations evolve to protect your data. but by then, your understanding of the risk and value has also grown, as well as your reservations and control mechanisms to protect it.

To eliminate the imbalance and create a pro-sharing data economy, we must change the basic rules of information trading: You may access my data on my terms ONLY. This may seem extreme, but think of the implication: the owner has no inhibition to share their data. They can revoke and withdraw your right to access it (within the legal framework) - but when handled correctly - you have the freedom to gain deeper and wider insights. Governance becomes trivial and controls - obsolete.

How do we create the future economy of data?

Owners must have a platform or an agent which allows them to expose or revoke data. This must happen seamlessly and effectively across all industries, products and services where their data is provisioned. This also means you cannot offer a data-based product or service without the guaranteed control to the data owners that allows them to control its use.

Think about harnessing medical records with travel information to identify products and services. Think about precision marketing based on comprehensive in-depth knowledge of the consumer's habits,  measures and purchasing history. There are essentially endless applications. There is value for the owner as well as to the end user.

Data Mirages

There are all kinds of illusions in the world. Those that are orchestrated by humans (magic) and those enforced by the laws of nature (mirages). These both hold a common theme of creating a perception that something we think is true - is in fact false.

This can play a significant role in information management. It can affect how data is being accessed, what its true quality is, in terms of its intended usage, and it ultimately impacts how data is being governed.

Think about steganography. While encryption is an explicit way of hiding information, steganography does not tell you that information is being hidden. This gives you the illusion that there is no more that what you see, when in fact there is a hidden message. Only the people who know about the hidden information are likely to know how, and successfully extract it from the concealing medium.

A data mirage, however, is more a matter of opinion, and what I mean by opinion - is perception. What appears to one party as an accurate and complete account of an observation, may in fact be partial in the point of view of someone else. Like any natural mirage, this “opinion” is circumstantial and will depend on various “natural” factors such as different point-of-views, the taxonomy gap between topics, inconsistencies of data quality standards and the differences in objectives between the parties involved in the information exchange.

To manage the risk of “seeing” a data mirage, make sure you understand the differences between your language and that of your partner you are communicate with (think about: knowing your audience) ; ensure your service level agreements, or expectations, are explicit, not only in terms of the protocol being used but also in terms of the quality of the information as it relates to what is being measured and how;  Finally, gain an understanding of what are the priorities of the other party you are communicating with, and what might be concealed from you, either intentionally or inadvertently.

Have you identified all the data mirages in your world? and are you sure you are truly separating data facts from data fiction?

Out of the Box Data Governance

To implement effective data governance you need to think out of the box. I don't mean just being creative and finding new innovative ways of doing things, which by its own right - is great. Rather, I am referring to thinking out of the box of your responsibilities. You have a certain responsibility to look after the data under your custodianship - and that can easily be blurred by your team's performance indicators.

We all have a role in data governance, because we all manage data. Whether you direct teams that implement solutions, manage resources that operate solutions, build new solutions or ensure solutions are operating as needed on a daily basis. What you do, and how you evaluate your success has a direct impact on the fitness of the data for its usage. If your team's goals are not fully aligned to serve the intended usage of the data - your priorities will not best-serve the effective and efficient usage of the data.

Take for example, a business that sells products under warranties. The manufacturer has an interest in knowing who actually purchased the product, but the retailer might only care about sales and customer loyalty. The retailer will prioritize in-shop experience, products quality, variety and pricing which would make it harder for the manufacturer to capture accurate data on the end consumers. To address this challenge manufactures learned to rely on end users to provide purchase information as a means of maintaining the important connection between them and the consumer. This only works, if the consumer sees value in registering their product. In other instances, the manufacturer would have to depend on the retailer to collect this type of information. The only way this will work is if the manufacturer provides a benefit to the retailer for collecting this information on their behalf. What we see in this second case is the manufacturer influencing the responsibilities by aligning their information needs with their partner's objectives.

In any data handling operation, the level of fitness of the data for its intended usage is directly dependent on the knowledge workers ability and motivation to support this usage. This is why data governance is important, and this is why understanding the context of the data from both a consumer and a manufacturer perspective is important.

If your knowledge workers think only inside the box, either due to lack of motivation or constraints of your business operating model, ask yourself if you are really delivering the value proposition your information handling is offering? If you cannot even answer this question, maybe it is time to think about how you measure the success of your information handling processes, and what you need to do to get people to think outside the box.

The Minister for Information Affairs

Governments have wisely coined the term "minister of communication". Whether or not the mandate covers all aspects related to information management - is a separate issue, which relates to politics, semantics and priorities. None the less, in my opinion, with that term in place - information management should  certainly  be part of this minster's portfolio.

But seeing that companies share a lot of similarities with governments in terms of having to control a large pool of resources, products and services  - it does beg the question: where is the minister of communication, or a "CCO" for companies?

Yes, we all know that the CIO, by definition should cover the responsibilities of Information Management, but we also know that this role is often executed as a pure CTO role with the Information Management piece disappearing in good intention.

Even in governments, the Information Management responsibilities are scattered across  health and safety, security, internal affairs, finance and so forth. This is natural and should actually not be restricted. Every area in an entity needs to have freedom to manage information.

While the role of the minster of communication might extend to information management,the focus there needs to be on governance rather than implementation. The role of the CIO in terms of the same portfolio needs to be executed in the same manner. The problem is that information is increasingly embedded in technology and as the power of the CTO grows in terms of being able to influence how information is practically managed, the segregation of duties in terms of information governance needs to grow. This does not only help the CTO focus on their domain and on servicing the business, but also nurtures a healthier and better trusted framework to manage information.

To put it bluntly - Information Management should not be the responsibility of the same person who is in charge of the tools used to control the information. An additional benefits to this is improving the focus on resources within the data management  domain on the collaboration and information exchange needs, rather than on the economy of information handling.

Ask your self: who is looking after information management in your organization? by definition, and in reality?

Max in / Max out - The Marshal Art of Managing Data

In order to be effective, a data strategy need to be implemented correctly at a micro level, that is at a field definition and value validation level. At the same time, for the strategy to be effective it needs to be implemented correctly at a macro level. Being able to switch between those two perspectives - is crucial for the success of the strategy.

This applies not only to the data storage design, business rules and view points designed to support decision making, but also to other dimensions of data management, including governance, quality control and meta-data management (no name a couple). This may sound obvious in theory, but from an implementation perspectives - the challenges are countless. You need to worry about macro issues such as business and technology strategy alignment and internal politics to micro level issues such as resource prioritization and technical.

How do you then navigate these rough waters to reach the shores of success? As the title of this post suggests - learn to max in / max out in terms of your influence of the implementation. To steer the strategy correctly, you need to consider your macro influences, and when the need arises, dive-in to the detail to ensure implementation guidelines are followed sensibly. Ideally, if your work focuses on the detailed implementation, aside from your detailed execution, you need to "jump-out" and be able to "step back" and look for the value proposition of your implementation from both a business and a data strategy.

To "Step back" you need to ask questions such as: does this storage design make sense in terms of being able to expand the company's products according to our strategy? (of course you need to know what is first); Are the business rules defined to filter, validate and govern the data in place? do they make sense in terms of our business model? in terms of the value proposition of our products? (of course you need to know what the business model and value proposition of the products are first).

To "Dive-in", ask your implementers to demonstrate examples that directly contribute to the benefits of the strategy. Ask them to quantify those, not in terms of money, but in terms of impact. For example: by applying a date validation on the transaction record we are able to reduce invalid dates which in turn provides us a more accurate view on the periodic sales amounts and hence allow us to better understand how our products are preforming. It also increases our accuracy in financial and regulatory reporting. Then demonstrate the value by showing a metric. For example: after the initial application of the validation rule, we were able to increase transaction date accuracy by 20%, which resulted in 5% increase in correct period reporting... and by the way, we were able to identify inefficiencies by isolating the specific cause of some of those invalid dates.

The art of max in / max out, which can be analogous to zooming in / out of a picture can go along way, but can be hard to master. To complete the analogy, consider a famous painting. From a distance it has its meaning and its beauty. From a close-up one can appreciate the craftsmanship and complexity.

I argue that your data management implementation is only as good as the accumulation of implementers and guiding strategists throughout the history of your business. Are you employing the right mix of people to deliver and guide a high-quality data strategy implementation? and are the able to max in / max out effectively?  

Lost Precision in Dataminea

Things are looking good, they say... Our data is getting more structured, and data mining has become a standard working tool in the business toolbox. We can analyse our customers, their behavior, the markets in which we operate and our own supply-chain environment.

With all this apparent maturity - you would think we are in a good place.

Initially we gave away a lot of our data without realizing it. Now, at least, we are aware of what data we share and kind-of how it is used. It sounds sensible and fair and for the most part of it - this is true.

The danger we have opened ourselves to, however, is an increased sensitivity to information misconceptions. While we may have increased our precision in representing data, we have insufficient tools to control its accuracy.

Before going further, I think it is important to highlight the distinction between the two. Precision, refers to the ability to generate results which are consistent and repeatable - meaning that our tool is reliable in generating the same result over and over again. Accuracy, on the other hand, is how close the measurement is to the actual truth.

Now as I have noted in the past, the truth can be perceived from different points of view, and while we may be able to generate more reliable results, they tend to serve a limited set of perspectives. This is no accident, as these views are used to satisfy specific measures and drive  specific behavior. This is not new. Politicians, advertisers and a lot of other groups and individuals continue to use this ability to distort the view on certain realities and create an arbitrage in opinion to their advantage.

Some may call this the art of doing business, and perhaps that is what it is.

The bottom line is, however, that with all this Dataminea going on around us, we are becoming much more sensitive to data miss-representation which can be a good or bad thing - depending what you are trying to achieve, and how you manage your data.

My question to you is: do you understand the perspectives and the level of precision of the data you handle?

Price Tagging Data

How much is data worth? Is it based on how rare it is? how hard it was to obtain? how it serves the business opportunities or risk of those who buy it? If other people sell your data, shouldn’t you get a cut? what should that be? How do you price tag data?

 

The truth is that this is like any other economy of trade. As a producer  of data, you would be interested in making a profit, and hence will optimize your costs and look to price your data at a level that will fit the market in which you operate. As a consumer, you will be willing to pay based on the profit you believe the information is likely to deliver for you, coupled with the market conditions for obtaining the information. This of course is bundled with quality and timing.

How is the price of data affected differently than other commodities? Timing, for one, is a big factor. When access to certain information becomes pervasive - the cost of the information diminishes dramatically. No one will pay to know what the name of our planet is. Start talking about very old information, which is no longer pervasive- and the price will start going up. You may want to pay someone to know certain facts about the history within a particular region. Although the underlying value is the access to information, the influencing criteria is time.

Quality is probably the second point worth mentioning. While I do not see a major difference in the impact of quality of data on the value and the cost - the objective valuation of the quality of data is harder to achieve than in other commodities. Many products are evaluated for quality over a clear a predefined criteria. Gold has carets, cars have performance indicators and design styles, services have customer satisfaction and food has taste and presentation. Quality of information not only mean different things to different people, businesses define data quality differently, depending on how they intend to use it. This dual quality obscureness of both people and organization leads to an incoherent set of standards to which to value information against. While it could be plotted on a 2 dimensional surface - in reality people tend to draw subjective conclusions on the quality of the data - and it almost becomes a matter of taste rather than a skilled practice.

So while experience counts, and some data will prove itself more valuable than other. Are we simply in an open marketplace for information exchange? without regulation for each data type, we will continue to buy data fruits while being blindfolded. While everyone else is doing this - I guess it is fair game. But I am not too sure that is the case.

How do you, or would you, put a price on the data you manage?

Good Design or Fast Results? The Truth? Learn to Manage Expectations

I write about data management, but this topic is deliberately none data specific. It actually applies to many aspects of one's work and life - but since I am a data geek - the content and examples will focus on data. You can extrapolate and draw behavioral conclusions on other domains of productivity.

Let's get started. Some people would say that a good design is imperative for success, and sufficient time should be spent on that phase of the project. They are of course - right. Other people say you need to deliver results, respond to market changes and generate revenue and contain costs, otherwise the business does not survive. These people are of course - also right. So how can both groups be right? The answer is balance.

One of the most valuable lessons you will learn (if you have not done so already) is that balance is as important as substance. What I mean by this is that there is an optimal point, where over-engineering becomes a hindrance, while lack of design can lead to unacceptable results. We will get to data in a minute, but my favorite example on this life-lesson is from civil engineering: suppose you are tasked to build a bridge across a river. You need it long enough, but not too long. If you measure your materials to the millimeter - chances are you are over-engineering the process and will spend too much time and money getting the exact "right" measure for your bridge. On the other hand, if you just grab a few planks of wood and start building, not only do you have a risk of bridge instability, you might actually have too little wood, or too much left over. In all cases, you will probably need to extend the time and effort to complete the bridge in order to compensate for these shortcomings. What you need is a balance: make a "good-enough" plan quickly enough, and then build a bridge correctly with optimal time and effort.

Now let's talk data management. One of the typical problems in today's information chaos age is the need to build or upgrade the train bridge while the train is already speeding along the tracks. Another well-known metaphor is building a plane while it is already in flight. The overwhelming criteria in this type of operation is stability. As long as we do not lose business or cause harm. This often leads to patches and workarounds and compromises in good design which eventually leads to issues in costs and strategic alignment. Raise your hand if you felt frustrated on not being able to implement a good design, because of time pressure or concerns about systemic risk. For example, not only is it expensive to change a data model (migration costs will make your executives cringe), but there often seems to be limited foresight as to the strategic benefits.

This is when the conversation needs to shift gears, and the balance needs to be struck by managing expectations. Understand where the business is going, and agree what is best from a technology and design perspective. Then start creating a balance by managing changes such that new parts are created ahead of time, and then brought to the bridge when opportunity allows it. Justify the "extra" efforts by identifying and quantifying the architectural benefits (in other words what you stand to gain or lose in the longer term). You may conclude that your data model should not change right now, but every time it changes it does so in a deliberate direction. It will also help you to understand its limitations and sensitivities better which helps with systemic risk management. The notion of managing expectations and creating a common architectural vision will also funnel independent views of resources in the business to work together under a joint effort, and will allow you to harness many minds to improve progressing in a common and agreed direction.

Now tell me, are you working towards implementing a strategic data management plan, or just doing localized data fire-fighting?

Motivating Data

What motivates people? A belief, a hope, a goal. The only thing that makes you read this is the belief or hope of finding something useful here. Something you can learn from, quench your curiosity or help you reach a certain goal in learning or performing.

What is motivating data? Well, data makes no decisions, and cannot apply any resources to a particular action. So, the statement is an absolute absurdity. Or not ... While data cannot change the way it interacts with the world - people certainly can affect this. And that is the point.

When I first started writing about data management, yes the notion of people affecting how data is applied seemed very trivial. Of course people manage data, and of course we affect how it is being applied. But what I am realizing more and more is how pervasive the subjective motivation of data really is. Every day, all of us, make dozens of decisions to disseminate or block information from flowing. Whether it is of personal nature, such as protecting loved ones from anxiety or pain, or professionally, where we use our professional judgment to improve the outcome of our efforts and those in our teams.

On a larger scale, companies and organizations make conscientious decisions as to what information to expose, when and to whom. This is really part of doing business, or interacting with the world. There are many strategies one can apply to affect these type of results. You can provide too much information with the intent to overwhelm, or create an impression of sharing everything, while carefully omitting certain bits of information. Whether it is right or wrong depends on the situation and the parties involved.

On the other side of the coin, we are information seekers. We look for information that can help us satisfy our beliefs, hopes and goals. We subscribe to channels of information in hope to receive the information we seek. We continuously fine-tune these subscriptions, replace or change the optionality’s on those channels. But in reference to what I said above - this is a tall order, as without understanding your information provider's intention - you are subject to their information filters.

Strategies to combat disinformation of that sort, include evaluating consistency and patterns related to the details of information you receive (sounds complicated, but we do this all the time naturally). A second strategy involves subscribing to multiple channels in hope of verification, or ability to have a more comprehensive perspective.

As a small bolt in the global information system, we can originate or terminate information flows by applying some of the strategies I noted above. We also need to motivate other people to behave the same in order to reach a level of meaningful influence.

My question to you is how are you motivating "your" data, and more importantly - what is your intention?

The Data Ocean

Are we ever going to get tired of comparing data to water? We heard of trickling data, data waterfalls, now people are talking about data lakes and guess what – I am going to talk about a data ocean. Yes, I am referring to the notion of the largest bodies of data in the world, comprising of a multitude of sources and consumers across many, many data domains.  But what I really want to focus on is what these oceans currently look like, what they will look like in the future, and how we should prepare ourselves to maximize their value.
 

What makes water so powerful is its combined force, its chemistry and its consistent and predictable behavior. Our current way of handling data is more like trying to mix water with oil, mud, rocks, milk, sand and lots of other stuff. That is far from the elegant nature of water.
 

While each entity in the world feels the need to derive its own chemistry of data, the truth is that the nature of data is as pure as water. We perceive data as murky and hence treat it as such. Therefore it molds, by our own actions, and becomes difficult to manage.
 

What am I saying? We have no common (agreed) perceptions, models or governance on data. As our data management models mature, we will see more harmony in data.  The essence of the simplicity in data has always been there and the notion of a data lake, and a data ocean, never caught me by surprise.
 

To truly “see” these bodies of data, we need to ensure we are able to view them as such. True “global” harmonization of perceptions on datasets is key to drive governance, management and hence data chemistry.
 

The sooner and broader you can tune in your organization and your business partners to mature and harmonize data perceptions – the better prepared you will be for Teneo Vulgo.
 

I predict a world where ALL data flows through a central data delivery framework, probably centered around a few major providers. In this world entities which have prepared and invested in orchestration power over data will hold the advantage. This is not a world where data is fair (when has it ever been fair), but rather a world where data is managed to serve those who have the strongest ability to influence and exploit it.
 

Think about data tsunamis, data storms as well as data seaside holiday homes, data sanitation systems and data feeding into our daily life. The power of information is only starting to emerge. For me, the famous saying comes to mind: may you live interesting times…

The Sixth Data Sense

Initially, I thought of writing about the importance of our ability to sense, or be receptive to information, but then it occurred to me, that I could also take this further and break to concept into 6 dimensions of "data sensory" categories. So here goes:

Our data sense is our ability to pick up data signals, filter and analyze them into meaningful information. While all our human senses do this all the time biologically in our body, there is a relevant analogy here to the "business body".

Companies are often compared to a human organism. We talk about the ability of various units within a company to communicate with each other; their ability to respond to change, and how their internal design and components affect their effectiveness and consumption of resources.

Similarly, we can look at the "data sensitivity" of a company as a fundamental sensory capability: The effectiveness of the company to absorb and integrate information into their processes - undoubtedly affects their ability to cope with its environment. So we can talk about over sensitivity (information overload), information deficiencies (disabilities in information processing) as well as information blindness (which may sometimes be aided by information sight-aiding tools, such as consultants and investors).

This now leads us to a natural need to define a set of categories which can be used to assess how "data blind" (or not) your business might be?

1. Information Blindness: The company is unable to translate information from its environment into "normal" response to stimulus. There are a lot of aspects to consider in this definition, starting from where to failure occurs (from leadership to field agents) to possible broken means of integrating data into decision making.

2. Information blind spots: The company is able to absorb and process some data, but is acutely deficient in processing certain types of information, or respond to certain environmental conditions. Again, there are a multitude of factors that may lead to this condition.

3. Information blurriness: Information is absorbed, analyzed and translated to responses, but the details are inaccurate, and the actions lead to an over, or under estimation of the required behavior. Here you could expect either an issue in correctly translating data into information, or flawed decision making processes, which might allow for too much subjectivity or restrictions in response.

4. Information vision delays: Here we are looking at time-to-market of information. How old is the data being used for decisions? How well is the company adapting to the information it has received? You can expect a large and "clunky" company to struggle in this category.

5. Information distortion: Information is misrepresented to the business, or the business is responding to the information through an incorrect interpretation of data. This is an issue with the way data is filtered and/or delivered to the business.

6. Information sensitivity: This category looks at how expensive the sourcing and integration of information is to the company. If the costs are too high, and the results are poor - we have a condition of low sensitivity. If the company consumes too many sources, and struggles to respond consistently to its environment, we may be dealing with information over-sensitivity.

These definitions, while presented here to stimulate thought, could in fact prove helpful in characterizing information sensory issues and their causes, which when treated correctly, could help your business succeed.

So what are you waiting for? Why don't you give your business a data vision assessment based on the categories above?

The Internet of Things? No, of Attention

The technology is here. It has been here for a while. Ever since the first microchip was built - you could, in principle, buy the right components and "smart up" your life. But we never really went that extra mile. Why? because it was not practical, economic or really applicable to our lives.

What has changed since those days? the economy of technology. Processing power has become so cheep, you can install it in throw-away items. In addition, our lives have grown to depend heavily on technology. You cannot move, without checking your e-mail, social media, or tracking your pulse, or Geo-location or calorie-burn rate. We have become so accustomed to digitizing everything - that it has become second nature, if not first.

So the internet of things, as coined by some big companies out there, is a slogan that, to my understanding, relates to the ability to "smart-up" devices and consumer needs that have been more remote from the internet and from the automated integration into other consumer services. Now while a completely agree that we can, are, and will continue to "Smart-up" our "technologies" - I am more skeptical on the true value-maturation related to the consumer services aspect.

Time for an example. Suppose you buy a smart refrigerator, that can tell you that your low on milk, or out of eggs. Sounds amazing. Check your phone while you are doing your shopping, and voilà - you know what you need to buy. Really?  Here is the problem I have with this proposition: prioritization. How many communication channels do you have open (types of information sources)? How many content items (messages, articles, tweets etc.) do you follow? There is no point avoiding the "elephant in the room" - your attention has become expensive. This is probably the highest-value commodity of modern times. Example: when you get the alert to get the milk - you would be the perfect marketing candidate for a discounts and product information related to milk and milk products. Why would you install and app that will diverse your attention further? You already recognized that you are likely to go get groceries, and milk, was probably at the top of your list.

To integrate such information into our lives, will require either automation, or a real value-add to the consumer. Beside the initial buzz around this "new" capability, there is actually very little value of out sourcing your mind further in terms of this type of planning. Perhaps in the business world (or not). You may be going on holiday - and you do not need more milk, or you might have stopped using it because you prefer a different product. Whatever the case is, the amount of effort needed to configure your life to consume this type of integrated technology - falls short of leaving you with a sense of a real value-add. This may change in the future when true semantic web matures - but I do not see this happening for another generation.

While we continue to evolve our lives into a digital maturity (Matrix - here we come), let's not forget that digital is just second-to, base-on and completely dependent on the real world. Now what kind of house can you build on a poorly maintained foundation? Nothing I would ever want to brag about.

My message to you is: build new technologies, but do not forget to respect the priorities of consumers in terms of real needs which will always prevail (refer to topics such as Maslow’s hierarchy of needs).