Human Cognition - Cultural Bias Beyond the WEIRD World

Anyone who has lived for any length of time in a culture that is different from the one they were originally raised probably noticed differences, sometimes subtle, in the way that those in the different culture observe the world and things around them.  The more different the culture, the sharper these differences become.  Many people simply discount these differences are quirks.  What scientists are beginning to find is that these "quirks" are often significant, and the differences challenge long held assumptions of human psychological universality. A recent write up in Pacific Standard (found here) goes into detail on what scientists are beginnning to find.  Both the article as well as the research study behind it are an excellent read.

In short, it states that the "generalized standard" for the vast majority of human psychology, cognition and behavior research studies in the world's top journals is based almost entirely upon people who come from Western, Educated, Industrialized, Rich, and Democratic societies (WEIRD).  What scientists are now finding is that there is substantial variability in the way that different cultures perceive the world, from fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self concepts and related motivations; and that population groups from the "generalized standard" are particularly unusual and frequent outliers compared to the rest of mankind.  Americans in particular fall at the extreme end of the scale even within the unusual population of Westerners, making them "outliers among outliers".

This brings into question a wide body of underlying assumptions, ones that form much of the foundation of societies.  If there are fundamental differences in the fabric of how people from different cultures observe the world, how can this be reconciled within the rules that make up everything from civil norms, the ways that people do business, and even education?  Might it even be possible to exploit the potential richness of different mental models within a business or community to make it a business or community more successful?

While different models can and do cause confusion and misunderstanding, they can also provide a richness that spurs creativity and innovation.  Perception that is different than your own is neither necessarily better or worse than your own.  It is simply different.  Exploration of these differences allow you to see potentially useful concepts that otherwise you may have missed.  People, from businessmen to the intellectually curious, have studied concepts as far ranging as Japanese business practices to Buddhism and Eastern mysticism. Steve Jobs, someone that many find to be a creative genius, was inspired by Zen Buddhism.  Others have been heavily influenced by experiences they and their families have experienced, from famines and depressions to wars and idealogical conflicts. They all leave imprints upon the way people approach problems, and being aware of them and their impact upon your mental models goes a long way to allowing you to not only better understand yourself, but also to help open yourself up to other models as well.

Those who are open to exploring often find entirely new ways of thinking, allowing themselves to question the previous certainty of the world around them and opening up entirely new opportunities that they would have otherwise completely missed.

Can businesses also take advantage of this?  Many businesses talk about diversity, but more often than not retain a certain rigidity to their own internal culture.  Opening up to different models, not in a soft HR-led politically correct way but in a much more scientific approach, may allow businesses to be both more innovative.  They also may help companies spot potential "perception misalignments" that may introduce risks to the business.

I apologize, but we appear to be separated by a common language

 "I apologize, but we appear to be separated by a common language"
  -- overheard in the hallways of an international corporation

The role of communication is enormous in the modern world. It has become the backbone of entire industries, allowing for ideas to spread ever more quickly, merge with others to create completely new paradigms, all while destroying many of the monopolies of old. In many fields the speed of communication is the new arms race. However, as we all become ever more interconnected, we increasingly have to run the obstacle course that is our unique backgrounds, cultures, mental models, and even speech patterns. While mental models is a big enough topic for another post, I thought it would be worthwhile to start with the joys of communication in what many mistakenly think of as one consistent and unified language: Modern English.

For those of us who were (un)fortunate enough to grow up with English as our native tongue, one would think that we would be seemingly blessed with the fact that our language is becoming ever more ubiquitous, especially in booming fields such as technology and business. While it is definitely great to be able to convey thoughts and ideas in your first language (as those of us who have tried to in a language we came late to knowing can attest), the infinite flexibility, adaptability and regional differences of English have opened a myriad of opportunities for misunderstanding, confusion, hilarity and very occasionally complete system failure even between native speakers. What are non-native speakers to do? 

Having now lived and worked internationally for a long time, you encounter misunderstandings and lost meaning amongst people almost every day.  After a while, you have to learn to watch your language and to check to see that you were understood, especially when using cultural references. Amusingly, one of the places where I have found some of the biggest and most dangerous misunderstandings were amongst those native to the supposed birthplace of the language:  England.

If you are from a large and fairly cultural and linguistically homogeneous place like Canada/US (where I am from originally) and Australia/New Zealand (where my wife is from), you might occasionally run into communication problems if people are either non-native speakers, not American/Canadian or Kiwi/Australian, or have one of a very small number of challenging regional dialects (such as Cajun, Newfoundland English, Deep South, and some New Yorker and Bostonian accents).  It is generally very rare for the misunderstanding to last long, and (unless it is a Québécois trying to be difficult) rarely does it devolve into complete communication breakdown.  Perhaps it is due to the relatively recent settlement of these countries and the rise of modern media that has helped.  Regardless, this homogeneity gives a false sense of certainty that what you say will be understood, at least for the way you meant it to be.

Across English speaking country groups the confusion really begins.  Even though the language and structure is more or less the same, and even many of the literary and media references are mostly shared, regional terms and cultural references are often lost. Between the North American cluster and the Australia/New Zealand one there are occasional misses, though whether it is due to similar historical backgrounds or the more direct nature of the cultures to quickly catch and fix these usually keeps these to a minimum.

This all seems to go out the window in the UK.  While one would expect some differences between Wales, Scotland, Northern Ireland, and England, there are also significant difference across England itself, sometimes even between two towns that sit right next to each other.  There are the many better known dialects such as Cockney and Geordie, but there are also many dialects and subdialects across Yorkshire, the Midlands, East Anglia and across the south.  These dialects are far more distinct than those found in the Americas.  Few speak the BBC English that many outsiders would be familiar with.  There are also differences between people from different classes, even those from the same area. The class boundary is a particularly odd one for Americans who, despite all of the recent press lamenting growing socio-economic gaps in the United States, simply have little grasp of the class concept. 

While this is all rather confusing for a foreigner, many English also suffer from a similar problem that many Americans have:  they expect by default that everyone will understand what they are saying.  The empire once was strong, and it is technically called "English".  But even when there are plenty of examples within one's own country that hint that this assumption may be flawed, it does not seem to affect the prevalence of complete communication breakdowns, many that often compete with Monty Python sketches for their absurdity. 

This becomes a huge problem in business, especially within multinational companies.  I have seen time and again language subtleties causing misunderstandings that derail teams, projects, and even big initiatives.  All of this affects morale and trust, let alone the productivity and success of the business.  Assumptions that everyone understands what you mean can be extremely dangerous, even, as with the English, everyone happens to come from the same country.

I have found when trying to convey and idea or concept, that it is always good to test in a non-threateningly way, whether it is by asking questions back of people of their thoughts or what they think they might do in response to the new information, to see whether it was really grasped.  This also works well the other way, which I use often.  That way drift can be caught early, leaving far less room for wild and dangerous surprises later. 

This can go further, and certainly is not restricted to just English speakers.  When looking at the health of a company, it is also useful to look at the means for which people communicate.  We all have stories of misunderstandings developing because someone misinterpreted something said, especially if a low bandwidth medium such as email was used.  Teams and companies also suffer when communication quality is not up to the mark.  I have personally seen very bad situations develop, not only in English and polyglot companies, but also in German and Spanish speaking ones as well.

People who have to interact with one another often work much more effectively together when they know each other well enough and both sides have the ability to further develop communication links. This allows people to better understand each other's context, and correct small misunderstandings at a ground level very quickly. 

Visual indicators, whether they be facial expressions, pictures, or even trends on a graph are also high bandwidth ways to convey complex information.  They work quickly and can be very effective to communicate concepts from diagnostics to thoughts to emotions, bringing truth to the old adage that "a picture is worth a thousand words".

Companies need to pay attention to the quality of communication across their organizations.  Without doing so, they risk creating a tower of incomprehensible babel and failure.
 

Release Batching - When is it a Problem?

I have seen quite a few flame-ups, particularly in the Kanban community, around release batching. It is a subject that I have spent a considerable amount of my career addressing.

Most IT Operations teams abhor and resist change. Change usually brings with it uncertainty and the potential for sleepless nights around a failed service. As it is often impossible to replicate the exact production environment in a test setting, there is a real risk that changes have not been tested adequately. There is also the problem that changes often mean a service outage is required, directly affecting customers.

What makes this worse is the fact that most software engineers as well as IT Operations people are less than sufficiently diligent with properly utilizing configuration management techniques in order to track the changes they make.  It is always far easier to hack in a change directly into a production system than it is to write it, check it in, package it and release it properly while managing all the dependencies.  But such changes cause configuration drift  that makes even the most ideal situation where development and test hardware is identical to production still produce different behaviours.

Release batching is the way that many organizations try to solve this conundrum.  By piling up all of the changes into one big release, the number of change events goes down.  This gives false security to the IT Operations people, who feel that they can heavily man those few events and field all the failures at one time.  It comforts the testers, who feel that they can spend the time to test through all the various changes.  It also allows the developers to be a bit lazier about the way they check in code and manage builds, as there are long periods where they have to code and fix builds.

But all of these supposed benefits not only hide dysfunction, but take value away from the business.  The longer that code is sitting waiting to be shipped, the longer it is sitting not being used to provide value to the business.  While the code is sitting, assumptions that were made that resulted in the code being written are not being tested, and even if they were correct at the time the moment may have been missed and the  market may have moved on.

The idea that fewer big changes means less downtime is also flawed.  Bigger changes, by definition, usually mean that more has changed.  More changes often make for more things to potentially go wrong.  It also means that when something goes wrong, the haystack you are having to dig through to find the problem is far bigger.  The idea that QA is going to be able to catch all defects, especially with large changes, is also flawed.  The measure of defects found is a gauge of how effective your QA team is at finding them against an always unknown metric of the total number of defects.  It is not a particularly good indicator of the number of defects in the code, the quality or maintainability of the code, or even the amount of problems you might encounter when releasing the code.  Lots of changes mean an exponential increase in the number of potential test cases required to find issues.

While pull mechanisms such as Kanban help capture an understanding of flow within the development phase, release batching tends to counteract much of the benefits by hiding dysfunctions under a false security blanket.  Difficulties in configuration management and automated deployment are solvable, and can be tackled incrementally to reduce release cycle time.  By moving progressively towards a system of continuous delivery, ever smaller changes can be understood, tested and released, allowing for very quick feedback and even faster rollback in case of problems.  Changes become far more atomic, risk becomes better understood and easier to manage, and pull can ultimately be achieved across the entire product lifecycle.

The Importance of a Vision and Flexibility

As the world keeps increasing the rate of change while the barrier to entry into the market fades steadily away, many leaders today seem perplexed by the ability for some companies to go from success to success, adapting swiftly to the ever quickening changes in the market while never compromising themselves. When they study these companies, what they often miss is the idea and power of freedom within the ranks to improve and self organize themselves within the scope of a powerful shared vision that they are working towards. This isn't the vision statement drivel that so many companies produce and then ignore, or the command and control orders that get barked down the ranks that don't invoke or allow for thought and creativity. It is an overarching set of high level and far reaching goals that runs deeply throughout the blood and soul of a company. These goals should be very relevant to the industry the company is in. They must be espoused by the executives and be part of the values that they live and are guided by day to day. Finally, though most importantly, the vision needs to be both real and, at the same time, is so bold and far reaching as to be nearly impossible to achieve.

Taichi Ohno's one piece flow is one example of such a vision. Ohno professed the ideal of creating an environment of ultimate flow, where there were no batches and only what was purchased by the customer would be made. This is extremely difficult to do in manufacturing, especially in the vast complexity that exists within automobile manufacturing where Ohno worked. It also goes completely against the commonly held belief, and the beliefs within modern accounting, of optimizing through maximal utilization of factory capacity rather than optimizing on the flow of the pull from a customer. Yet the power was not solely contained within the vision itself. The workers in the trenches and their line managers had to internalize the vision. They did not wait for a list of orders of what exactly to do to come from above so that they could carry on like mindless drones, but instead organized and challenged themselves to use their own experiences and knowledge to continually look for ways to improve their area and the business in the direction of that vision.

The workers on the ground have a critical amount of understanding and control over what is going on. They also are the raw material that can be coalesced to a form that is far mightier and wiser than the traditional corporate structure made of a few brains and many mute and dumb hands. Without their alignment to a vision or a free hand in finding ways of improving towards the goal, progress will be extremely difficult. However, at companies such as Toyota (where Ohno hailed) and Honda, line managers create goals that lead towards the vision, while the employees are given leeway to achieve those goals in the best way they see fit. This allows for what Ikujiro Nonaka calls "reflection-in-action". It creates a world where the collective creativity of the employee base can be harnessed, opening up entirely new possibilities. Such flexibility moves the company away from rigid and less efficient structures and concepts that may have unknowingly become obsolete. It removes the unintentional inhibition of the flow of communication, experimentation and the development of new ideas by allowing for knowledge to flow and recombine in new ways across the organization. This flexibility of thought and structure also has the magic of providing room for new meaning for work and a feelings of ownership to develop for staff that not only helps the company but brings with it pride and a real buzz about work that can be felt on the shop floor.

The strategy of vision with flexibility exists and is the key to success in many other areas, from self organizing nonprofits, to open source projects, to much of the way the US Armed Forces works. It is enshrined in modern manoeuvre warfare, and the concept of decentralized command structures expects that rapid changing situations may out pace communications and create gaps in the knowledge within the chain of command might have. This thus puts the need for lower levels to understand overall intent and adjust themselves on the fly in the battlefield accordingly in order to ensure success. This doctrine is even more pronounced in the Marine Corps, Special Forces and other elite units where flexibility and dynamism from the men in the field is even more important, down to the expectation that they will be "T shaped people" that have many well rounded skills and can adapt immediately to new roles to fill in any gaps that develop as conditions change.

There are several characteristics that each of these types of organizations possess that aids them in their success. They each have a big vision that is professed by all members ("save wildlife", "create the ultimate operating system", "win the war"). They will have somewhat more defined missions and campaigns formed in the upper and middle ranks that lead in the direction of the big vision. Within those missions/campaigns, the people on the ground are able to work in a mesh and organize and change tact to adapt to changes and opportunities as that can be exploited as they develop to lead towards achieving the desired goal. Any lack of flexibility and/or deep understanding of intent and vision quickly gums up the gears. Nonprofits and open source projects fragment as people walk away from them, much like customers will walk away from a business that no longer meets their needs, while such failure in the armed forces is much more lethal. The natural reaction of many large organizations to this is often to become even more rigid and prescriptive, insisting upon tightly defined roles and handoffs that not only deter collaboration but go to further accelerate the feeling on the ground of a lack of ownership and control over their destiny. While work might seem to be progressing to management, the workers themselves become detached, leading to an even greater lack of response to change, greatly harming the organization's long term health and profitability.

The struggle to move away from the more traditional command and control structure that exists in most companies to such a new and different way is daunting and scary for many. The feeling of delivering orders and directly managing people is a bit like a security blanket that somehow feels more certain than simply guiding and coaching. It pushes organizations to trust their employees more and value their knowledge and intellect rather than insisting upon treating like mindless children that need to be told what to do. It also forces thought leadership, and for leaders to step up and provide a far reaching strategic vision rather than spend all their time in the instant feedback of the tactical. Until leaders accept their new role and free and leverage the collective wisdom of their staff, they will doom the organizations that they lead to slow and cumbersome mediocrity, capital destruction, and gradual irrelevance.

Constraints of the Cloud Provider: The Operator's Dilemma

As one who has worked on building and operating on-demand services for many years, there are several areas where I feel most mainstream thinking around cloud have not yet sufficiently developed enough to allow operators to efficiently and effectively provide services for their customers. This frustrates me greatly, as having worked either for or with the often quoted but still poorly understood top Internet service companies out there (like the ones that start with "Y", "A", and "G"), very advanced techniques have been created that when used properly largely overcome these challenges. These approaches are in many ways very different than the traditional industry methods that so many vendors tout, so different that they often turn traditional thinking completely on its head and thus are not hindered by limitations that most of the industry have taken as law. This has left me either chuckling or exasperated as so called experts who have never worked for those sorts of companies jump up and down and insist that the world is flat and that there are dragons at the edge of the horizon.

While I can (and, if my friends and colleagues have their way, probably will) write books on the subject, I will try and first lay out some of the key constraints that I see must be overcome in order to really provide effective and efficient cloud services, whether in the large or in the small.

* flexibility and agility of the stack- this is the one that seems to befuddle people the most, and the one that many companies unnecessarily jam in a hypervisor to coarsely "tune". Service providers usually experience peak and trough traffic demands, with the occasional spike or "super peak" event interspersed. Understanding how customers use the service, including the standard load period, amplitude, how the load manifests itself (CPU, I/O, memory, bandwidth, etc within particular parts of the stack), and (if possible) peak and trough times allows a provider to calibrate not only the size and type of required capacity, but how quickly demand can increase. Providers can build out to fully accommodate for peaks or spikes, but that means that for much of the time there may be underutilized resources. At very small scale this might not be much of an issue, but at large scale, as well as when resources are otherwise very tight, this can be unnecessarily costly.

If a provider wishes to respond to load increases and decreases in a more dynamic way, they need to be able to bring up and down more capacity quickly, as this response time determines whether the end user experiences degraded service or an outage. The biggest challenge here usually comes down to the how finely grained the unit of capacity is. Most people treat the OS "unit", whether it is on physical hardware or a virtual guest, as the one (and sometimes only) building block. This is unnecessarily limiting, as it does not easily allow for scaling that discretely addresses the bottlenecked resource. Mechanisms for building and bringing up and down an OS "unit" are cumbersome outside a virtualized environment that has an image library. Without the right frameworks building a server can be slow and error prone, especially if the physical equipment is not already on hand. Within a virtualized environment, images are not always well maintained, and the instrumentation and service management tools are often either inadequate or improperly used.

Finally, agility needs to be provided in a way that is resource-effective and reliable. It must not require large numbers of staff to respond and maintain, and infrastructure changes need to be known, atomic, reproducible, and revertible as much as possible to maintain consistency and reliability. Agility also needs to include the relocatibility of infrastructure and services for contractual, legal, regulatory or other business factors. I have seen many companies get these wrong, or worse hope that by merely adopting industry practices such as ITIL without developing any deeper understanding or refinement of the underlying services and environment will magically improve agility, reliability and resource-effectiveness. Until this is overcome, the service will never approach its cloud-like potential.

* Data Classification and understanding- I covered this in an earlier post. Understanding the nature and usage of the underlying data allows you to work through how best to approach dynamism within your stack. Without this understanding, data can become stuck, fragmented, inconsistent, or insecure, and negatively affect the reliability of the service.

* Interconnectivity- a truly dynamic cloud infrastructure needs to efficiently and securely communicate both across the components within the infrastructure but also with the end user community. This means that networks must be flexible, dynamically scalable, and have minimal latency (often referred to as "near wire speed"), as well as manage to also create secure "pipes" for moving data. This poses challenges as traditional network topology is often either rigidly dedicated to enhance security or flat and insecure. Also, dynamic storage networks are still in their infancy, as can be seen by the as of yet lack of one definitive winning standard, despite the hype around FCoE and iSCSI.

* Power- this constraint can be defined in any number of ways, whether by power, cooling, space efficiency, or redundancy. As compute power continues to grow and miniaturization improves, achieving effective compute densities in a cost efficient manner is quickly becoming an industry of its own. Innovations such as the Yahoo Computing Coop and Google's shifting datacenters are far out in front. Others battle with lower compute densities, more moving parts, and therefore higher overall costs. Not addressing this limits the availability and cost effectiveness of cloud.

* Security- this constraint is more about perceptions, with some additional sophisticated understanding of the usage of the underlying service, than the problem that many hawkers of "private clouds" will have you believe. This is not to say that many cloud providers have already figured this out. In fact, some are only slightly less sloppy than the standard IT department. Better providers will understand the dynamics of the underlying service, will properly classify and segment where applicable data within the service. They will also properly track and audit access to the data to ensure proper handling.

Where security becomes a more interesting challenge is in the Contractual, Legal and Regulatory (CLR) arena, where requirements may be introduced that are less about security and more around data handling and locale. There is still a great deal of FUD that has built up due to the constant challenges that have arisen in the consumer space. As technology, understanding, and the legal and regulatory landscape evolves, this should improve to allow for customers to enjoy the power of the cloud.

Cloud and the Edge Device: What about the User

(Those who know me know I have had my hands more than a little full the last couple of months. Hopefully, things will settle down soon enough so that I have more time to devote to this.)

While a lot of the discussion in the last year or so has concentrated on the infrastructure and service side of Cloud, arguably the biggest effect upon the whole stack will be driven by what happens on the edge. As more and more people acquire, use and become comfortable with smart phones, tablets, and various roaming laptop form factors, demands for speed, flexibility and portability of applications and services to such mobile devices will skyrocket. No longer will road warriors want to have to be tethered to the office or a wired network to be productive and stay connected to the business and friends. They will also steadily have less and less patience for those "sticky situations" where applications and/or the data they need is "stuck" on a system in the office. While some might see a VDI approach as a way to overcome this, it is a far from perfect hack that twists a traditional paradigm to create a stop gap intermediary layer rather than have the edge device communicate directly with the wanted service itself.

The first place to go in order to best understand the edge is to look at the constraints that exist there. The first is connectivity. In the early days of the Internet when line speeds were measured in Kbps it was impractical to run data hungry applications, and it was not until broadband became more widely available that applications like streaming media became popular. Wifi, LTE and 4G will go a long way to do this with mobile devices however reliability and any latency issues will still need to be addressed. Uneven coverage, clogged masts and backhaul, as well as coverage holes and shadows must be dealt with through further improvements in the technology as well as presentation resilience of the service on the edge device itself.

The second constraint is form factor. While people have different thresholds for what they are willing to carry and use, it is clear that the edge device needs to be lightweight and relatively easy to carry and stow away while still being big and powerful enough to use comfortably. The advent of devices like the iPhone and iPad have just begun to cross the threshold for many. However, insufficient processing power, fast connectivity, and the proprietary nature of iOS have limited the eventual revolution thus far. This has driven the market for specialized applications, but has not yet quite opened the floodgates towards showing what might be possible with cloud services. As it is now clear there is a strong demand for these devices, I expect rapid improvements in technology, as well as the spread of more open operating systems such as Android, should allow operators, consumers and cloud service providers alike to rapidly hone in one this sweet spot and create an even larger market and drive ever more innovative usage patterns.

The third constraint is power. As people are always on the go, and wifi connections are generally power hogs, battery life is critical. As form factor is also important, these batteries must be small and lightweight, yet powerful, long lasting and easily recharged. While I have been impressed with the iPad (though not with any of my other mobile devices), a lot of progress still needs to be made here.

Finally, the last constraint, which I view as perhaps also the biggest opportunity for Cloud, is security. As regulators start to create ever harsher laws for the loss of data, it will become ever more important to tighten the amount of data that is allowed to reside on mobile devices. This may mean either creating a much sharper tiered storage policy towards sensitive and personal data that limits the amount that is actually stored on the edge device. If connectivity, power, and form factor are improved, this will allow for less resident data to exist on the device. Data can also be keyed back to, or have holes punched in it like a puzzle to be filled by, a central cloud service to allow it to be accessed in whole. As the other constraints are slowly pushed back, there will be ever more creative ways that this can be addressed.

The Art of Data Classification and Management

Anyone who has tried to build and manage dynamic infrastructure and services knows that data handling is one of the most complex parts. While it is slowly becoming more and more obvious to people how to generalize and template infrastructure, operating systems, application and service stacks, many still get stuck on how best to deal with the underlying data. Data for them feels unduly "sticky". Due to its size it is often difficult to dynamically move en masse, it changes frequently enough, its nature is often poorly understood, and to people who treat systems as a black box is unpredictable enough ways to make people nervous.

This uniqueness is a very serious problem. It also highlights one of the biggest weaknesses of most companies that try and tread into the cloud space, which is not understanding the nature and uses of the entire ecosystem. Being able to deploy, move around, and even virtualize systems is novel and potentially useful, but without a true understanding and model for the end to end service and how it is used gaps form that limit the usefulness of cloud, or worse run roughshod over the service quality expectations of the user community. This increases the amount of FUD against dynamic services and muddies the water for people to truly realize their real strengths.

Back to data. In my experience, the nature of data in any system can and should be understood, tracked, and put into classifications for better management and usability. One form of classification is the dynamism of the data. Is is static, either not changing at all or very slowly over time. Things like birth dates fit into this category, as your birthday never changes. Is the data dynamic, meaning that it actively changes, and the patterns of change can be captured, tracked, and replayed 90%+ of the time? Most common transactions in a database will fit this pattern, which is why redo logs and heavy tuning of their application and the way they are replayed on backup or replicate databases is an important skill in an HA environment. Is the data interstitial, data that is caught between the customer and the system or service that the customer is interacting with? This is the data that will almost certainly be lost if the underlying service goes away during the interaction, and needs to be minimized to an amount that the customer is willing to accept. Understanding the patterns for which the customer interacts with the system, and the types of states that the system can be left in during an outage help not only set customer expectations but also allow for a much better understanding of ways to more dynamically manage an environment in a way that maximizes service quality, resilience, utilization and cost effectiveness. Mapping the data across these classifications helps with the understanding of the environment in ways that greatly improves the effectiveness of targeted data management techniques.

There are other classifications that may be useful as well. Is the data relational or block? Is it big (such as media) or small (simple text)? Is the data latency tolerant (RSS feeds) or intolerant (financial feeds)? Is it quickly processed/rapid access (transactional system or distributed hash table) or batch (such as data processing in a map reduce or data warehouse cluster)? Is the service distributed globally or centrally located? Is data tolerance absolute or resilient to lazy updates? Does it need to be secure or have other regulatory or legal constraints that need to be taken into account while storing and handling the data? Each of these and others help with understanding the end to end system and allow for a much more targeted battery of approaches and tools that can be used to manage the environment more effectively. Often I build a matrix of sorts that allows for people to map and understand the nature of the various types of data to assist with such targeting and break the mindset that all data is created (and thus treated) equally, or the worst and most common mistake that it must be all treated at the same lowest common denominator.