Yikes! Big Data...too much to mine for the gold?

When you see the words "Big Data" of what do you think?  The information produced by your company?  Facebook?  Twitter?  LinkedIn?  Newspapers online?  Information storage for Amazon?  Take one of these entities alone and attempt to decipher how much storage would be needed to store their information. The truth is we cannot even imagine what it takes.  Big data is a term used to describe the exponential growth and availability of storage, both structured (data defined by their type and relationships to other data) and unstructured (not organized in a pre-defined manner).  Unstructured data accounts for more than 90% of the digital universe.  Let's put this into perspective.  "If all 7 billion people on Earth joined Twitter and continually tweeted for one century, they would generate one zettabyte of data".  We almost doubled that amount in 2011.  Here are some examples:  for every 30 minutes that a Boeing jet engine runs, the system creates 10 terabytes of operations information and a four-engine jumbo jet can create 640 terabytes of data in a single journey across the Atlantic Ocean (multiply that by 25,000 flights flown each day), every person in the world having over 215 million high-resolution MRI scans per day.  Moore's Law states that "power of technology hardware doubles every 18 months." Wow!  Can we possibly store this much information, much less be able to analyze it efficiently and use it effectively?

The answer is yes!  It requires smart management.  For many, "Big Data" would be considered clutter.  However, when an organization utilizes the proper tools to keep their information organized according to its various publics and stakeholders, it turns into organizational gold!  We have to mine through the digital dirt for the valuable contents to be revealed.  This leads to more accurate conclusions and predictions of the digital data being utilized.  There are tools that organize data into the proper security bracket to which it belongs, minimizing duplication, how power is routed to supply these command/storage centers, and tools to aid in search and analysis.  The goal is to "extract value from chaos."  

How do we do this as communication professionals?  We need to be continually aware of the changes and advances developing, make adjustments accordingly, and systematize the processes by which we do our research and analyze the plethora of information that we encounter.  Big data can be used to predict future purchasing habits, political canvassing and communication being streamlined, improved weather forecasting based on technology implemented in mobile devices, increasing doctors' performance and knowledge of patients/prescriptions/symptoms, fairer pricing based on different variables per person, recommendation engines used by companies such as Netflix and Amazon, allocating police resources by predicting where and when crimes are most likely to occur, and so many more.  "New ways of linking datasets have played a large role in generating new insights. And creative approaches to visualizing data—humans are far better than computers at seeing patterns—frequently prove integral to the process of creating knowledge." (Shaw)   There is actually a Master of Science degree now offered in Predictable Analytics.  Obviously, big data serves an integral and necessary function when used appropriately.

What are some advantages to having such a vast array of information available to us?  One is the convenience it offers.  There are so many things we can do from home now, and we are better able to communicate with our political leaders.  The converging and unifying of technology, community, devices, and various industries provides many wonderful opportunities.  As communication professionals, we need to be the catalyst in building confidence instead of intimidation within our organizations regarding digital clutter.  How do we do this?  We continually work with IT to develop programs that efficiently analyze the data to improve our business results, we stay involved in social media and aware of how are publics view our organization and how we can improve what we offer by improving the integrity and security of our data, we need to analyze and understand the information available to us and make well-informed decisions, we need to archive old application data and streamline new application deployment with test data management, integration, and data quality, and we need to make sure the teams around us are properly trained and involved.  A community environment is essential in creating and maintaining a vibrant and visionary organizational culture.

In his TED talk video, Clay Shirky posits, "What happens when a new medium all of a sudden puts a bunch of new ideas into circulation?  It changes society, the more ideas there are in circulation, the more ideas there are for individuals to disagree with, and more media always means more arguing.  That's what happens when media space expands."    Therefore, as communication professionals we must embrace "Big Data" and media convergence because it is not going anywhere.  We need to have our information organized, easily accessible, and easy to retrieve.  We need to take ample time to utilize what is available to us to minimize the ability for our stakeholders and publics to "argue" facets of our organization.  In regards to the "Invisible College" Shirky discusses, he refers to the sea of information and properly using tools such as social media and digital media organization tactics to be more prepared to make more reliable arguments.  

Peter Hirshberg says (quoted by Edwards), "We're not in the business of keeping media companies alive.  We're in the business of connecting with consumers."  If we are not connecting with our consumers and providing them an organized way to communicate with our organization, we are running on a treadmill while the digital world around us is running the Iron Man.  This requires viewing "Big Data" as an asset and sparkling sea of information that propels our organization to the next level.  In applying effective strategies in the communications world, we cannot operate off of fear of "Big Data," but relish the opportunities available to us to effectuate credibility, reliability, trust, growth, new standards of effective operation and communication, and expedite the vision of the organization we represent.

Does your organization embrace "Big Data" and use it to their benefit or are there obstacles in the way?  What are the obstacles?  How does your company embrace it and how do they use it effectively?  Thank you for joining me on the quest to understand the benefits of "Big Data".