Top 5 Big Data Misconceptions

Featured article Anil Kaul, CEO of Absolutdata

Big Data is a game-changer that has already transformed specific business functions, and further applications of Big Data-driven analytics are expected to continue to take the world by storm over the next few years, utterly changing the way data is used and consumed. A huge number of companies today are sitting on massive amounts of data coming from multiple sources, including point-of-sale information, digital and social data, mobile/app data, financial data, research data and other information sources.

There’s no denying that a combination of Big Data analytics and advanced technology produce incredibly powerful tools that can transform operations. However, at present, Big Data is still largely a potential revolution rather than a fully realized one. Many misperceptions about Big Data remain, including a mistaken impression that new tools make it simple to analyze enormous datasets. In reality, harnessing Big Data is still a very labor-intensive business. Here are five big misconceptions about Big Data:

1. Big Data has the ability to deliver self-learning algorithms. Companies have successfully automated responses using Big Data technologies. For example, Amazon and other leading e-commerce companies use Big Data technologies to recommend the next best product based on searches made by the user. However, the system doesn’t work perfectly. Sometimes the recommendations are not relevant because the data was not refined sufficiently. Data insights must be processed and refined, with human intervention and updated algorithms for best results. Prediction functions require an investment of time and need to be refined and developed to reach their full potential.

2. Big Data always provides accurate answers. Next-generation predictive analytics focus on combining, measuring and harnessing multiple sources of data. The greater the quantity of data, the larger the number of witnesses, but increased data volume alone won’t necessarily get analysts closer to the truth. Accuracy requires human judgment to reconcile conflicting evidence. Big Data is still not all-powerful enough to solve this ambiguity and will have to evolve to provide a complete holistic picture.

3. Data lakes will replace the data warehouse. A commonly cited prediction is that data lakes will become the enterprise-wide data management platform that makes it possible to analyze disparate sources of data. In reality, it’s misleading for vendors to position data lakes as replacements for data warehouses or as critical elements of customers’ analytical infrastructure. A data lake’s foundational technologies lack the maturity and breadth of the features found in established data warehouse technologies. Data warehouses already have the capabilities to support a broad variety of users throughout an organization.

4. There is no shortage of skilled data management professionals. Data management and analysis may have the power to unlock the potential of a business, but educated, professional Big Data experts are thin on the ground. Most analysts currently operating as data management professionals in business are still enthusiastic early adopters, motivated to self-educate. To truly unlock the potential of Big Data, companies need access to deep levels of expertise, either by building in-house teams, which is challenging given the shortage of experts, or via partnerships with data experts.

5. The “black box” approach fits all. Many people think an analytics strategy means choosing a specific black box to be fed with specific data and the application of specific pre-set algorithms. But every business problem is unique. Some are too complex and too subtle to automate in this way, and the correlations that pop out the other end are not magically wrapped up in valuable business insight.

What these five misconceptions signal is that, while Big Data has huge potential, many obstacles still confront businesses that seek to use it to the fullest. And Big Data isn’t a monolith: Not all Big Data is the same, and not even Google can get insight from an unstructured dataset. To get the most form their valuable data, companies need a decision engineering strategy.

This is a departure from the traditional approach to analytics. When companies embrace a decision engineering strategy, analytics evolve from an insights generator into a decision driver. And when that happens, companies can finally realize the promise of Big Data.

Dr. Anil Kaul is the CEO and co-founder of Absolutdata. A prominent and well-known personality in the field of analytics and research, Anil has over twenty years of experience in marketing, strategic consulting and quantitative modeling. Before starting Absolutdata in 2001, Anil worked at Personify and McKinsey & Company. He has a PhD in quantitative marketing from Cornell University.

Anil is dedicated to the cause of exploring the best practices for implementing analytic solutions in progressive organizations and building capabilities towards robust decision-making. What distinguishes him is his lens, painted by the brush of analytics, that enables him to scope out opportunities across a wide range of scenarios, be it the performance of fortune 500 companies or like the time he used his base in classical analytics to crunch an entire undergraduate semester’s syllabus into two weeks’ worth of study.