Big Data – boon, bane, or both?

Over the years, I’ve met many a CIO, and if there’s one thing that few of them think on the same line, it’s what they think about “Big Data”. Many of them cogitate that it is indeed overhyped, and so is machine learning and predictive analytics. They think of it as a term that the IT industry coined something that was “invented” to sell their timeworn technologies - just in a prettier, more colorful package.

Every coin has to have two sides, right? Of course, there is this other group, the data experts, and the other side of the coin that thinks Big Data is the next big revolution that will change human life as we know it and solve problems even before they can turn into one.

Then there’s me. Personally, I’d like to think of Big Data as an “Evolution” and not a revolution – It is not something that should be ignored but at the same time, get too excited about.

Most of the enterprise uses only 30% of their available data for decision-making. 70% of the data out there is never used because the data is either never collected or there are no experts to understand and interpret the data from social media like Facebook & Twitter, Data from sensors & IoT, Dark Data, etc.

What’s the point in having data if it’s not analyzed and used? Unused data is as good as non-existent data. (A series of articles will be posted in this blog that would help the readers to understand in making sense of unused data and how to use it to ensure security, to increase revenue, to improve productivity and customer satisfaction.) 

Big Data – The Buzz word

This term has been thrown around a lot since the early 1990s. Lets give John Mashey credit for coining this term if not, making it popular. Enterprises have been collecting data for decades but use only a portion of it because they either don’t have the tools and technology to manage the data or they can’t make any sense of what they see.

There are multiple ways organizations gather data:

• Data from enterprise applications such as ERP, CRM, etc. 
• Data from campaigns, roadshows, surveys, and exhibitions 
• Data from the company website 
• Data from Social Networks such as Facebook, Twitter, LinkedIn, Instagram, Pinterest, etc. 
  
• Data from market, trade and news feeds 
• Data collected from raw surveys, enterprise sources such as application log files, server logs, previous employee data, emails, GPS coordinates, employee notes, application usage behavior, authenticated IPs, user device/software usage, etc.
• Data from network switches, sensors, RFID and Internet of Things (IoT)

Some of the above data sources are structured and many of them are semi-structured or unstructured. Data from various sources must be combined together, processed to a unified model to visualize a way that makes sense to the business.

Big Data processing Workflow

1. The collected data is loaded into a single data source (Ingestion).
2. Ingested data is processed (clean & format) to turn it into meaningful content (Discovery & Cleansing). 
3. The cleaned data is linked, entities resolved, extracted, indexed, prepared (Integration), and stored. 
4. Stored data is analyzed (Analyze) and Machine learning models are created. 
5. Dashboards are designed (Visualize) for continuous analysis to improve the process, quality, market reach, customer satisfaction, and quick decision making.

Analytics

Many enterprises don't know how to use this huge amount of data that’s available. Sometimes, the most difficult decision they fail to make is, “where do I begin?” It is important for the organization to see where they need improvement, what kind of data is needed, and how to analyze the available data? What are the types of Analytics available? What types of analytics would benefit my enterprise the most? 

All these questions can be answered only with clearly defined use cases. Not doing this would only cause more harm than good when implementing big data. 

There are many models out there. How do I know which one is right for my business?

• Predictive Analysis: Provides likely scenarios of what might happen for the given situation based on historical data. Sentiment Analysis is one of the commonly used in this type to predict consumer sentiment scores for quick business decisions. 
• Prescriptive Analysis: Recommends the next steps and multiple courses of action for each decision. It is a combination of a multiple predictive models running in parallel for every possible input action.
• Diagnostic Analysis: Think of this as a postmortem (root-cause) to see what happened and why it happened? This helps uncover new patterns in business processes.
• Descriptive Analysis: It is a data mining process that helps analyze the data in real-time and creates a dashboard for monitoring.  

What are some of the use cases for Big Data?

• Sentiment Analysis for brand promotion and customer satisfaction
• 3600 consumer view to understanding customer behavior and motivation
• Ad-hoc analysis to get customer perception 
• Real-time analysis for quick business decision and targeted marketing
• Multi-channel marketing
• Tailored and targeted messaging
• Fraud detection for pro-active security
• Clickstream analysis to improve user experience

The Market

Industries like Manufacturing, Automotive, BFSI, Insurance, Retail, Supply chain and Healthcare are going to be some of the major beneficiaries of the Big Data platform. 

IDC predicts a 50% revenue increase from the sales of big data and business analytics applications, tools, and services, from nearly $122 billion in 2015 to more than $187 billion in 2019. By 2019, the revenue generated by the US market alone for big data and business analytics solutions is expected to exceed $98 billion closely followed by Europe. The analyst firm estimates revenue by technology, industry, and geography. 

Conclusion

By combining big data with emerging technologies like IoT and subscribing to the social media platform, the modern world is going to see new trends in the way the services are delivered even before they can be requested. I have no doubt that big data and analytics will have a considerable impact on just about every industry.

To put things in perspective, I quote IDC’s Program Director, Jessica Goepfert’s words on Big Data and her words perfectly echo my thoughts. “Its promise speaks to the pressure to improve margins and performance while simultaneously enhancing responsiveness and delighting customers and prospects. Forward-thinking organizations turn to this technology for better and faster data-driven decisions."

Are you in, or are you out?

Cheers,

PS: I am going to publish a series of posts in this blog covering industry-specific machine learning, predictive analytics, and big data from a thought leadership perspective. Please subscribe to my blog to get these updates.