Eddie Elizondo | Last updated September 2024 | No Hype Analytics blog homepage
What happened to big data? It is still around but irrelevant in most cases. More data exists than before - technically data keeps getting bigger - but most businesses do not need “big data” in AI or day-to-day decisionmaking, so the hype passed. Generally, unless real-time streaming data is involved (e.g., high-traffic web clicks, IoT devices), data is not “big”. And even if there is “big data” somewhere in the organization, analytics teams rarely work directly with it. For example, IoT devices in manufacturing may be streaming records across a large factory floor multiple times per second, but analytics teams don’t work with that data. There is a low-level ETL process that aggregates the second-level data, which can be very noisy, into minutes; then, analytics tools building on the minute-level often aggregate it even further. Most AI tools and people are not engaging with “big data”, even if it exists.
What can be learned from the hype?
- Start small with technical services and infrastructure, especially early in an analytics journey. Often, companies build with tools for the most complex potential use case (which is not even on a development roadmap) or because of a fear of refactoring. This overburdens and overcomplicates most of the work, and add unnecessary cost for computing, service subscriptions, and engineering resources. For example, there are few analytics uses for “big data”, most analytics do not need a Spark instance and related services when a new table in an existing SQL database works just fine (often, it’s even faster.)
- Challenge how a new trend or technology will be applied in your business. What are the specific functions and use cases that can come from it, and what value could be delivered? In 2023-2024, the hot trend is GenAI. While there will certainly be new value delivered from it, it will have limited impact for most functions and existing use cases, especially in the short-to-medium term. Have a healthy discussion about how it can specifically deliver value.
Do you have questions, thoughts, feedback, comments? Please get in touch - I would love to hear from you: eddie@betteroptima.com