The Big Data Rush
All too often, trending buzzwords like Big Data tend to become either plagued with oversimplification or increased scope of their capabilities.
Big Data is key to enabling effective machine learning (ML) and nascent AI and is what can help evolve a company's decision-making from what happened? to why did it happen? to what will happen? and finally, to the holy grail of how do we make it happen? Just like erecting a new building, you have to start at the bottom (gather data) before you can get to decorating the penthouse (AI-driven operations).
Consequently, to reach this holy grail and a positive ROI on those long-term Big Data aspirations, the following fundamentals need to be addressed. As the volume of data needed to be ingested to support used cases increases, there is a strong need for automation, governance, and, most importantly, change management. Without those being well-established and defined, the path to positive ROI and being a truly data-driven company will stay out of reach for much longer.
As to automation, when facing the volume of data sources with different timing and volume requirements needed to be brought in, defining clear patterns that teams can leverage and build upon to ensure reliability, velocity, observability, and availability of the data is primordial. It is also key to keeping to a minimum the MTR (Max Time to Recovery) in case of incidents.
As to governance, it is very often one of the most forgotten steps of building and leveraging big data. As more data flows into a centralized location, engineers and business users need to understand the available data, such as what a table's fields mean from a business standpoint, how metrics are and should be calculated, how often data is added or updated and finally where the data is located. It is key to democratizing all the data available to a company. Too often, governance faces the same industry funding struggles as security teams, which is until problems are experienced, efforts are often delayed to later dates. Waiting to implement good data governance will visibility impact the ROI timeline of the big data efforts and will be costly to retrofit.
Finally, concerning change management, it is the most difficult part of the business transformation in leveraging Big Data. The teams that used excel or other tools well suited for smaller data sets need to be re-skilled/trained to be fluent at least in SQL, and for those who will want/need to do more advanced analytics, languages like Python and/or Spark are a minimum requirement. Without such evolutions, getting the most out of your data and new tools will be very difficult and will raise a question as to the benefits of these big data investments.
Now, while on this multi-year journey, there is a need to keep an eye on the horizon. While it takes time to implement the items above, the big data world is constantly evolving. As such, there is a need to keep adjusting the plan and tools selection to avoid being left with what the industry sees as best-of-breed three years ago at the end of big data rush.
To avoid such an outcome, two resources can be leveraged (there might be more in this constantly evolving field). The first one is to hire and leverage strong Principal Data Engineers to help keep not only an eye on the future but also of the technology maturity of the users to absorb changes. The second is to keep an eye on VC funding of data startups. Those companies are the potential disrupters that can upend the plan. If one wants to be the best, those are the ones to look at, not what the other Fortune 500 are using.
Finally, keep openly questioning the plans at least every six months. Big Data is still a fast-moving field, and to paraphrase Ferris Bueller, « Big Data moves pretty fast. If you don’t stop and look around once in a while, you could miss it ».
H-E-B has been on this journey for over three years, tackling interesting challenges to be a leader in the data world. If solving hard problems and thinking outside of the box with massive amounts of data to deal with is of interest, reach out and JOIN US!