Predictable Hybrid Hadoop Blog Series – The Hadoop Deployment Continuum

By March 1, 2017Blog

In working on the recent ODPi White Paper, a few things have come into much sharper focus to the team here.

First is that “Production” is a loaded term. Even though you’ve got really good research from places like AtScale reporting that 73% of respondents run Hadoop in production, we think this term needs to be unpacked.

That’s why we worked across our community, including ODPi members and participants in our User Advisory Board, on this Enterprise Hadoop Deployment Continuum graphic.

The very simple idea here is to plot Hadoop deployments from the lab all the way to enterprise-wide production use and to lay against the gates between phases the primary considerations Big Data teams review before taking the next step.

Many of the folks we talk to in our UAB, our membership and at conferences agree that right now, their Hadoop deployments are straddling the last gate, between Point Solution (sometimes these are massive with big business impact and huge volumes of data, but still focused on a single department/application) and looking to go Enterprise-wide. Some folks we’ve talked to even said they could put specific dates on this image when Hadoop has passed through these different phases. Can you?

It’s a very exciting juncture in the history of this amazing technology. Here at ODPi, we are squarely focused on collaborating as an industry to ensure the needed governance, security models and portability are in place to bring about predictable hybrid Hadoop.

In addition to our Runtime and Operations specifications and our ODPi Interoperable Applications program, we are also ushering in greater predictability through the work of our Special Interest Groups (SIGs), any of which we invite you to participate in:

  1. Data security and governance
  2. BI and Data Science
  3. Spark and Fast Data Analytics

These groups bring together downstream consumers of Hadoop and Big Data technologies ( Hadoop Platform Vendors, ISVs/IHVs, Solution Providers, and End-users ) to discuss and provide recommendations to our technical community on the key challenges and opportunities in each area. Participation doesn’t require code contribution – just the contribution of your insights and expertise on how to bring about predictable hybrid Hadoop for the larger Big Data world.

Inside Big Data said it well: “Enterprises that apply Big Data analytics across their entire organizations, versus those that simply implement point solutions to solve one specific challenge, will benefit greatly by uncovering business or market anomalies or other risks that they never knew existed.”  We couldn’t agree more.

The next blog in this series will contrast the operational consideration when running Hadoop in the lab/limited production versus running it enterprise-wide.