Predictable Hybrid Hadoop Blog Series – The Hadoop Deployment Continuum

By | Blog

In working on the recent ODPi White Paper, a few things have come into much sharper focus to the team here.

First is that “Production” is a loaded term. Even though you’ve got really good research from places like AtScale reporting that 73% of respondents run Hadoop in production, we think this term needs to be unpacked.

That’s why we worked across our community, including ODPi members and participants in our User Advisory Board, on this Enterprise Hadoop Deployment Continuum graphic.

The very simple idea here is to plot Hadoop deployments from the lab all the way to enterprise-wide production use and to lay against the gates between phases the primary considerations Big Data teams review before taking the next step.

Many of the folks we talk to in our UAB, our membership and at conferences agree that right now, their Hadoop deployments are straddling the last gate, between Point Solution (sometimes these are massive with big business impact and huge volumes of data, but still focused on a single department/application) and looking to go Enterprise-wide. Some folks we’ve talked to even said they could put specific dates on this image when Hadoop has passed through these different phases. Can you?

It’s a very exciting juncture in the history of this amazing technology. Here at ODPi, we are squarely focused on collaborating as an industry to ensure the needed governance, security models and portability are in place to bring about predictable hybrid Hadoop.

In addition to our Runtime and Operations specifications and our ODPi Interoperable Applications program, we are also ushering in greater predictability through the work of our Special Interest Groups (SIGs), any of which we invite you to participate in:

  1. Data security and governance
  2. BI and Data Science
  3. Spark and Fast Data Analytics

These groups bring together downstream consumers of Hadoop and Big Data technologies ( Hadoop Platform Vendors, ISVs/IHVs, Solution Providers, and End-users ) to discuss and provide recommendations to our technical community on the key challenges and opportunities in each area. Participation doesn’t require code contribution – just the contribution of your insights and expertise on how to bring about predictable hybrid Hadoop for the larger Big Data world.

Inside Big Data said it well: “Enterprises that apply Big Data analytics across their entire organizations, versus those that simply implement point solutions to solve one specific challenge, will benefit greatly by uncovering business or market anomalies or other risks that they never knew existed.”  We couldn’t agree more.

The next blog in this series will contrast the operational consideration when running Hadoop in the lab/limited production versus running it enterprise-wide.

Improving Production Hadoop: ODPi Member Conversation with Ampool

By | Blog

Last month, John Mertic sat down for our first ODPi Member Conversation podcast with Milind Bhandarkar, founder and CEO of Ampool.

The exciting discussion centered around the challenges production Hadoop deployments face and how to make the framework faster, easier and more productive.

As he’s spent the last 11+ years working with the various versions of Hadoop – first starting at Yahoo!, where Hadoop was invented – Milind had some interesting context to share with podcast listeners.

After highlighting the changes the space has seen since Hadoop was first introduced to the world, he explained that today’s projects usually “depend on different projects or on different components in the Hadoop ecosystem.”

The importance of interoperability within these offerings, to ensure today’s software-defined companies are able to harness the full power of their data, cannot be understated – as John and Milind agreed that one of Hadoop’s biggest challenges in production has been ensuring that commercial distributions are compatible across multiple components and the applications that have been written to use these components.

To hear more of Milind and John’s expert insight, including more ways to improve production Hadoop, tune in to the episode on our YouTube channel!

Subscribe to our YouTube channel and follow us on Twitter to catch upcoming episodes of the ODPi Member Conversation podcast series!

2017 Predictions: What’s Next for Hadoop

By | Blog


By: John Mertic, Director of Program Management for ODPi

If you follow ODPi insight closely, you might remember these 2017 Big Data Predictions from our VP of Technology, Roman Shaposhnik. After the start of the new year, I started to think about what his predictions and emerging trends like Big Data’s “Push to the Cloud” might mean for our ecosystem – especially as it relates to the Hadoop landscape.

Last year, Apache Hadoop celebrated its tenth birthday. It was a milestone for the diaspora of the early team at Yahoo! that invented the technology and the worldwide community, along with The Apache Software Foundation that shepherded the growing platform since its launch. However, this decade-iversary also showcased something less obvious than Hadoop’s staying power: it brought to light that the canonical state of Hadoop is breaking apart.

Over the last couple of weeks, I’ve spent a lot of time reading through Hadoop and Big Data landscape articles written in the past few years. The most popular conversation was clearly the expansion of the stack – meaning new projects for every possible nook and cranny of the space. Fast data? Check. 12 ways to perform a SQL slice and dice? Done. AI (artificial intelligence) and ML (machine learning) capabilities? Yup. To see what I mean, take a look at this enormous Hadoop Ecosystem Table – summarizing current Hadoop related projects – here.

Traditionally, the role of Hadoop distribution providers within the ecosystem was to help make sense of a fast-changing and often-confusing landscape for customers. Showcasing their own preferred tools, distros gave the enterprise a stack of components that (more-or-less) worked well together – provided users stayed within confining application architecture walls. While this wasn’t ideal, it worked fairly well if enterprises were happy to stay in the “safe zone” their selected vendors laid out and could blissfully ignore other distros and solutions.

Though this may seem simple, the nature of deploying Big Data is quite varied. Reading through AtScale’s recent “Big Data Maturity” report, 53% of respondents reported using cloud in their deployment but only 14% have all of their data in the cloud. Not to mention Tony Baer’s recent ZDNet article citing that Hadoop in the cloud is a varied product depending upon the provider – and not in the traditional sense with how Cloudera CDP differs from Hortonworks HDP. This emergence of cloud brings into focus a fundamental shift emerging within the entire Big Data landscape.

If there is one overarching lesson the drive to PaaS and IaaS taught us, it would be the benefits of being lean. For example, you can throw more CPU, RAM and disk drives onto your on-premise environment with negligible cost increases; but for cloud instances, each addition counts against you quickly. Knowing this, the best cloud architectures include the ability to compartmentalize, identify focus areas of work and optimize each resource used – as wasting resources on the cloud has in-your-face cost ramifications.

Now combine Hadoop’s push to the cloud with the forced fiduciary responsibility of using cloud resources, and it’s quickly apparent that a traditional one-size-fits-all Hadoop distro is at natural odds – especially when that distro comes with a number of projects and tools that you’ve long-since outgrown.

My biggest prediction for 2017 is that the Hadoop of 2016 is going to become much more modular, special purpose and leaner than what is currently being shipped. We’re are already seeing these trends in the following ways:

  • IBM’s Watson Data Platform is centered around Spark – notice anything missing?
  • Cloud vendors are moving away from traditional HDFS and, instead, making their native block stores the data lake
  • Even traditional Hadoop distro vendors are recognizing this trend and launching offerings leveraging containers as a stopgap solution

This slow elimination of the one-size-fits-all ideal leads me to my second prediction: Hadoop and Big Data will no longer be discussed as their own beings – they’ll instead just be referred to as “Data.” I see this acknowledgment as the separation line between vendors who will be successful in 2017 and those who will not. Connecting the entire landscape story together, and speaking to customers about their data strategy vs. shiny new Hadoop or Big Data products, will separate this year’s data winners from its data losers.

My third prediction for Hadoop: ridding the marketplace of the “traditional Hadoop” baggage, and having the important conversations around data strategy, will employ the needs of traditional business to highlight leading technologies in this space. While this may sound pretty obvious, try answering this: how many traditional businesses are bragging about the efficiency of their Hadoop/Big Data/Data solutions and strategies right now? Not many. However, these businesses know that in order to remain competitive they’ll need to become “data driven.” I think we’ll start seeing organizations drive their needs back to vendors like never before and their successes will be much more prominently showcased. In other words, less focus on Amazon, Netflix and Facebook, and more narratives around companies like Progressive Insurance.

It’s a key year for Big Data as it crosses its biggest chasm yet, but as greater focus comes to this industry I think we’ll start seeing a noticeable push forward – setting up some even more impressive leaps in 2018 and beyond.

ODPi Community Lounge @ Apache Big Data Europe

By | Blog

Join the Discussion at the ODPi Community Lounge

Once again ODPi is sponsoring the Community Lounge at Apache Big Data Europe, November 14-16 in Seville, Spain.  Apache project members and speakers are welcome to hold their meetings and after-session discussions.  This is a great way to have a deeper intimate conversation with fellow attendees, and to introduce new potential collaborators to your project

Please choose a time on the Community Lounge Schedule  for your topic or project.  We’ll help promote your upcoming meeting.  Be sure to tell your followers as well.  Time slots are 30 minutes each and can be scheduled on a first come, first served basis.

ODPi Community Lounge – ApacheCon EU 2016

Discussion Schedule

Monday, November 14

Time Speaker or Project Name Topic
12:30  Apache Giraph – Roman Shaposhnik  Discussion session: Practical Graph Processing with Apache Giraph
13:30 – 15:30 Lunch 
15:30  Apache MADlib – Roman Shaposhnik  Distributed In-Database Machine Learning with Apache MADlib (incubating) – Roman Shaposhnik, Pivotal
16:00  Apache Geode – Greg Chase  Meet Apache Geode – graduated for Apache Incubator

Tuesday, November 15

Time Speaker or Project Name Topic
13:30 – 15:30 Lunch
15:30  Apache Big Top & Greenplum Database – Greg Chase & Roman Shaposhnik Discussion: Massively Parallel Data Warehousing in the Hadoop Stack

Wednesday, November 16

Time Speaker or Project Name Topic
10:30  John Mertic, Director, ODPi and Open Mainframe Project, Linux Foundation  Discussion: Keynote: Lessons from the Trenches: How Apache Hadoop is Being Used & The Challenges Its Users Face –
11:00  ODPi – John Mertic  Discussion: Standardizing data governance across Hadoop distributions
11:30 ODPi – Roman Shaposhnik and John Mertic Discussion: Security in Hadoop
12:00 ODPi – Roman Shaposhnik and John Mertic Discussion: Streaming data in Hadoop
12:30 ODPi Discussion – Roman Shaposhnik Discussion: Hadoop Compatible File Systems across Hadoop Distributions
13:00  ODPi – Alan Gates  Discussion: Standardizing Hive in Hadoop distributions
End of conference

Is Your Data Clean or Dirty?

By | Blog

downloadOver the weekend I read an incredible post from SAS Big Data evangelist Tamara Dull. I love her down-to-earth and real life perspectives on Big Data, and your analogy of cleaning the car hit home for me. She is spot on – clean data pays dividends in being able to get better insights.

But, what is clean data? What is that threshold that says your data is clean versus dirty?

Could data even be “too clean”?

(pause to hear gasps from my OCD readers)

Clean data and clean houses

Taking this to a real life example, I can say first hand there are often different definitions of what clean is. For example, my wife is very keen on keeping excess items off our kitchen counters, to the point where she’ll see something that doesn’t belong and put it in the first cabinet or drawer she encounters that has space for it. Me on the other hand, I’m big on finding what I believe is the right place for it. Both of us have the same goal in mind – get the counters clean.

To each of us, there’s value in our approaches – which is efficiency. Hers is optimized at the front end, mine at the back end. However, the end result of each of our “cleaning” could have negative impacts (with my approach, it’s my wife’s inability to find where I put something – with my wife’s method, it’s having items fall out of a cabinet on me as I open it).

Is “clean” to one person the same as everyone?

The life lesson above teaches something critical about data – clean isn’t a cut and dry threshold. And taking a page from Tamara’s post, it’s also not a static definition.

The trap you can quickly fall into is thinking of data in the same terms as you would have looked at structured data. While yes, part of the challenge is to understand what the data is and its relationships, the more crucial challenge is how you intend to consume the data and then use it. This is a shift from the RDBMS thinking of focusing on normalization and structure first and then usage second. With the Big Data-esque ways of consuming and processing data (streaming, ML, AI, IOT) combined with velocity, variability, and volume, the use-case mindset is exactly where your focus should be.

“Use case first approach” is how we look at these technologies at ODPi. We look at questions like “Here is the data I have, and this is what I’m trying to find out – what is the right approach/tools/patterns to use?” and how they can be answered. We ensure all of our compliant platforms, interoperable apps, and specifications have the components needed to enable successful business outcomes. This provides companies the peace of mind that they are making a safe investment, and that switching tools doesn’t mean that their clean data becomes less than optimal to leverage the way they want.

This parallels on the discussion of cleaning in our house – are we trying to clean up quickly because company is coming over, or are we trying to go through an entire room and organize it. Approaching data cleaning is the same thought process.

ESG Whitepaper: ODPi Simplifies Apache Hadoop Application Development and Portability

By | Blog



Over the last decade, Apache Hadoop has generated many popular open source software projects, spawned a number of rapid growth startups with commercial distributions and complementary products, and has been a reliable distributed data platform for analytics. As Apache Hadoop adoption continues to grow, the larger Hadoop ecosystem is expanding, too. However, some debate remains about the future direction of the technology.

In this paper created by ESG Senior Analyst Nik Rouda, he discusses Apache Hadoop support from businesses, governments, academia, and technology vendors and how this large and diverse community differs in their specific goals and objectives for harnessing this technology.

Rouda dives into how ODPi is helping to bring maturity and choice to the Hadoop ecosystem in several ways, offering:

  • More confidence that Hadoop will remain a safe data platform choice for companies.
  • Simplified application and compatibility testing for third-party software developers.
  • Vendor-neutral coordination of efforts between vendors to build synergies across their offerings.

Download this free report to learn more about simplifying Apache Hadoop application development and portability.

Join author Nik Rouda, ESG Senior Analyst and ODPI Director John Mertic for a complimentary webinar Monday November 7 from 12-1 PM Eastern. All registrants will get a free copy of this valuable white paper.

Is Open Source Big Data a broken promise?

By | Blog

An article caught my eye this past week, where Robert Hof of SiliconAngle asserted that the challenges of Apache Hadoop adoption are a byproduct of the open source development approach. Hof argues that the various pieces do not integrate well together and some projects are not living up to their promises, which has resulting in additional work being required by organizations for them to see their true value. This has lead to a small pool of available talent and end-customers that are uncertain about where to direct their investments.

On the heels of this article, I watched the below video from Rakesh Kant of US Bank that I found just as insightful.

His sentiment rings loud and clear:

  • “I’m not seeing any signal, only noise.”
  • “The landscape is evolving into more experiments”
  • “A standard is required to help businesses”
  • “I’d like to focus time on delivering business value”

The Hadoop ecosystem has always been a technology focused one, and its clear this technology has been ground breaking and impactful. However, I do think that, over time, this technology has evolved to solve the needs of technologists. Enterprises have been largely been left without a voice and to struggle to embrace it with confidence.

In my view, open source as a development model is not the problem. Rather, it’s the lack of feedback from end-users like US Bank into the process. ODPi would like to solve this problem and help end-users share their feedback.

If you are an end-user of Hadoop, we’d love to have you as part of our End User Advisory Board to discuss these issues and help us focus on making adopting these technologies less risky for you.

ODPi Interoperable at Strata NYC 2016

By | Blog
By Susan Malaika, IBM ODPi member

ODPi Interoperable Solutions: Tested against ODPi Compliant Distributions

On September 29, 2016 I shared a session with my excellent colleague and IBM Fellow Berni Schiefer at Strata and Hadoop World Big Data Conference. I was substituting for John Mertic from the ODPi, a non-profit organization for the simplification & standardization of the big data ecosystem, as he was unable to participate. Strata occupied a large portion of Jacob Javits Conference Center in NYC, with many thousands of attendees, and a massive expo.

In the ODPi session we described the new concept of ODPi interoperable, announced on September 27, 2016, where big data solution and application providers can self-certify to be ODPi interoperable if they run their tests successfully against ODPi run-time compliant distributions. The benefit of ODPi interoperable is that when an application runs against one ODPi compliant distribution, it will run against all ODPi compliant distributions, therefore simplifying and reducing testing. A number of Hadoop applications were announced as being ODPi interoperable from Data Torrent, IBM, Pivotal, SAS, SyncSort, and WanDisco.

Berni talked about Big SQL which is one of the ODPi interoperable applications from IBM. Other ODPi interoperable applications from IBM includeAnalytic Server (SPSS), Big Replicate, and IDR for Apache Hadoop.

The audience in the session, which included representatives from banks and hardware companies, asked questions about the ODPi including how projects are added to the ODPi Runtime Specification 2.0. There is a nice description from Alan Gates of Hortonworks on how Apache Hive was selected by ODPi members to be added to the ODPi Runtime Specification 2.0. The audience also asked about participating in the ODPi, and were interested in seeing a roadmap for candidate projects for the Runtime Specification, beyond HDFS, Yarn, MapReduce, Hive & HCFS.

Call to Action: One of the ways that enterprises can participate in the ODPi is to join the ODPi User Advisory Board which will provide technical guidance and feedback on planned initiatives such as future projects to be incorporated into the ODPi. Ways individuals and companies can engage include:

The following is a recording of Berni Schiefer talking about ODPi and Big SQL. It was a wonderful experience to present alongside Berni.

Adding Apache Hive to ODPi Runtime Specification 2.0

By | Blog

By Alan Gates, ODPi technical steering committee chair and Apache Software Foundation member, committer and PMC member for several projects

Today, ODPi announced that the ODPi Runtime Specification 2.0 will add Apache Hive and Hadoop Compatible File System support (HCFS). These components join YARN, MapReduce and HDFS from ODPi Runtime Specification 1.0

With the addition of Apache Hive to the Runtime specification, I thought it would be a good time to share why we added Apache Hive and how we are strategically expanding the Runtime specification.

Why Hive?
ODPi adds projects to its specifications based on votes from ODPi’s diverse membership. We have a one member, one vote policy. In discussions regarding what projects to add to the next Runtime specification, many members indicated that they used Apache Hive, which is data warehouse software that facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Members indicated that by adding Apache Hive to the ODPi Runtime Specification 2.0, ODPi can reduce SQL query inconsistencies across Hadoop Platforms, which is one of the key pain points for ODPi members and Big Data Application vendors in general.

What is the process?
As with everything we do in ODPi, the addition of any project to the ODPi Runtime specification is done collaboratively, with participation from everyone who has interest. ODPi has established the Runtime Project Management Committee (PMC) to maintain the Runtime Specification.

In order to make sure all voices were heard and use cases considered, the Runtime PMC formed an Apache Hive working group. This group included Runtime PMC members, as well as other ODPi contributors who wanted to be involved. It included representatives from several distributors and application vendors, including: Hortonworks, SAS, IBM, Syncsort, and DataTorrent.

The working group came together over the course of a month, meeting regularly, to determine how to add Apache Hive to the spec.

What are we adding?
The working group decided early on to focus on SQL and API compatibility rather than matching a specific version of Apache Hive. We chose Hive 1.2 as our base version that distributions must be compatible with. This gives distribution providers freedom in what version of Hive they ship, while also guaranteeing compatibility for ISVs and end users.

What has to be compatible?
The working group focussed on interfaces that the ISVs and the distributors’ customers use most frequently. We agreed that SQL, JDBC, and beeline (the command line tool that allows users to communicate with the JDBC server) are used by the great majority of Hive users and so we included them in the spec. We also included the classic command line, the metastore thrift interface, and HCatalog as optional components; that is the distribution may or may not include them, but if it does they must be compatible. We chose to make these optional because they are frequently, but not universally, used.

Where can you see our work?
The initial draft of the Runtime PMC is open to the public and everything is published on Github.

How Can You Be Involved?
We are still writing tests for distributions to check that they comply with the specification. We would love to have your help writing tests. You can also give feedback on the spec. Participation in the ODPi is open to anyone, with all work being done is public on GitHub. Developers can join the conversation on the mailing lists or Slack channel.

My Experience at Global Big Data Summit: Discussing the Importance of Standards

By | Blog

I had a good day last week presenting to the audience at the Global Big Data Summit in Santa Clara. The tail end of the the last day of any conference is a bit slow, but was thrilled when many came barreling in right as I was ready to start working through my slidedeck which spoke to the point of the importance of standards, like ODPi, in driving future investment in Big Data and Apache Hadoop.

I had one critical question after the talk that I thoroughly enjoyed answering. A gentleman pushed back on my point that standards need to be the focus. In his experience, staff training and education were the biggest concerns and it didn’t make sense to focus on standards until a critical mass of developers and practitioners were properly trained first. It was a fair argument, and one that Gartner has found as a key blocker to Apache Hadoop growth as well, but to me one that tries to treat the symptom more than the core issue, and I pushed back saying that standards enables better education and enablement. My point made sense to him, but I walked away wanting to discuss this more in a blog with better data points behind it. After all, we are in the data industry here and should be data driven!

If there is one industry where standards are at the forefront, it’s education. Education standards are a very touchy subject (disclaimer here – I’m a parent of 4 school aged children and good friends with several educators) and while I’ll attempt to steer clear of the execution for this article, the concept of what is trying to be driven makes perfect sense. Does what skills a first grader have in one state equate with another state? What are reasonable benchmarks for defining competency? Can trends in learning/teaching method and outcomes be better correlated?

I came across an interview with a leader in educational standards entitled “How and Why Standards Can Improve Student Achievement: A Conversation with Robert J. Marzano”. The interviewee made some interesting insights which drew parallels to the critical question I received at the talk. Here’s a few quotes from the interview and their relation to Apache Hadoop standards:

“Standards hold the greatest hope for significantly improving student achievement. Every other policy mandate we’ve tried hasn’t done so. For example, right after A Nation at Risk (Washington, DC: U.S. Department of Education, 1983) was published, we tried to increase academic achievement by making graduation requirements more rigorous. That was the first wave of reform, but it didn’t have much of an effect.”

This makes a great point – creating a measuring stick for competency without some sort of standard to base education from hurts more than it helps.

The interviewer goes on to ask about what conditions are needed to implement standards.

“Cut the number of standards and the content within standards dramatically. If you look at all the national and state documents that McREL has organized on its Web site (, you’ll find approximately 130 across some 14 different subject areas. The knowledge and skills that these documents describe represent about 3,500 benchmarks. To cover all this content, you would have to change schooling from K–12 to K–22. Even if you look at a specific state document and start calculating how much time it would take to cover all the content it contains, there’s just not enough time to do it. So step one toward implementing standards is to cut the amount of content addressed within standards. By my reckoning, we would have to cut content by about two-thirds. The sheer number of standards is the biggest impediment to implementing standards.”

Lots and lots of content and things identified to learn across a diverse set of subject areas, with a finite time to turn out individuals competent in the space. Seem similar to the situation in the Apache Hadoop ecosystem?

The interviewer then follows up with asking how can you do this with knowledge continuing to expand.

“It is a hard task, but not impossible. So far the people we’ve asked to articulate standards have been subject matter specialists. If I teach music and my life is devoted to that, of course I’m going to believe that all of what’s identified in the national documents is important. Subject matter experts were certainly the ones to answer the question, What’s important in your content area? To answer the question, What’s absolutely essential? you have to broaden that population dramatically to include all constituents—those with and without college degrees.”

This response aligns very well with the ODPi approach to creating Apache Hadoop standards. We aren’t in the business of creating full end-to-end comprehensive standards of what an Apache Hadoop Platform should offer, or an Apache Hadoop-Native Big Data Application should adhere to, but instead focus on what’s truly important to provide that base level –  those essential pieces for what a platform should offer. And I particularly like the last point “expanding the scope of the conversation around standard to get diverse opinions and experiences,” which is something ODPi is uniquely positioned to drive.

One last quote, which I think shapes the “Why?” on this effort.

“Whether we focus on standards or not, we’re entering an era of accountability that has been created by technology and the information explosion.”

The enterprise has the same expectations – they want to lower risks in Big Data investments, which those risks are a byproduct of not having staff to manage them. Fortune 500 executives need this in place to have any confidence in this technology, which the abysmal adoption rates have shown to be problematic. In short, Apache Hadoop needs to be accountable for its enterprise growth.