The recent decision of IBM to expand its partnership with Hortonworks is another sign that big data is maturing. The hype is yielding to reality and investors want to see returns on their investments. It’s also a “canary in the coal mine” warning to the emerging AI industry. 

Hortonworks is a pioneer in the big data field, and its Hadoop platform has been widely adopted. IBM’s decision to partner more deeply with Hortonworks is a perfect example of the old adage, “If you can’t beat ‘em, join ‘em.”

When viewed from a purely strategic perspective, IBM’s move seems logical.  Since the days of Lou Gerstner, Big Blue has been steadily trying to move up the value ladder, shedding its traditional role as a maker of foundational tech and emerging as a savvy provider of high-level business services. 

But in some instances, IBM has been unable to resist the urge to move backward and climb down into the weeds. I personally do not believe that Hadoop will ever become a commonly used business tool. That said, it’s an essential part of the plumbing, so to speak. My guess is that Hadoop will become a commodity. IBM probably arrived that the same conclusion, which is why it’s apparently pulling the plug on its own Hadoop platform.

Why is that an important lesson for the nascent AI industry? Here’s why: Fairly soon, there will be companies working on foundational AI technologies and companies working on high-level AI services for business. There will be consolidations at both ends of the AI spectrum. The foundational tech will become a commodity, but the higher level business services will continue to drive value.

For the present, you can’t do AI without hiring armies of data scientists. What we often refer to as “AI” is really a combination of technologies and techniques, including big data, machine learning, natural language processing, neural networks, deep learning, cognitive learning and associative learning. So far, there’s no real “AI in a box;” you have to assemble it yourself. 

For most companies, AI isn’t worth the effort – at least not yet. By next year at this time, however, things might be different. There will be hundreds of companies offering AI-driven business services and business tools. There will be a race to the top, not to the bottom. 

Remember, the value is usually at the top of the pyramid. There’s nothing intrinsically wrong with Hadoop – it’s just not where the value is. The AI industry will soon come to a similar realization.