Two recent announcements have demonstrated to me why data automation is strategic for organizations that are preparing data for analysis – which should be every organization.
In the first, Talend announced that private equity firm Thoma Bravo has offered to acquire it for approximately $2.4 billion. That valuation represented a premium of approximately 29% to Talend's closing price on March 9 and an 81% premium to the volume weighted average price over the previous twelve months.
The second was publication of the 2021 Data Science Interview Report. This report, published on the Interview Query Blog, analyzed over 10,000 data science related interview experiences and concluded that “Data Engineering is the new Data Science” – specifically because over the previous year, data engineering interviews had grown by more than 40%. Data science interviews had grown by 10%, down from 80% growth the year before.
What do these two seemingly disconnected facts have to do with demonstrating the strategic importance of data automation?
Let’s start by looking at the growth in demand for data engineers. It has been an axiom in data science for a few years that data scientists spend as much as 80% of their time doing data preparation – which includes identifying, collecting, cleansing, and aggregating data. I previously wrote about this in a post called Reversing The 80-20 rule in data wrangling for AI and machine learning.
Gartner recently stated in a report called the The State of Metadata Management: Data Management Solutions Must Become Augmented Metadata Platforms that “Clients now report approximately 90% or more of their time is spent preparing data (as high as 94% in complex industries) for advanced analytics, data science and data engineering.”
To me, the growth in job postings clearly demonstrates that the 80/20 rule in data science remain a significant issue. So, companies are hiring dedicated resources to do data engineering, rather than have it continue to be a large portion of data science roles. The growth in the data engineering specialization is a clear indicator that the industry needs to adopt data automation to improve productivity around data integration, data preparation and data management.
The problem is that it has been demonstrated time and again in the tech industry that you cannot just throw more bodies at a problem – the mythical man month is real! Imagine if a company could reduce the need to hire additional data engineers but could instead dramatically increase the productivity of existing data engineers and data scientists when it comes to preparing data for analysis. A technology that could accomplish this would indeed be of strategic importance and value. This is what data automation provides!
And what about the recent acquisition announcement? What does one private equity acquisition in the data management space imply regarding the strategic value of data automation? It turns out that it isn’t just one acquisition. In the last few years, Thoma Bravo has made several acquisitions in the data management space. And they are just one of several private equity firms investing in this technology area.
But why all the investment? Private equity firms, venture capitalists and other investors typically look at where they believe there is a growing need that will drive companies to spend money on certain technologies in the future – say in the next one to five years. They then purchase or invest in technology companies that are providing key solutions in these areas.
If investment in data automation and data management companies is accelerating – it is likely because investors believe there is and will be growing demand for solutions that automate getting data ready for analysis.
So the fundamental question businesses should consider is this: why is data automation strategic for organizations? The answer is clear, because there is a pressing need to improve and accelerate the ability to prepare data for analysis – in every industry – and hiring more data engineers will not solve the problem in the long run. Organizations need to invest in data automation in order to keep up with their internal demand for data. I would argue that this belief is reinforced as evidenced by the number of acquisitions of data automation and data management companies by investors.
Would you like to learn more about data automation? Check out this industry analyst report published by 451 Research.