Data Scientists Versus Domain Experts - Who Will be the Key to Better AI?
Is the data science skill set short-lived? As we look to the future of AI, data scientists will need to expand their domains – and industry experts could turn out to be the best data scientists of all.
Ten years down the line, there is some speculation that the data scientist’s role may be short-lived.
In a 2012 Harvard Business Review article, Thomas Davenport and DJ Pati claimed, Data Scientist: Sexiest Job of the 21st Century. In 2022, the authors posted again with the title: Is Data Scientist Still the Sexiest Job of the 21st Century?” They didn’t take a position on this, but they made some useful commentary:
“The role was relatively new at the time. Still, as more companies attempted to make sense of big data, they realized they needed people who could combine programming, analytics, and experimentation skills. At the time, that demand was primarily restricted to the San Francisco Bay Area and a few other coastal cities. Startups and tech firms in those areas seemed to want all the data scientists they could hire. We felt the need would expand as mainstream companies embraced business analytics and new forms and volumes of data.”
The prototypical data scientist in 2012 was recruited from the ranks of PhDs from fields like physics, statistics or math or other quantificational disciplines with experience in experimentation in their respective areas of research. As the saying goes, their skills were necessary but not sufficient. A significant data science project, from end-to-end, involves too many steps and too many skill domains for one person. Often responsible for the entire effort, the data scientist was required to provide expertise in the following areas:
- Statistical analysis and computing
- Machine learning
- Deep learning
- Processing large data sets
- Data visualization
- Data Wrangling
- Mathematics
- Programming
- Statistics
- Exploratory data analysis
- Visualization of results for presentation including BI, Powerpoint and in-person presentation
From The future of Data Scientists: Will the next decade witness a complete extinction?
What does Data Science imply?
“For starters, Data Science is the art of extracting relevant information, major insights, and knowledge from a given set of data. Hence, the result of any data science project is mostly an array of slides or a PPT that basically concludes the entire scenario for business heads to make decisions, or for a group of technical and product experts to conclude how to work on a site.”
The key steps directing the workflow of any data science project are mainly:
- Collecting a lot of data (for verified sources)
- Analyzing data and iterating it multiple times to develop a sound understanding of the data
- Suggesting hypotheses or actions that include periodically analyzing and updating data.
- Organizations currently look towards setting up a wholesome Data Science framework by hiring large and consistent deep-seated teams, benchmarked by setting up industry guidelines.
As a result of the likelihood of a data scientist being proficient in more than a few of these requirements, according to Noah Gift:
Many data science teams have not delivered results that can be measured in ROI by executives. For complex data engineering tasks, you need five data engineers for every one data scientist.
The pace of automation software is pretty dramatic and will affect the nature of data science work, including machine learning. Every major cloud vendor has heavily invested in some type of AutoML initiative. A data scientist is no longer characterized by coding skills, which is confirmed by the rising prominence of no code, auto ML arrangements like DataRobot, Amazon, Dataiku, Google Cloud, Databricks, H20, Rapid Miner, and Alteryx. Does this mean data science is a poor career choice? Without a doubt, the position is evolving. Data scientists should look toward improving their skills in things that are not automatable:
- Communication skills
- Applied domain expertise
- Creating revenue and business value
The squeezing recruiting process has moved to critical thinkers, analysts, and problem solvers who comprehend all the nuances of the business, its area of expertise, and its various collaborators. Hence, the mere know-how of several software packages or the ability to disgorge a couple of lines of code would not get the job done.
The only sure thing is change, and changes are coming to data science. While the job title data scientist will recede, the data scientist’s work will be distributed to machine learning engineers, data engineers, AI wranglers, AI communicators, AI Ethics managers, AI product managers, AI Production administrators, and AI architects. What current data scientists should do to stay relevant is to embrace soft skills. They should deviate from tasks that can be easily automated — feature engineering, exploratory data analysis, trivial modeling – and turn to tasks that defy automation and produce a AI system that has a measurable business impact with verifiable business metrics and revenue enhancement.
Companies that want to be ahead of the curve can embrace the pragmatism and automation of machine learning tasks to gain a clear strategic advantage.
My take – what can be concluded on the fate of data scientists?
Obviously, data science can never go extinct. However, the titular positions and roles of data scientist will surely notice a dynamic change. In another decade or so, data-savvy industry domain specialists, business analysts, and field experts will become data science experts by getting trained in AI and ML.
These specialists will be able to permeate analysis with their deep-rooted industrial knowledge, regardless of if they can code. Their titles will mirror their aptitude instead of the methods they execute.
I don’t believe data scientist was the sexiest job of the 21st century in 2012, and it isn’t now. In fact, we are already seeing a drop in employment ads for data scientists and a corresponding increase in advertisements for the decomposition of the original conception of data science.
