I recently read the book Data Centric Machine Learning with Python. The thesis of the book is that Machine Learning (ML) pipelines benefit more from good data compared to good models, yet most of the focus of ML experts is on their models. The book is in four parts. Part 1 defines what the author means by Data Centric ML and the evolution of ML from model-centric to data-centric. Part 2 covers how to wotk effectively with Subject Matter Experts (SME). Part 3 covers programmatic techniques that ML practitioners should know to succeed with these Data-Centric pipelines. And Part 4 talks about ways in which you can implement Data Centric ML practice in your organization.
To be fair, I think ML practice has already shifted to being Data Centric for quite some time now. Some notable examples I can remember include the Netflix Prize (2007-2009), the ImageNet (2006-2010), and the release of Word2Vec (2013). The first led to the popularity of Collaborative Filtering (and unsupervised methods in general), the second was a very impressive effect of crowdsourcing and led to the achievement of super-human image recognition systems, and the third led to the popularity of semi-supervised algorithms, and eventually, Artificial Intelligence (AI) using Large Language Models (LLM) as we know it today.
The Internet contributed to the rise of Data Centric ML as well, as it gave rise to planet-scale companies that uses it to reach customers in every corner of the world, and generate enormous amounts of data as a result (the so-called Data Exhaust). Training ML models to handle critical functions using this data is not only cheaper than hiring an army of humans, it is often the only practical way to handle the volume.
Early ML efforts were mostly in academics, where humans (often graduate students) would label (relatively) small amounts of data that were later used to train small models. The practice continues even in today's data-rich corporations, where human Subject Matter Experts (SME) label samples of data that are used to train models that will potentially automate critical steps at scale in the company's pipeline. As models have become larger and more powerful their data needs have grown as well, as SME labels are often enhanced with raw data and synthetic labels, using weak or self-supervision, Transfer Learning, Active Learning, etc. Finally, with the advent of LLMs, we seem to have come full-circle, using LLMs as SMEs to generate labels for us.
I think this last point makes the book particularly relevant in today's AI assisted brave new world. LLMs are famously stochastic by design. Their predictions reflect the bias and inconsistencies of the data they were trained on, somewhat (although not exactly) like their human SME counterparts. The skills and techniques you would deploy to handle these biases and inconsistencies from your SMEs apply to a large extent to your LLM judges as well. In many organizations, interacting with SMEs is regarded as a specialized skill, so some ML engineers may not be familiar with it, but now everyone is expected to interact with LLM judges. Folks in this situation will find Part 2 useful.
Most modern organizations are awash with data. The problem is that most of that data is unusable as-is in an ML pipeline. Significant effort is needed to clean up the data to make it consistent, missing data may need to be imputed, features may need to be identified and extracted, data may need to be de-identified, systemic biases identified and removed, etc, etc. Another effort might involve recognizing patterns in data or using Weak Supervision to extract features from knowledge bases, writing labeling functions, starting with this noisy data to build a model, and iteratively using Human in the Loop Active Learning to continuously refine the data and build better and better models, the so called Data Flywheel. This is all covered in Part 3. Folks looking for techniques to work more effectively with data in your ML pipeline will find this part most useful. In addition, Transfer Learning, Adversarial Learning and Semi-supervised Learning are also covered here, which you can adapt as well.
Finally, Part 4 covers useful information about introducing Data Centric ML in your organization, including a checklist of steps you might want to take. This part is most useful if you are in a senior role and can drive or influence the direction of ML practice in your organization.
It is true that LLMs are nowadays so powerful that many tasks can now simply be done by writing a good prompt, tasks that used to require a few weeks of an ML engineer's time cleaning data, training a model, evaluating it, etc. However, I feel you can achieve a lot more if you think of an LLM as a component in a traditional ML pipeline, be it as a generator of labels or a source of weak supervision that allows you to build scalable models out of unlabeled or partially labeled data, and I think the contents of this book can help you get there.
