Book Review: Software Engineering for Data Scientists

As a Software Engineer (backend Web Development then Search) turned Data Scientist, I was particularly interested in what the book Software Engineering for Data Scientists by Andrew Treadway had to say about the reverse transition. Transitioning between sub-disciplines is a given in our industry -- I started life as a sales/support engineer, then moved to application programming, then back and forth between architect, programmer, part-time sysadmin and full-time DBA, before getting into backend Web Development with Java. Nevertheless, the shift from that into Data Science (DS) has been the most challenging for me. Having lived through the time when Data Scientist first became a job title, to the genesis and evolution of Deep Learning to Transformers to Large Language Models to Agents, the field continues to be a moving target, growing and changing at breakneck speed.

Most applications today incorporate a healthy dose of Data Science based components. As a result, Data Scientists are increasingly being integrated into these teams and are expected to work collaboratively within team frameworks. The book addresses this new requirement in four parts -- the first part covers information that Data Scientists transitioning into such teams would need to know to get going, the second part covers scaling to larger datasets and compute clusters, the third part covers issues around production deployments, and the fourth covers monitoring. Here is my somewhat detailed, chapter by chapter review of the book.

Part I: Getting Started

Chapter 1: Software Engineering Principles -- Josh Wills, an early DS practitioner and evangelist, famously defined a Data Scientist as someone better at Statistics than a Software Engineer and better at Software Engineering than a Statistician. I think, as the field has matured over time and tooling has improved to incorporate the necessary statistics, the bar around Software Engineering (SE) has gone even higher. This chapter describes a typical DS workflow, with EDA / Data Validation, Data Cleaning, Feature Engineering, Model Training, Evaluation, Deployment and Monitoring, and how having DS with good SE skills result in better code structure, code collaboration, efficient scaling and testing, and easier deployments.

Chapter 2: Source Code Control for Data Scientists -- the author describes git, a distributed source code control system (and currently the de-facto standard) and common git commands for typical DS / SE work and introduces the reader to the feature branch workflow. I noticed that the author did not cover data version control systems such as dvc, but that could be because nowadays many companies prefer central data catalogs where the DS no longer needs to worry about versioning.

Chapter 3: Code Structure and Style -- nowadays it is possible to enforce a common coding style across an appliation using tools such as pylint and black. The chapter introduces these tools and talks about the PEP-8, the Style Guide for Python Code. The author provides some additional general guidelines such as modularizing code to avoid repetition (DRY). It also goes into some additional details such as incorporating type safety into Python using mypy, exception handling, and creating documentation from inline comments using pdoc (there is also the less capable but built-in pydoc).

Chapter 4: Object Oriented Programming for Data Scientists -- this chapter covers basic concepts of Object Oriented Programming (OOP) such as classes, methods instances, constructor, etc., and provides an example of using OOP in a Machine Learning (ML) based pipeline based on scikit-learn that demonstrates how OOP can improve code modularity. Having come from a Java / Spring background, I would have liked to see some discussion of Dependency Injection (DI) with Python here, but I guess this may be something the DS is expected to pick up as they get more familiar with SE.

Chapter 5: Creating Progress Bars and Timeouts in Python -- even though it may feel a bit strange to see these two items lumped into their own chapter, it makes sense when you realize that DS jobs are typically long running batch jobs, and the ability to show progress and stopping long running jobs with degraded performance are both quite important. The chapter shows how to use tqdm to show progress in your Python code, and how similar functionality is integrated into scikit-learn. It also covers how to respond to timeouts using the stopit package.

Part II: Scaling

Chapter 6: Making your Code Faster and More Efficient -- there is lots of good information here, some of which I knew and some that I didn't. It starts by introducing the Big O notation, then showing how you can profile a block of code using the kernprof line profiler and the @profile decorator. It also describes several strategies for making your code faster, such as replacing loops with select on Pandas dataframes, parallelizing with Numpy, avoiding Pandas apply, using list comprehensions and numpy.vectorize functions. It also introduces multi-processing using the built in multiprocessing library in Python and the n_jobs parameter in Scikit-Learn. It touches on Multithreading and Asynchronous Programming as possible additional techniques to address slow code but does not go into details. It introduces caching using functools.lru_cache and @lru_cache decorators. Finally, it describes some useful built-in Python data structures like set and Priority Queue, and the Numpy array, which uses vectorized operations internally.

Chapter 7: Memory Management with Python -- another useful chapter for me. It covers the use of the Python memory profiler guppy and the @profile decorator. It also discusses memory management strategies for Pandas and Scikit-Learn (using model.partial_fit()), using Numpy arrays in favor of Python lists, and Parquet as a more memory efficient alternative to CSV files.

Chapter 8: Alternatives to Pandas -- this chapter covers Dask and PySpark, two popular "big-data" libraries that work with Dataframes and distribute the workload across a cluster of machines, and support datasets too large to fit into RAM. Both do lazy evaluation, unlike Pandas which does eager execution. Examples are provided for both Dask and PySpark. The chapter also mentions the modin package, which allow you to create custom Pandas operations that delegate to Dask or Ray (another big data platform). It also mentions Polars, a Pandas-like package written in Rust for speed.

Part III: Deploying to Production

Chapter 9: Putting your Code into Production -- this chapter talks about various strategies for making the results of your DS artifact (e.g. a trained model) available to consumers. The first strategy covered is the simple recurring batch job. An important consideration is protecting user credentials, so strategies such as keyrings are discussed. Another slightly more advanced approach is to create a REST API, with tools such as FastAPI and uvicorn highlighted in the examples. Another strategy discussed is to create a high level CLI to help users to call your model without knowing too much about the internals.

Chapter 10: Testing in Python -- while unit testing is very important in the SE context, DS has traditionally not been very strict about this. But there is value in testing DS pipelines and config files as well, to ensure that all supported edge cases work correctly. Unit testing packages such as pytest and unittest are discussed, as well as the test coverage tool coverage.

Chapter 11: Scheduling and Packaging your Code -- this chapter covers scheduling your DS pipeline on Windows and Unix, packaging code with build and twine so application code can call your code as a local library, creating desktop based executables with PyQt and pyinstaller. I found this chapter particularly informative, since previously I had been exposing my DS artifacts using APIs and Streamlit. Always good to learn new ways to do things.

Chapter 12: Reporting and Logging in Python -- covers customizing logging formats so application logs can be parsed to produce useful insights about runs. Additional material includes generating PDF reports using reportlab and sending them automatically over email. I prefer markdown reports rendered on the user's browser, with notifications sent via email, but PDF looks interesting as well.

Part IV: Monitoring

Chapter 13 - Introduction to Web Development for Data Science -- this is a generic chapter on web development using Flask because the author feels (rightly) that DS should be capable of building simple web applications, and provides an example of building a web application that helps with ML model training. However, I feel that perhaps this chapter should have gone into an Appendix along with the Dask appendix. Monitoring is covered in some depth in the previous Part already.

Appendix: Dask with Coiled and AWS -- covers using Dask with the Coiled tool on the Amazon Web Services (AWS) cloud platform.

Overall, I thought the book provided good value. It is interesting how much of a head start I got as a SE first. However, in keeping with the grass is greener mindset, I feel that the move from DS to SE is probably less of a hurdle than in the other direction, but I will defer to those who have made the move in this direction.

Trip Report: PyData Global 2025

I attended PyData Global 2025 earlier this month. I had hoped to write this up earlier, but I've been busy, so only now getting the time Christmas morning. Merry Christmas to all my readers and best wishes for a Happy New 2026, hopefully it will be even better and more exciting (on the technology front) than this one! Taking stock of this year earlier today, I think I have some serious catching up to do in terms of reading about new stuff that just happened while I was busy doing other things. So hopefully I should have some writeups about them here in the coming year, although I am aware I have made similar promises earlier and broken them.

Anyway, back to PyData Global. It was held over 3 days December 9-11 and the baseline timezone was UTC, so for me the talks started very early in the morning (2:30-3:30 am) and ended at midday (1:30 pm on the first day and 11 am on the other two). So I ended up watching a lot of recordings. Basically I would attend the talks that were live past 6-7 am my time and then loop back to watch the recordings of the ones I missed from earlier in the day. Since I was watching a lot of recordings, it was tempting and easy to skip over prologue that speakers need to include in their presentation to ensure level setting with everyone in the audience, and I am afraid I succumbed repeatedly to that temptation. I was also multi-tasking with some work stuff, which meant I ended up picking and choosing more than I otherwise would (in a "real" physical conference).

Here are the talks I attended and my take aways from them.

Day 1

Scaling Fuzzy Product Matching with BM25: A Comparative Study of Python and Database Solutions -- this attempts to solve a product name matching problem, where the same product can be referred to by slightly different names. The strategy is to use BM25 search (available to DuckDB) to find similar names, reducing an O(n²) problem to a much smaller one, and finally using Dask and cuDF to merge the data. I also learned about the bm25s package for sparse BM25 matching in Python.

Lessons learnt in optimizing a large-scale Pandas application using Polars, Fireducks and cuDF -- nice coverage of optimization strategies for DataFrames, the presenter calls them T1 (replacing for-loops with iterator-loops), T2 (replacing loops with vector operations) and T3 (strategically filtering before applying join or aggregate functions), and compares the performance of different DataFrame handling packages. The speaker talks about the strengths of each library relative to Pandas (lazy mode and multi-threading for Polars and Fireducks, GPU parallelism for cuDF) and finds that performance of Polars and Fireducks on his dataset is better than Pandas because of multi-threading and best on cuDF because of GPU parallelism.

From Feature Engineering to Context Engineering for Agents -- the speaker makes the argument that Context Engineering for Agent based applications is the same as Feature Engineering for more traditional Machine Learning (ML) applications. The example he cites is Retrieval Augmented Generation (RAG) systems, where the retrieved context is used for in-context learning to help the LLM Agent return better generations. The speaker is also the author of Building Machine Learning Systems with a Feature Store, which he offered a free download of to the audience (and which I have downloaded and look forward to reading once I have some time).

Python Worst Practices: Learn from the Expert -- very entertaining talk about what not to do when building Data Science applications. To be fair, the practices he highlights are not all that uncommon, and underscores why the ability to think in terms of the domain rather than algorithms is so important.

Text Mining Orkut's Community Data with Python: Cultural Memory, Platform Neglect and Digital Amnesia -- Orkut used to be Google's answer to Facebook (and MySpace) but it never took off in the US. It was more popular in Brazil and India, until Google pulled the plug on it. The speaker is from Brazil and he describes his project to text mine Orkut to analyze how and why it failed. Even if you don't care about the history of Orkut, the talk is worth it if you are curius about the text mining and visualization techniques used it it. The repository behind the talk is at rodrigosf672/orkut-pydataglobal2025 on GitHub.

Using traditional AI and LLM to automate complex and critical documents in Healthcare -- description of a case study using Clinical Trials data from a Project Manager's point of view. Lot of useful lessons for someone looking to implement an AI solution in Healthcare.

Why Julia's GPU Accelerated ODE Solvers are 20x-100x Faster than JAX and Pytorch -- I don't use Julia, but may someday. However I am intrigued with the idea of applying ODE solvers to non-neural optimization problems as well. The talk goes into a lot of detail around Julia's ODE solver and how it is superior (in terms of scope and performance) to the ones built into JAX or Pytorch.

Where have all the Metrics gone? -- the speaker makes the point that traditional metrics are still relevant inthe age of AI, except that wrongness is now multi-dimensional, i.e. the LLM can make mistakes in more than one way, often at the same time. She then goes on to describe different kinds of failure modes for LLMs and classifies them as Domain Failure, Form Failure, Mode Collapse, Consistency Failure, Boundary Failure and Temporal Failure, and suggests a pragmatic way to manage these failures by ranking and measuring them separately. The application needs to be structured so wrongness of multiple components can be measured separately. She advocates for using traditional metrics as well as coming up with new ones.

The Boringly Simple Loop Powering GenAI Apps -- very nice talk that attempts to unify different AI architectures as variations of a nested two-loop pipeline. Speaker shows how simpler pipelines such as RAG or workflow systems are just specializations of the general pipeline. Along the way he also talks about the advantages and disadvantages of each architecture. Definitely worth watching if you are interested in AI architectures.

When AI Makes Things Up: Understanding and Tackling Hallucinations -- the speaker talks about why LLMs hallucinate, and strategies that the developer can adopt to alleviate where possible. She also talks about how to detect hallucinations using both human and LLM based oversight (consistency checks), how to estimate model confidence (2 approaches requiring access to token statistics and measuring output variance which does not need this). She also covers some high level strategies to prevent hallucinations.

Day 2

PyData/Sparse and Finch: extending sparse computing in Python ecosystem -- this is mostly about Finch, a sparse tensor compiler written in Julia, which can be accessed from Python for Sparse Array programming. It creates an intermediate notation (finch assembly code) that can translate to the underlying architecture (CPU, GPU, etc).

How to effectively use text embeddings in tree based models -- Tree based algorithms (Random Forest, XGBoost, etc) typically work with data decomposed into low-dimensional feature vectors, where these features are usually manually selected. Using embeddings directly as feature vectors would result in very deep trees and overfit. So the solution proposed is to use the embeddings to build multiple feature predictors, outputs of which would be used to create the feature vectors for the tree based model. The speaker demonstrates this technique using a StackingRegressor to create a 2-layer model ensemble. This technique can help with feature generation and results in explainable models.

Bayesian Decision Analysis with PyMC: Beyond AB Testing (Downey) -- this is a 90 minute workshop on using PyMC for doing Bayesian Decision Analysis. Specifically he attempts to do Bayesian AB testing of digital marketing strategies. This is a very hands-on session, where attendees are guided through various modeling approaches using PyMC. All notebooks are available at AllenDowney/BDAWithPyMC on GitHub. Great session, as it is always with Dr Downey's talks. I plan to go back to this again in the future.

UQLM: Detecting LLM Hallucinations with Uncertainty Quantification -- the speaker introduces their package UQLM for Uncertainty Quantification. They define hallucination as non-factual content that sounds plausible, which is impossible to prevent at scale using Human-in-the-loop (HITL) strategies. Their solution is to quantify the uncertainty of the model during text generation. Their package offers black-box and LLM-as-judge scorers that can work without requiring access to the token statistics, and white-box scorers that do. The project is hosted at cvs-health/uqlm on GitHub.

Lessons in Decision Making from the Monty Hall Problem -- The Monty Hall problem illustrates why probability is so non-intuitive. However, the speaker illustrates how extending the problem from 3 doors to N (where N >> 3) can make it less of an edge case and much more intuitive. I thought this approach might be useful as something applicable to other situations as well. He also covers applications of this kind of thinking in industry.

Let Me Structure Freely? How to Improve LLM Structured Output Quality -- I have been working mostly with Anthropic models which don't have as much formal support for Structured Input and Output as OpenAI's models. So the dependence on Structured I/O was new to me, prompting me to Google this separately (and apply it to cases where I am working with OpenAI's models). But apart from the benefits of Structured Output, the author also talks about an extension to the DSPy library (not yet merged) called StructureOfThought that allows for structured chain of thought like introspection in LLMs for reasoning problems.

Optimal variable binning in Logistic Regression -- the speaker introduces variable binning for Logistic Regression. The idea is to discretize continuous variables into categorical bins. Computing the weight of evidence per bin, or the information value globally across all bins, can provide a useful feature selection metric to decide if the variable is predictive or not. The binning criteria is an optimization problem to maximize a specific metric such as GINI or Information Value. The speaker reports that optimal binning on age feature resulted in best results for his application. More details on the guillermo-navas-palencia/optbinning.

Decisions under uncertainty: A Hands-on Guide to Bayesian Decision Theory -- Bayesian Decision Theiry is all about picking the action that optimizes the expected utility or cost. In its simplest form, it involves defining each possible action at each state, and estimating the utility / cost of each action based on your domain priors, and choosing the action that leads to the highest expected utility. Predictive models can also be Bayesian since the threshold of a particular utility is specific to the domain. Probabilities can also be estimated by a distribution where exact values are unknown, and Gaussian processes can be used to optimize costs. The speaker covers applications of Bayesian Decision Theory such as Hyperparameter Optimization and Experiment Design.

From Pandas to Policy as code; the future of ML Data Engineering (keynote) -- this was a keynote presentation containing lot of good general advice for Data Engineers. The gist of the advice is to minimize data movement, by processing data at the point of generation, only shipping the result of the processing rather than the entire payload. Indirectly, this is also an argument for efficient edge processing. Once processing is done, the raw data can be archived using a slower process since it is no longer time-sensitive. This also allows pipelines to compliant with regulations such as GDPR and minimizes exposure. The message is to apply data policy at the source.

Day 3

Revolutionalizing Safety Log Analysis in Oil and Gas: A Multi-Stage LLM Approach for Enhanced Hazard

How big are SLMs -- I have been interested in the possibility of deploying multiple special purpose Small Language Models (SLMs) in place of a single general purpose LLM driven by prompts, so I thought this talk might be interesting, and I was not dissapointed. The speaker defines SLMs as models with 1M-10B parameters and references some popular SLM (Phi-4, Mistral Small 3, Gemma, Llama 3.2, SmolLM v2, Qwen2) which I plan on exploring further. She enumerates some popular approaches for fine-tuning SLMs, both at the model and data level. She also mentions the possibility of distilling LLMs to SLMs, specifically Llama to BabyLlama. I thought it was a very good overview. If someone went ahead and went down all the rabbit holes the talk covered, one would have a very comprehensive and useful book on SLMs.

Beyond Just Predictions: Causal Thinking in Machine Learning -- this talk introduces Causality in Machine Learning, where you want to estimate the effect given the data. It describes a few approaches to estimate this in a focused manner, such as Uplift Modeling. The speaker covers Conditional Average Treatment Effect (CATE) and how to estimate this using Meta-Learners, the type of Meta-Learners (S-Learner where one predicts the effect with and without the treatment and computes the lift, and T-Learner to capture heterogeneous treatment effects, where one predicts the effect with different levels of treatment and computes the diff).

Detecting Regime Shifts in Time Series with Python: Entropy based Change Point Detection -- detecting changes in a time series where the change cannot be explained by randomness is the goal of change point detection. It has applications in anomaly detection, quality control, data drift, etc. Changes can be in the average, variance or frequency. The speaker describes some techniques to do this such as periodic sliding window stats and metrics to measure it (KL Divergence for continuous variables and Pearson distance for discrete). There is discussion on estimating the optimal kernel width and threshold.

Overall, I thought it was a good conference. I got to hear about cool things that the Python Data Science community did, and got a few ideas that I would like to try out for my own applications. The talks listed above are the ones I attended, if you have favorites and you don't see it listed here, please let me know so I can check it out.

EDIT 2026-01-09: presentation videos are now available on Youtube!

Book Report: Pandas Workout

Unlike many Data Scientists, I didn't automatically reach for Pandas when I needed to analyze data. I came upon this discipline (Data Science) as a Java Software Engineer who used Python for scripting, so I was quite comfortable operating on JSON / CSV / text files directly, loading data into relational databases and running SQL against them, and building visualizations with Matplotlib. So when Pandas first hit the scene, I thought it was a nice library, but I just didn't see the logic in spending time to learn another interface to do the same things I could do already. Of course, Pandas has matured since then (and so have I, hopefully), and when faced with a data analysis / preparation / cleanup task, I often now reach out not only for Pandas, but depending on the task, also its various incarnations such as PySpark, Dask Dataframes and RAPIDS cuDF. When I use Pandas (and its various incarnations) I often find myself depending heavily on Stack Overflow (and lately Github Copilot) for things I know can be done but not how. To some extent I blame this on never having spent the time to understand Pandas in depth. So when I was offered the chance to review Pandas Workout by Reuven Lerner, I welcomed it as a way to remedy this gap in my knowledge.

The book is about Pandas fundamentals rather than solving specific problems with Pandas. For that you will still want to look up Stack Overflow :-). In fact, in the foreword the author specifically targets my demographic (needs to look up Stack Overflow when solving problems with Pandas). But he promises that after reading the book you will understand why some solutions are better than others.

Pandas started as an open source project by Wes McKinney, and has grown somewhat organically into the top Data Science toolkit that is today. As a result, there are often multiple ways to do something in Pandas. While all these ways may produce identical results, their performance characteristics may be different, so there is usually an implicit "right" way. The book gives you the mental model to decide which among the different approaches is the "right" one.

The book is organized into the following chapters. Each chapter covers a particular aspect of Pandas usage. I have included a super-short TLDR style abstract for each chapter for your convenience.

1. Series -- Pandas Series objects are the basic building block of Pandas and represent a typed sequence of data, that are used to construct DataFrames and Indexes. Many methods on the Series object apply in a similar way to DataFrames as well. This is a foundational chapter, understanding this will help with future chapters.
2. Data Frames -- DataFrames represent tabular data as a sequence of Series, where each Series object represents a column in the table. Pandas inherits the idea of DataFrames from R, and the incarnations I listed (and a few that I didn't) use DataFrame as a basic abstraction as well. This chapter teaches you how to select from and manipulate DataFrames. Unless you've used Pandas extensively before, there is a high chance you will learn something useful new tricks here (I did, several of them).
3. Import and Export -- covers reading and writing CSV and JSON formats to and from DataFrames. Covers some simple sanity checks you can run to verify that the import or export worked correctly. I learned about the pd.read_html method here, probably not that useful, but interesting to know!
4. Indexes -- Indexes are used by Pandas to efficiently find data in DataFrames. While it may be possible to get by without Indexes, your Pandas code would take longer to run and consume more resources. The chapter deals with indexing techniques. I happened to know a lot of them, but there were a few that I didn't, especially the techniques around pivot tables.
5. Cleaning -- this chapter teaches a skill that is very fundamental to (and maybe even the bane of) a Data Scientist's job. Statistics indicate that we spend 80% of our time cleaning data. Along with the techniques themselves (remove / interpolate / ignore), this chapter contains commentary that will help you frame these decisions on your own data cleaning tasks.
6. Grouping, Joining and Sorting -- these three operations are so central to data analysis, so much so that SQL has special keywords for each operation (JOIN, GROUP BY and ORDER BY). This chapter covers various recipes to do these operations efficiently and correctly in Pandas.
7. Advanced Grouping, Joining and Sorting -- this chapter goes into greater detail on how to combine these operations to deal with specific use-cases, the so-called "split-apply-combine" technique, including the concept of a general aggregation function agg. It also shows how to do method chaining using assign.
8. Midway Project -- describes a project and asks questions that you should be able to answer from the data using the techniques you have learned so far. Comes with solutions.
9. Strings -- one reason I don't have much experience with Pandas is because it is focused on numeric tables for the most part. However, Pandas also has impressive string handling facilities via the str accessor. This chapter was something of an eye-opener for me, showing me how to use Pandas for text analysis and pre-processing.
10. Dates -- this chapter describes Pandas date and time handling capabilities. This can be useful when trying to work with time series or when trying to derive numerical features from columns containing datetime objects to combine with other numeric or text data.
11. Visualizations -- this chapter describes visualization functionality you can invoke from within Pandas, that are powered either by Matplotlib or Seaborn. This is more convenient than exporting the data to Numpy and using the two packages to draw the charts.
12. Performance -- performance has been a focus for most of the preceding chapters in this book. However, the recipes in this chapter are in the advanced tricks category, and include converting strings to categorical values, optimizing reads and writes using Apache Arrow backed formats, and the using fast special purpose functions for specific purposes.
13. Final Project -- describes a project similar to the Midway project with questions that you should be able to answer from the data using the techniques you have learned so far.

I think the book has value beyond just teaching Pandas fundamentals though. The author sprinkles insights about Data Analysis and Data Science throughout the book, around learning to structure the problem and planning the sequence of steps that are best suited for the tools at hand, the importance of critical thinking, the importance of knowing the data and interpreting the results of the analysis, etc.

Each exercise (there are 50 in all) involves downloading some dataset, dealing with subjects as diverse as tourism, taxi rides, SAT scores, parking tickets, olympic games, oil prices, etc. I think the information about the availability of such datasets (and possibly related datasets) can also be very valuable to Data Scientists for their future projects.

I think the popularity of Pandas is because of the same reason as the popularity of Jupyter Notebooks. It is a nice, self-contained platform the allows a Data Scientist to demonstrate a series of data transformations from problem to solution in a clear, concise and standard manner, not only to customers, but to other Data Scientists as well. More than any other reason, I feel that this will continue to drive the popularity of Pandas and its various incarnations, and as a Data Scientist, it makes sense to learn how to use it properly. And the book definitely fulfils its promise of teaching you how to do that.