Book Review: Time Series Forecasting using Foundation Models

As someone who primarily works in NLP and Search in the Health Domain, I don't have much use for Time Series. However, while exploring the Financial domain based on personal interest, I have been curious about Time Series for some time. Recently I attended the OpenHPI course Time Series Analysis taught by Mario Tormo Romero (even did the quizzes and the certificate of completion!). I was familiar with traditional techniques such as ARIMA (and its derivatives), but the course also covered Neural Network based techniques using CNN and RNN architectures, as well as some Transformer based models such as N-BEATS, Autoformer, Informer and TFT. Overall, I loved the course and learned a lot from it. If I had to complain, it would be to point to the lack of practical code examples and/or exercises, but I suppose it may not be that hard to Google (or now ChatGPT) that stuff on my own.

As I get older, I find I learn faster using what I know already to create analogies for what I am learning rather than starting from scratch. So it seemed to me that there is some similarity between predicting the next word in a sentence and predicting where a stock price will be headed next week given its previous history. Thus methods useful in NLP, including the relatively cutting edge methods around Transformers and Generative AI, could, at least in principle, be applicable for Time Series forecasting. Of course, NLP involves discrete entities, i.e words in a vocabulary, while Time Series involve continuous values, so there are bound to be differences as well.

So when I came across Marco Peixeiro's Time Series Forecasting using Foundation Models I was actually quite intrigued (sorry if I sound Victorian, but thats the closest word I can think of to indicate the mixture of vindication and curiosity I felt when I saw the title). Being a relative outsider to the world of Time Series forecasting, I felt vindicated that there is a research community that is actually looking at this connection, and was also curious to see where they had taken it. So I read the book and here is what I learned.

High level feedback -- overall, this book fulfils the promise it makes in its title, and then some. It covers 7 different Foundation Models (loosely speaking, some of these are more methodological framework than model) covering encoder-only, encoder-decoder and decoder-only (and even a couple of Mixture of Experts) models. In each of these model specific chapters, it provides code examples for using in zero-shot mode and fine-tuning where applicable. For models that produce point estimates, it demonstrates cross-validation based methods to produce a forecast distribution, as well as code for anomaly detection where applicable. Over the course of these seven chapters, it contrasts and compares these models with each other, so by the end of the book, the reader has a good grasp of what each model can or cannot do, and where they might shine. There is also a capstone project with a different dataset which serves to cement the reader's understanding of these various models. I think the material is not only comprehensive, but also prepares you to intelligently follow advances in the field of Time Series forecasting using Foundation Models, which is important given that it is still a relatively nascent and fast-growing field.

Detailed per chapter feedback -- the book is organized in three parts (four if you include the Capstone Project which is really a large exercise). Part 1 is mostly background, Part 2 covers 5 models specifically developed for Time Series forecasting, and Part 3 covers 2 models where the Time Series task is converted to a Language Task and a LLM used to handle it.

Part 1

• Chapter 1: Understanding Foundation Models -- covers the Transformer architecture, with detailed coverage of its building blocks. Of note is the coverage of positional embeddings, which becomes even more crucial in the context of Time Series (an meaningless stream of numbers rather than a semi-meaningful stream of words). It also covers why (and why not) one would want to use Foundation Models for Time Series forecasting. --
• Chapter 2: Building Foundation Models -- covers the N-BEATS model architecture. N-BEATS was also one of the models covered towards the end of the OpenHPI course, so this represents a sort of progression towards the use of FMs for Time Series forecasting. In addition, it covers different evaluation metrics used in this area, and the effect of forecasting horizons on performance.

Part 2

• Chapter 3: Forecasting with TimeGPT -- covers the TimeGPT model, an encoder-decoder model that can predict future values in an univariate Time Series with exogenous variables. Code examples that illustrate how to use this model for zero-shot forecasting as well as fine-tuning, as well as cross-validation over different forecasting horizons and anomaly detection.
• Chapter 4: Zero Shot Probabilistic Forecasting with Lag-LLaMA -- this is an open-source model built on top of the decoder-only LLaMA model from Meta. It supports univariate Time Series only, and is trained using lagged values of many different Time Series to create features. Lag-LLaMA provides probabilistic forecasts rather than point predictions. Code examples similar to the previous chapter are also provided.
• Chapter 5: Learning the language of time with Chronos -- this chapter covers Chronos, a framework that allows using T5 and GPT-2 like language models with Time Series data. It describes various techniques like as mean scaling, mixup (convex combinations of multiple Time Series) and KernelSynth for data augmentation. The framework yields probabilistic forecasts as well, and median is usually used for point predictions if needed. As in previous chapters, code examples for zero-shot forecasting and fine-tuning, as well as cross-validation and anomaly detection are provided.
• Chapter 6: Moirai a Universal forecasting Transformer -- Moirai is an encoder only model, provides probabilistic forecasts, and supports exogenous features out of the box. It uses a technique called patching to combine multiple consecutive inputs into a single element, similar to how one might use n-grams in NLP, which allows it to capture local semantic meaning and support longer context lengths. The output is sent through a linear projection layer. Moirai comes in two flavors, this one and Moirai-MoE, a mixture-of-experts version which is based on a decoder-only Transformer model.
• Chapter 7: Deterministic Forecasting with TimesFM -- TimesFM produces determinisitic point predictions rather than a probabilistic forecast. It cannot be used for anomaly detection since we cannot construct confidence interfavals directly. One innovation with TimesFM is the use of residual blocks. The output is in the form of patches which goes through a linear layer to produce the final prediction. Exogenous variables are supported through the use of additional regression model. Unlike the other chapters, this does not cover fine-tuning since that requires JAX and was considered out of scope for the book (but maybe its a good reason to learn JAX?).

Part 3

• Chapter 8: Forecasting as a Language task -- this chapter covers PromptCast, another technique that turns the Time Series forecasting task into a language task. The LLMs used here are Flan-T5 and LLaMA 2.3 3B-instruct. Essentialy it consists of creating prompts that specify an input sequence, optionally describing the task and asking the LLM to provide the next value. The chapter illustrates using zero-shot, few-shot and chain of thought prompting. The approach is likened to the Pudding mit Gabel festival, where people use forks to eat pudding.
• Chapter 9: Reprogram an LLM for forecasting -- this chapter covers TimeLLM, another framework that reframes a Time Series forecasting task as a language task. It uses patches and reprogramming it by running it through a vocabulary, along with a prompt, as input, and a linear layer to produce the prediction from the learned embeddings. Training involves updating the weights of the patch reprogramming and linear layers. While it produces point predictions, it can be used for anomaly detection by using cross-validation to generate forecasts across multiple time horizons.

Part 4

• Chapter 10: Capstone Project -- forecasting daily visits to a blog -- the chapter provides the dataset and asks to build models that predicts future daily visits. The provided solution starts with a SARIMA baseline, then uses the different models that the book discussed, to produce better and better predictions.

So there you have it. As I have mentioned earlier, I found this book quite useful, not only in its coverage of various models and how it is used for time series, but also as a primer to follow research progress in this field. Hopefully you found this review helpful and I hope this book will serve you as well as it has served me.

Book Review: Hands-On Artificial Intelligence for IoT

For those in similar professional circles as I am in, i.e. looking forward into the Generative AI space, yet with one foot pragmatically and firmly stuck in Machine Learning (ML) and Deep Learning (DL) techniques of the (recent, ok, not very distant) past, you will find Dr Amita Kapoor's recent book Hands-On Artificial Intelligence for IoT: Expert Machine Learning and Deep Learning Techniques for developing smarter IoT systems, 2/ed published by PackT a very useful resource into the use of these techniques applied to applications in the Internet of Things (IoT) domain. My own interest in IoT is driven primarily by previous personal (and failed) forays into Home Automation, but I do have some background in ML and DL techniques. So I approached this book from the perspective of a reader trying to understand the challenges and applications of these techniques in the IoT domain. This perspective shaped my reading of the book, and to some extent this review as well, as I looked for insights that would help me bridge my existing knowledge with the nuances of the IoT domain.

The book is organized into 4 parts. The first part introduces foundational techniques that are common to both the fields of AI (this term includes ML and DL) and IoT, while the second part covers advanced techniques. The third part focuses on specific IoT applications and AI techniques to handle them, while the fourth part covers IoT applications at different levels of granularity (personal/home, industrial, smart cities, etc.). The book is quite large (approximately 400 pages) and covers a lot of ground, some of which you may already be familiar with depending on your background. However, even in those cases, it may be worthwhile to skim the text to make sure you don't miss something you didn't know about, since things move quickly in this field. In any case, I present below my summary of each chapter, organized into a loose table of contents type structure. Hopefully they help you make the decision to read versus skim and optimize your reading experience.

• Part I: Principles and Foundations of IoT and AI
  • Principles and Foundations of IoT and AI -- covers the theoretical foundations of IoT (think ISO network stack), various applications, and the necessity of using Big Data techniques and ML. It concludes with a list of tools used in the text, which includes Keras3.0 to support DL in IoT applications.
  • Data Access and Distributed Processing for IoT -- this chapter covers processing data in various formats (text, CSV, Excel, JSON, HDFS, and various SQL and NoSQL databases) using Python. This is because IoT devices often present data in proprietary formats, and you need to be able to read it into your application.
  • Machine Learning for IoT -- covers traditional ML algorithms such as Naive Bayes, Logistic Regression, Decision Trees, SVM, etc (remember my quip about having one foot firmly in the distant ML past? This is about as far back you would go), and one example using a simple DL model. Even though these may not be on par with more recent models such as BERT or small LLMs, these are typically deployed for solving simpler problems and have lower latency requirements, and are often adequate for the problem at hand.
• Part II: Advanced AI Techniques and their application in IoT
  • Deep Learning for IoT -- introductory DL chapter, covers DL basics, CNN, RNN and AutoEncoders. It also provides a brief description of OpenVINO for IoT vision applications and TinyML for low-power on-device analytics, and using Keras Tuner for Hyperparameter Tuning.
  • Techniques for IoT -- explores alternative optimization techniques to Gradient Descent (GD) such as Simulated Annealing and Swarm Optimization. Also covers the use of Evolutionary and Genetic Algorithms (EA and GA) using libraries such as PyGAD and DEAP. While not mentioned explicitly, I will guess that EA/GA are included here because they are less resource intensive compared to GD, and can often be more efficient depending on application.
  • Reinforcement Learning for IoT -- this chapter covers the basics of Reinforcement Learning (RL), Q-Learning (DQN, DDQN, Policy Gradients, etc). As before RL based training can be particularly suitable for IoT applications because they are physics based and reinforcement signals can be cheaper to obtain and more relevant compared to supervision signals.
  • Generative Models for IoT -- this chapter covers Generative Adversarial Networks (GAN) and Variational AutoEncoders (VAE), which are probably not the Generative Models you had in mind if you are in the current "GenAI" space, but these are the OG models that generate images from noise (rather than the next token from a stream of tokens). Primrily their utility in the IoT space seems to be data generation and simulation (GAN) and anomaly detection (VAE).
• Part III: Implementing Intelligent IoT Solutions in Diverse Domains
  • Distributed Learning using Keras -- this chapter covers Distributed training using Keras3 (using the JAX backend). This is useful information if you were just curious about Keras3 distributed capabilities. The relevance of this to the IoT space is that training data may be aggregated from multiple edge devices, say for recommendations, or multiple resource constrained edge devices may be used to retrain on new data, such as maintenance models in industrial IoT systems.
  • AI Cloud Platforms for IoT -- covers the need for Cloud based APIs in the context of IoT, and IoT adjacent services provided by popular providers such as AWS, Azure and Watson. Also covers these providers from the point of view of ML services, including Google VertexAI and AutoML, AWS SageMaker and Bedrock, and IoT specific services such as AWS IoT Core, Azure IoT Hub and GCP IoT code.
  • Deep Learning for Time Series Data from IoT -- covers working with time series data using traditional algorithms such as Prophet and Spark-ML, and wirth recurrent neural networks (RNN), and using pre-trained Temporal Convolutional Networks (TCN) models such as Chronos. This is particularly relevant since IoT devices emit streams of data over time that can be analyzed and extrapolated to predict the future.
  • Leveraging AI for Visual Data from IoT -- covers the processing of visual data from IoT systems, including image segmentation and object detection and classification. Architectures covered include CNN, TCN, and ViT (Visual Transformers).
  • AI for Text, Audio and Speech Data from IoT -- IoT devices can listen for particular sounds or speech patterns in their input, so this chapter covers mechanisms for IoT devices to process speech and audio, as well as free-form text input from users.
• Part IV: Applying AI and IoT in Real-World Scenarios
  • AI for Personal and Home IoT -- mainly covers Personal and Home IoT applications, and considerations for creating them, along with a case study on a Smart Home implementation. It also includes pointers on getting started on your own IoT projects.
  • AI for IIoT -- there are already many IoT applications in use in industrial environments, and this chapter describes instances of these in various industries. Application areas are not only in manufacturing support, but could also be for preventative maintenance and forecasting load.
  • AI for Smart Cities IoT -- I felt initially that this may a bit of an aspirational chapter, in the sense that the typical reader of this book is unlikely to be in a position to influence the use of AI for smart cities, but the examples proved me wrong. Many of these are examples of smart solutions to everyday problems that are well within the realm of influence of people working for cities or local government, directly or indirectly.

In summary, I found this book to be a comprehensive resource to understand the concepts behind IoT applications. It's breadth of coverage is truly impressive -- spanning essential principles of IoT and AI, traversing through machine learning, deep learning, and optimization techniques, and culminating in thorough discussions on real-world deployments across domains such as smart homes, industrial IoT, and smart cities. While the book’s extensive coverage of fundamentals in areas like machine learning and distributed processing may at times feel broader than strictly necessary for readers already well-versed in these fields, it ensures that the material remains accessible to a broader spectrum of readers.

The progression of chapters from core principles to practical case studies equips readers with a strong theoretical foundation as well as a practical understanding of how intelligent systems can be implemented in the IoT space. The inclusion of dedicated chapters on time series analysis, computer vision (CV), and Natural Language and Audio processing, offer readers additional perspective in these areas. While I don't see an IoT applications in my immediate future, it was an interesting read, and having read it, I feel more confident about being able to tackle one should it come about.