MLtraq: Track your AI experiments at hyperspeed

Slides from my talk at Munich MLOps Community Meetup #7

Every second spent waiting for initializations and obscure delays hindering high-frequency logging, further limited by what you can track, an experiment dies. Wouldn't it be nice to start tracking in nearly zero time? What if we could track more and faster, even handling arbitrarily large, complex Python objects with ease?

In this talk, I will present the results of comparative benchmarks covering Weights & Biases, MLflow, FastTrackML, Neptune, Aim, Comet, and MLtraq. You will learn their strengths and weaknesses, what makes them slow and fast, and what sets MLtraq apart, making it 100x faster and capable of handling tens of thousands of experiments.

The talk will be inspiring and valuable for anyone interested in AI/ML experimentation and portable, safe serialization of Python objects.

Transcript

1. Michele Dallachiesa Data Products & AI Consulting michele.dallachiesa@sigforge.com MLtraq: Track your AI experiments at hyperspeed
2. Scope of this talk https://mltraq.com/ benchmarks/speed/ You will learn: • What is experiment tracking • What makes different frameworks fast and slow • How to select an experiment tracker for your projects
3. Experimentation • Definition: “The process of systematically changing and testing different input values in an algorithm to observe their impact on performance, behavior, or outcomes.”
4. Experiment tracking • Definition: “The process of recording the inputs, outputs, and performance metrics of an experiment.” • Examples: Code, notebooks, scripts, environment setup, parameters, configurations, evaluation metrics, model weights, system stats, inputs, outputs, accuracy, prompts, cost metadata, ...
5. Applications of experiment tracking • Explore and understand the impact on performance of different algorithms, parameters, and datasets • Automation and observability: live monitoring of long-term experiments, reproducibility, documentation, collaboration, ...
6. Modelling experiments • An experiment is a collection of runs • A run is an instantiation of the experiment with a fixed set of inputs
7. Why tracking speed matters: Initialization (1/3) • Slow imports negatively impact development, CI/CD tests, and debugging speed • High run initialization times impact on our ability to experiment with hundreds of thousands of runs Wouldn't it be nice to start tracking almost instantly?
8. Why tracking speed matters: High frequency (2/3) • At times, it's necessary to record metrics that occur frequently (loss, reward, state, ...) • Workarounds to handle too much information come at a complexity/completeness/accuracy cost: threading, downsampling, summarization, and histograms What if we could avoid these limitations altogether?
9. Why tracking speed matters: Large, complex objects (3/3) • Python data structures (dictionaries, lists, tuples), NumPy arrays, data frames, datasets, model weights, timeseries, forecasts, media files such as images, audio recordings, and videos, ... • Existing solutions are primitive and slow, using tech (JSON, uuencoding) from 25-40 years ago What if we could track more with less constraints?
10. • A new open-source experiment tracker designed to work with any SQL database, fast and interoperable • Serialization powered by native SQL database types, Numpy, PyArrow, and safe Python pickles • Funding: You can star the project on GitHub and/or hire me to make your experiments run faster
11. Tracking an experiment
12. Benchmarking experiment tracking frameworks Frameworks • Weights & Biases (0.16.3) • MLflow (2.11.0) • FastTrackML (0.5.0b2) • Neptune (1.9.1) • Aim (3.18.1) • Comet (3.38.1) • MLtraq (0.0.125) Latest update: 2024.03.06 Varying • Value type: float, ndarray • Count of values • Count of runs • Array length How • As MLtraq experiments! • 10 independent runs • Local storage
13. What takes most of the time? • W&B: threading, IPC • MLflow: Alembic migration • Aim: threading, RocksDB • Comet: threading • FastTrackML: fast but requires running server • MLtraq: SQLite operations • Neptune: direct writes to FS How much time to track 1 run and 1 value? Start up time Neptune vs W&B: 400x
14. MLflow: Alembic migration
15. • Entity-attribute-value database model with no batching kills MLflow/FastTrackML performance How much time to track 1 run and 100-10K values? 0.85 “accuracy” Experiment ID/name Source: https://community.intersystems.com/post/entity-attribute-value-model-relational-databases-should-globals-be-emulated-tables-part-1 High frequency tracking MLtraq vs MLflow: 355x
16. • Threading/IPC expensive for W&B How much time to track 10 runs and 1 value? MLtraq vs W&B: 1563x
17. How much time to track 1K runs and 1K values? What makes MLtraq faster • SQLite vs filesystem • Safe pickling vs JSON MLtraq vs Neptune: 23x
18. Source: https://www.sqlite.org/fasterthanfs.html
19. Python pickles harmless if limited to safe opcodes Source: https://infosecwriteups.com/vulnerabilities-in-python-serialization-pickle-d2385de642f6 and https://www.scaler.com/topics/pickle-python/
20. How much time to track 10^6 float64 values (8MB)? • MLtraq: Pickle, numpy.lib.format • W&B: wandb.Table, JSON format • Neptune: JSON, uuencoded binary blob • MLflow: mlflow.log_text, binary blob • FastTrackML: c.log_text, binary blob • Aim: run.track, binary blob • Comet: run.log_text, binary blob binary blob = weak semantics! MLtraq vs W&B: 113x Tracking large objects
21. • Write speed of np.zeros(size, dtype=np.int8) • Variants: MLtraq-fs vs MLtraq-db-mem vs MLtraq-db-fs How much time to track up to 10^9 int8 values (1GB)?
22. • Trade-offs: threading/IPC, data storage design, batching vs streaming • Uuencoding and JSON-like formats are slow with poor semantics, the future is native types with PyArrow • Beyond “tracking speed”: backward compatibility, cloud, backend, third-party integrations, reporting, complete model lifecycle management, ... • Disclaimer: lots of simplifications in these slides, check out full article and notebooks for details! Conclusion
23. Thank You! Michele Dallachiesa Data Products & AI Consulting michele.dallachiesa@sigforge.com