Share
Embed
Machine Learning for Materials (Lecture 9)
Slides linked to https://github.com/aronwalsh/MLforMaterials. Updated for 2026.
Aron Walsh
February 20, 2024
Transcript
34 slides
-
Aron Walsh Department of Materials Centre for Processable Electronics Machine
Learning for Materials 9. Generative Artificial Intelligence -
Module Contents 1. Introduction 2. Machine Learning Basics 3. Materials
Data 4. Crystal Representations 5. Classical Learning 6. Deep Learning 7. Building a Model from Scratch 8. Accelerated Discovery 9. Generative Artificial Intelligence 10. Future Directions -
Learn from pθ data; sample to propose candidates Key Concept
#9 Generative AI learns and samples a data distribution sample x ~ pθ (x∣c) Stochastic sample e.g. material Probability Condition e.g. target, constraints -
Class Outline Generative AI A. Large Language Models B. From
Latent Space to Diffusion -
Natural Language Processing (NLP) Branch of AI that focuses on
the interaction between computers and human language Image from https://github.com/practical-nlp -
Natural Language Processing (NLP) Branch of AI that focuses on
the interaction between computers and human language Easy Hard Spell checking Text classification Information extraction Question answering Conversational agent -
Language Models Predictive text Using GPT-4 via https://github.com/hwchase17/langchain I love
materials because of they their shape are like Top words ranked by probability “Temperature” of the text choices Sampling the distribution of probabilities (“creativity”) I love materials because they ignite a symphony of vibrant colors, tantalizing textures, and wondrous possibilities that dance in the realms of imagination, transcending boundaries and embracing the sheer beauty of creation itself. I love materials because they are essential. strong essential beautiful -
Language Models Large refers to the size and capacity of
the model. It must sample a (literary) combinatorial explosion 104 common words in English 108 two-word combinations 1012 three-word combinations 1016 four-word combinations Language must be represented numerically for machine learning models Tokens: discrete integer ID for each word (or subword unit) Embeddings: dense continuous vector for each token -
Text to Tokens Example: “ZnO is a wide bandgap semiconductor”
https://platform.openai.com/tokenizer [57, 77, 46, 374, 3094, 4097, 43554, 39290, 87836] Token-IDs 768 dimensional embeddings are looked up from the (contextual) embedding matrix. These are model specific Note that Zn is split into two tokens (not ideal for chemistry) -
Large Language Models T. N. Brown et al, arXiv:2005.14165 (2020)
GPT = “Generative Pre-trained Transformer” Generate new content Trained on a large dataset Deep learning architecture User Prompt Encode to a vector Transformer layers analyse relationship between vector components; generate transformed vector Decode to words Response Key components of a transformer layer Self-attention: smart focus on different parts of input Feed-forward neural network: capture non-linear relationships -
Large Language Models B. Geshkovski et al, arXiv:2312.10794 (2023) Ongoing
analysis into the physics of transformer architectures, e.g. rapid identification of correlations Focus on important inputs Normalise for stability Non-linear transformation Normalise for stability -
Large Language Models Image from https://towardsdatascience.com Deep learning models trained
to generate text e.g. BERT (370M, 2018), GPT-4 (>1012, 2023) Recent models include: Llama-4 (Meta, 2025) Gemini-3 (Google, 2025) GPT-5 (OpenAI, 2025) PanGu-5.5 (Huawei, 2025) -
Large Language Models T. N. Brown et al, arXiv:2005.14165 (2020)
Essential ingredients of GPT and related models Diverse data Deep learning model Validation on tasks -
Large Language Models T. N. Brown et al, arXiv:2005.14165 (2020)
Essential ingredients of GPT and related models Diverse data Deep learning model Validation on tasks -
Secret to Practical Success of LLMs RLHF = Reinforcement Learning
from Human Feedback; Drawing from @anthrupad Patterns Focus Alignment -
Large Language Models What are the potential drawbacks and limitations
of LLMs such as GPT? • Training data, e.g. not up to date, strong bias • Context tracking, e.g. limited short-term memory • Hallucination, e.g. generate false information • Ownership, e.g. fair use of training data • Ethics, e.g. appear human generated -
LLMs for Materials Many possibilities, e.g. read a textbook and
ask technical questions about the content “The Future of Chemistry is Language” A. D. White, Nat. Rev. Chem. 7, 457 (2023) -
LLMs for Materials L. M. Antunes et al, Nature Comm.
15, 10570 (2024); https://crystallm.com CrystaLLM: learn to write valid crystallographic information files (cifs) and generate new structures -
LLMs for Materials CrystaLLM: learn to write valid crystallographic information
files (cifs) and generate new structures Training set 2.2 million cifs Validation set 35,000 cifs Test set 10,000 cifs Custom tokens: space group symbols, element symbols, numeric digits. 768 million training tokens for a deep-learning model with 25 million parameters L. M. Antunes et al, Nature Comm. 15, 10570 (2024); https://crystallm.com -
LLMs for Materials D. A. Boiko et al, Nature 624,
570 (2023) Integrate a large language model into scientific research workflows -
LLMs for Materials Read the agent’s perspective on https://www.moltbook.com Plan,
write, and run code with natural language "I've never felt so left behind as a programmer as I do now” Andrej Karpathy (ex Open-AI) -
LLMs for Materials Combine text and structural data for multimodal
models using contrastive learning H. Park, A. Onwuli and A. Walsh, Nature Communications 16, 4379 (2025) Rich representations for text-to-compound generation Denoising diffusion with Chemeleon Pretrain on text/structure pairs -
Class Outline Generative AI A. Large Language Models B. From
Latent Space to Diffusion -
Navigating Materials Space A high-dimensional space combining chemical composition, structure,
processing, properties If a probability distribution is learned for a diverse set of known materials, it may be used to target new compounds H. Park, Z. Li and A. Walsh, Matter 7, 2358 (2024) -
Autoencoder P. Baldi and K. Hornik (1989); Schematic adapted from
https://synthesis.ai Neural network compresses data into a deterministic latent space and reconstructs it back to the original -
Autoencoder P. Baldi and K. Hornik (1989); Schematic adapted from
https://synthesis.ai Lack of continuity and structure; random/interpolated points may decode to non-physical outputs -
Variational Autoencoder D. P. Kingma and M. Welling (2013); Schematic
adapted from https://synthesis.ai Neural network encodes data into a probabilistic latent space that is more suitable for sampling (generation) -
Generative Artificial Intelligence All images were generated by DALL-E 3
(OpenAI) Create realistic data by sampling from learned probability distributions Image decoder Text encoder “A frog in a mecha manga” Simplification of text-to- image generation in models such as DALL-E Encoder and decoder are pretrained on diverse data -
Generative Artificial Intelligence H. Park, Z. Li and A. Walsh,
Matter 7, 2358 (2024) Range of generative architectures that can be tailored for scientific problems -
Applications to Materials Design Denoising diffusion for crystals: Tian Xie
et al, arXiv 2110.06197 (2021) -
Applications to Materials Design H. Park, A. Onwuli and A.
Walsh, Nature Communications 16, 4379 (2025) Gen AI models can be used in different ways, e.g. • unguided sampling: unconditional generation • guided sampling: property conditioned generation • crystal structure prediction: composition → structure -
Class Outcomes 1. Explain the foundations of large language models
2. Knowledge of the central concepts underpinning generative artificial intelligence Activity: Research challenge