Neural Thermodynamic Laws Emerge in Language Model Training

TL;DR;

This paper introduces neural thermodynamic laws, demonstrating how thermodynamic concepts naturally emerge in language model training. It presents a minimal river-valley model to produce experimentally testable predictions, studying fast and slow dynamics separately. The research establishes the equipartition theorem, computes the number of valley directions in language models, and explores the annealing process and Fourier's conduction law. It identifies that the learning rate has three roles: controlling temperature, entropic force, and time scale, and suggests future work in designing algorithms inspired by these insights to make LLM training more efficient.