AI Seminar Series - Charles H Martin

WeightWatcher is an open-source tool, based on the research towards a new theory of learning, that can analyze a pre-trained and/or instruction-fined tuned LLM and provide diagnostic information as to how well each individual layer has 'converged'.

WeightWatcher is based on SOTA research into Why Deep Learning Works, using advanced techniques from statistical mechanics, random matrix theory, and strongly correlated systems. The theory formulates the DNN learning problem in terms of how the layers converge, and, in doing so, provides new and unique layer quality metrics that engineers can use to build better LLMs.

The tool and the theory, can be used to detect which layers are (potentially) underfit or overfit, to set the learning rates dynamically during training, to help compress LLMs, and fine-tune models more efficiently and effectively.