Pretraining: The Phase That Determines Everything an AI Will Ever Know

In 2022, DeepMind published a paper that quietly changed how the AI industry builds models. They trained a 70B parameter model and tested it against GPT-3, which had 175B parameters — a model 2.5x lar