Yesterday, It Fit: Dividing a 70B Model Across 64 GPUs

Yesterday, it fit. A smaller model, shorter examples, and a cluster that handled the job comfortably. Then the workload grew: 70 billion parameters, 32K-token sequences, the same 64 GPUs. The next tra