GPU Kernels, Explained: CUDA, Triton, cuBLAS, torch.compile and KernelBench

When you run a neural network in PyTorch, Python is only the coordinator. Behind every tensor operation — from a simple activation to a multi-head attention block — sits one or more GPU kernels: speci