More Efficient Multi-Head Attention Implementations#
mha-implementations.ipynb contains and compares different implementations of multi-head attention
Summary#
The figures below summarize the performance benchmarks (lower is better).
Forward pass only#
Forward and backward pass#
Forward and backward pass after compilation#
