Pruning for Large Language Models — From SparseGPT to KV-Cache Pruning

Perplexity is the exponentiated average negative log-likelihood. Intuitively, a perplexity of \(k\) means the model is “as uncertain as if it were choosing uniformly among \(k\) options at each step.” Lower is better.

Numerical example: If a dense LLaMA-7B model achieves PPL = 5.68 on WikiText-2, and a 50% sparse version achieves PPL = 5.95, the relative increase is:

This is generally considered acceptable. An increase to PPL = 7.0 (23% relative) would be considered a significant degradation. An increase beyond PPL = 10 typically indicates catastrophic quality loss.

SparseGPT (Frantar & Alistarh, 2023) — Complete Deep Dive

#

SparseGPT is the foundational algorithm for one-shot LLM pruning. It demonstrated for the first time that GPT-scale models (up to 175 billion parameters) can be pruned to 50-60% sparsity in a single pass, without any retraining, while maintaining competitive perplexity.

Problem Statement

#

Given a pre-trained weight matrix \(W \in \mathbb{R}^{d_{\text{row}} \times d_{\text{col}}}\) and a calibration dataset producing input activations \(X \in \mathbb{R}^{d_{\text{col}} \times N}\), find a sparse weight matrix \(\hat{W}\) that minimizes the reconstruction error:

This is a layer-wise objective: we prune each weight matrix independently, aiming to preserve the layer’s input-output behavior on the calibration data.

The Optimal Brain Surgeon (OBS) Framework

#

SparseGPT builds upon the Optimal Brain Surgeon (OBS) framework. The idea is: when we prune (set to zero) a single weight \(w_q\), we should adjust all remaining weights to compensate for the error introduced. This is fundamentally different from magnitude pruning, which simply removes weights without compensation.

Derivation from First Principles

#

Consider the second-order Taylor expansion of the loss around the current weight vector \(w\):

where \(g = \nabla_w \mathcal{L}\) is the gradient and \(H = \nabla_w^2 \mathcal{L}\) is the Hessian. At a (local) minimum, the gradient \(g \approx 0\), so:

We want to find the weight update \(\delta_w\) that minimizes this loss increase, subject to the constraint that weight \(q\) is pruned (set to zero):

where \(e_q\) is the unit vector with a 1 in position \(q\). The constraint says: the new weight at position \(q\) must be zero, i.e., \(w_q + \delta_{w_q} = 0\), meaning \(\delta_{w_q} = -w_q\).

Solving via Lagrange Multipliers

#

Form the Lagrangian:

Take partial derivatives and set them to zero:

Condition 1 — \(\frac{\partial \mathcal{L}}{\partial \delta_w} = 0\):

Condition 2 — \(\frac{\partial \mathcal{L}}{\partial \lambda} = 0\):

Now substitute (1) into (2). The \(q\)-th component of \(\delta_w\) is:

Substituting into (2):

Substituting (3) back into (1):

This is the OBS weight update formula. When we prune weight \(q\), we shift every remaining weight by an amount proportional to the corresponding column of \(H^{-1}\).

Saliency Score

#

The increase in loss from pruning weight \(q\) (with optimal compensation) is obtained by substituting the optimal \(\delta_w\) back into the objective:

Using \(\delta_w = -\lambda H^{-1} e_q\):

Substituting \(\lambda = w_q / [H^{-1}]_{qq}\):

This score tells us the cost of pruning weight \(q\) — weights with low saliency should be pruned first. Note this differs from simple magnitude pruning (\(|w_q|^2\)) by the factor \(1/[H^{-1}]_{qq}\), which accounts for the curvature of the loss landscape. A weight might be large but cheap to prune if the Hessian indicates the loss is flat in that direction.

Hessian Computation for Transformers

#

For a linear layer \(y = Wx\) with squared-error reconstruction loss \(| WX - \hat{W}X |_F^2\), the Hessian with respect to a single row \(w_i\) of \(W\) is:

where \(X \in \mathbb{R}^{d_{\text{col}} \times N}\) is the matrix of input activations (\(N\) calibration samples). This is because the loss for row \(i\) is:

and \(\nabla^2_{\delta_{w_i}} \ell_i = 2 X X^T\).

This is a critical simplification: the Hessian is the same for every row of the weight matrix, and it depends only on the input activations, not on the weights themselves. This means we compute \(H = 2XX^T \in \mathbb{R}^{d_{\text{col}} \times d_{\text{col}}}\) once and reuse it for every row.

In practice, \(H\) is regularized to ensure invertibility:

The Row-Wise Pruning Trick

#

A key insight in SparseGPT is that the pruning problem decomposes across rows. Since each row \(w_i\) of \(W\) produces one output dimension independently, and the Hessian is shared, we can process each row separately:

┌─────────────────────────────────────────────────────────────┐
│                Weight Matrix W (d_row x d_col)               │
│                                                             │
│  Row 0:  [ w_00  w_01  w_02  ...  w_0,d_col ]  ──► Prune  │
│  Row 1:  [ w_10  w_11  w_12  ...  w_1,d_col ]  ──► Prune  │
│  Row 2:  [ w_20  w_21  w_22  ...  w_2,d_col ]  ──► Prune  │
│    ...        ...                                           │
│  Row m:  [ w_m0  w_m1  w_m2  ...  w_m,d_col ]  ──► Prune  │
│                                                             │
│  Each row is pruned independently using the SAME H^{-1}    │
│  (shared Hessian depends only on input activations X)       │
└─────────────────────────────────────────────────────────────┘

For each row, we iterate through columns from left to right. At each column, we decide whether to prune it (based on saliency) and, if so, apply the OBS update to all remaining columns in that row.

Lazy Batch Updates

#

Processing columns one at a time is computationally expensive because each OBS update modifies all remaining weights. SparseGPT introduces lazy batch updates: process columns in blocks of size \(B\) (typically \(B = 128\)).

Within a block of \(B\) columns, pruning decisions and local weight updates are computed. The global update to all remaining columns (beyond the current block) is accumulated and applied once at the end of the block. This reduces the number of expensive full-row updates from \(d_{\text{col}}\) to \(d_{\text{col}} / B\).

Let the current block span columns \(\mathcal{B} = {b, b+1, \ldots, b+B-1}\). Partition the weight row \(w\) and the inverse Hessian accordingly. Within block \(\mathcal{B}\):

1. For each column \(q \in \mathcal{B}\): compute saliency, decide whether to prune, and apply the OBS update only within the block.
2. After processing all \(B\) columns: apply the accumulated update to all columns in \({b+B, b+B+1, \ldots, d_{\text{col}}-1}\) in a single matrix operation.

The accumulated update for the remaining columns is:

This can be computed as a matrix multiply, making it efficient on modern hardware.

Cholesky Decomposition for Efficient Updates

#

Rather than explicitly computing and inverting \(H\), SparseGPT uses the Cholesky decomposition of \(H^{-1}\). Since \(H\) is symmetric positive definite (after regularization), we can write:

where \(L\) is lower triangular. This provides an efficient way to extract the quantities we need:

• \([H^{-1}]_{qq}\) is simply the squared norm of the \(q\)-th row of \(L\).
• The column \((H^{-1})_{:,q}\) can be extracted from \(L\).

As we process columns left to right and “eliminate” pruned columns, we can update the Cholesky factor incrementally rather than recomputing from scratch. This is mathematically equivalent to a Cholesky downdate operation.

Full Algorithm Pseudocode

#

Algorithm: SparseGPT (per weight matrix W, given input activations X)
────────────────────────────────────────────────────────────────
Input:  W ∈ R^{d_row × d_col}    — pre-trained weight matrix
        X ∈ R^{d_col × N}         — calibration activations
        s ∈ (0, 1)                — target sparsity ratio
        B                         — block size (default 128)

1.  Compute Hessian: H = 2 * X * X^T + ε * I
2.  Compute Cholesky of H^{-1}: L = cholesky(H^{-1})
3.  Initialize: Err = zeros(d_row, B)  // error accumulator
4.  For b = 0, B, 2B, ..., d_col - B:         // block loop
5.      For j = b, b+1, ..., b+B-1:            // column loop within block
6.          // Pruning decision for column j (across all rows)
7.          // Option A: global threshold on saliency
8.          // Option B: per-row top-k sparsity (prune s fraction)
9.          For each row i where w_{ij} is selected for pruning:
10.             Err[i, j-b] += w_{ij} / [H^{-1}]_{jj}
11.             w_{ij} = 0
12.         // Update remaining weights in current block
13.         W[:, j+1:b+B] -= Err[:, j-b:j-b+1] * H^{-1}[j, j+1:b+B] / [H^{-1}]_{jj}
14.     // Lazy batch update: propagate error to all remaining columns
15.     W[:, b+B:] -= Err * H^{-1}[b:b+B, b+B:]
16. Return W (now sparse)
────────────────────────────────────────────────────────────────

Complexity Analysis

#

• Hessian computation: \(O(d_{\text{col}}^2 \times N)\) — matrix multiply \(XX^T\).
• Cholesky decomposition: \(O(d_{\text{col}}^3)\).
• Per-row pruning: \(O(d_{\text{col}}^2)\) due to the block updates.
• Total per weight matrix: \(O(d_{\text{row}} \times d_{\text{col}}^2)\) for the pruning loop, plus \(O(d_{\text{col}}^3)\) for the Hessian inverse. Since \(d_{\text{row}}\) and \(d_{\text{col}}\) are typically both \(O(d)\), the total is \(O(d^3)\) per weight matrix, and \(O(L \times d^3)\) for the entire model.

For LLaMA-7B (\(d = 4096\), \(L = 32\), 7 weight matrices per layer): approximately \(32 \times 7 \times 4096^3 \approx 1.5 \times 10^{13}\) FLOPs. On a single A100 GPU, this takes approximately 10-15 minutes — remarkably efficient for pruning a 7B model.

Numerical Example

#

Let us work through a complete example with a small \(4 \times 4\) weight matrix. Consider:

Suppose from calibration data we compute:

(We drop the factor of 2 for simplicity and absorb it into the regularization.)

First, compute \(H^{-1}\) (with slight regularization):

Target: 50% sparsity (prune 2 out of 4 weights per row).

Process Row 0: \(w_0 = [0.5, -0.3, 0.8, 0.1]\)

Compute saliency scores:

• \(S_0 = \frac{0.5^2}{2 \times 0.556} = \frac{0.25}{1.112} = 0.225\)
• \(S_1 = \frac{(-0.3)^2}{2 \times 0.767} = \frac{0.09}{1.534} = 0.059\)
• \(S_2 = \frac{0.8^2}{2 \times 0.342} = \frac{0.64}{0.684} = 0.936\)
• \(S_3 = \frac{0.1^2}{2 \times 1.133} = \frac{0.01}{2.266} = 0.004\)

Sorted by saliency: \(S_3 = 0.004 < S_1 = 0.059 < S_0 = 0.225 < S_2 = 0.936\)

Prune columns 3 and 1 (lowest saliency). Start with column 3 (\(w_{0,3} = 0.1\)):

Update: \(w_0 \leftarrow [0.5 - 0.0025, -0.3 + 0.0252, 0.8 + 0.0007, 0] = [0.4975, -0.2748, 0.8007, 0]\)

Now prune column 1 (\(w_{0,1} = -0.2748\)), using the updated inverse Hessian (with column 3 removed):

Final Row 0: \(\hat{w}_0 = [0.4975 - 0.0573, \ 0, \ 0.8007 - 0.0079, \ 0] = [0.4402, \ 0, \ 0.7928, \ 0]\)

The same process repeats for Row 1. The key observation is that pruning with OBS compensation preserves the row’s output much better than simply zeroing the weights. Without compensation, Row 0 would be \([0.5, 0, 0.8, 0]\), but with compensation it becomes \([0.4402, 0, 0.7928, 0]\) — the remaining weights are adjusted to partially absorb the error from the pruned weights.

SparseGPT Results

#

Key observations: (1) Larger models are easier to prune — LLaMA-65B loses only 2% PPL at 50% sparsity. (2) Beyond 60% unstructured sparsity, quality degrades rapidly. (3) 2:4 structured sparsity is only slightly worse than unstructured at the same density.

Wanda (Pruning by Weights AND Activations, Sun et al., 2023) — Complete Deep Dive

#

Motivation

#

SparseGPT achieves excellent results but requires computing and inverting the Hessian matrix \(H \in \mathbb{R}^{d_{\text{col}} \times d_{\text{col}}}\), which for LLaMA-7B means storing and decomposing a \(4096 \times 4096\) matrix per layer. While feasible, this is computationally non-trivial and requires careful numerical implementation (Cholesky decomposition, lazy updates, etc.).

Wanda asks: can we achieve comparable pruning quality with a much simpler metric?

Key Insight: Weight Magnitude Alone Is Insufficient

#

Classical magnitude pruning ranks weights by \(|W_{ij}|\) and prunes the smallest ones. This ignores a crucial factor: how much the input actually activates each weight. Consider two weights:

• Weight A: \(|W_A| = 0.01\) connected to a feature with \(|X_A|_2 = 100\). Output contribution: \(0.01 \times 100 = 1.0\)
• Weight B: \(|W_B| = 0.5\) connected to a feature with \(|X_B|_2 = 0.01\). Output contribution: \(0.5 \times 0.01 = 0.005\)

Magnitude pruning would keep Weight B and prune Weight A. But Weight A contributes 200x more to the output! This discrepancy is especially severe in LLMs because of activation outliers — a few features have enormous activation norms.

The Wanda Pruning Metric

#

Wanda’s metric combines weight magnitude with input activation norm:

where:

• \(W_{ij}\) is the weight connecting input feature \(j\) to output neuron \(i\)
• \(X_j \in \mathbb{R}^{N}\) is the vector of activations for feature \(j\) across all \(N\) calibration tokens
• \(|X_j|2 = \sqrt{\sum{n=1}^{N} x_{j,n}^2}\) is the L2 norm across calibration samples

Weights with low scores are pruned. The intuition is clear: a weight is unimportant if it is small, or if the feature it connects to is rarely activated, or both.

Mathematical Justification

#

Consider the output of a linear layer for a single output neuron \(i\):

The error introduced by pruning weight \(W_{ij}\) (setting it to zero without compensation) is:

The squared error, aggregated over \(N\) calibration samples, is:

Taking the square root to get a more stable metric:

This is exactly the Wanda score. It is a first-order approximation of the pruning error for each individual weight, without considering inter-weight correlations (which is what the Hessian captures in SparseGPT).

To make this more rigorous, note the connection to the diagonal of the Hessian. For the row-wise squared-error loss, the Hessian is \(H = 2XX^T\). The diagonal entry is:

The OBS saliency score (from SparseGPT) is:

If we approximate \(H\) as diagonal (i.e., ignore off-diagonal correlations between features), then \([H^{-1}]{jj} = 1/H{jj} = 1/(2|X_j|_2^2)\), and:

This is exactly the square of the Wanda score. Therefore, Wanda can be understood as the OBS framework under a diagonal Hessian approximation — it ignores feature correlations but captures the first-order importance of each weight.

Per-Output Pruning

#

Wanda prunes within each row (output neuron) independently. For a target sparsity of \(s\), in each row of \(d_{\text{col}}\) weights, the \(\lfloor s \times d_{\text{col}} \rfloor\) weights with the lowest Wanda scores are set to zero. This ensures uniform sparsity across output neurons, which is important for maintaining balanced representations.

Algorithm

#

Algorithm: Wanda (per weight matrix W, given input activations X)
────────────────────────────────────────────────────────────────
Input:  W ∈ R^{d_row × d_col}    — pre-trained weight matrix
        X ∈ R^{d_col × N}         — calibration activations
        s ∈ (0, 1)                — target sparsity ratio

1.  Compute activation norms: a_j = ||X_j||_2  for j = 1, ..., d_col
2.  Compute score matrix: S_{ij} = |W_{ij}| * a_j
3.  For each row i = 1, ..., d_row:
4.      Find the indices of the ⌊s × d_col⌋ smallest entries in S[i, :]
5.      Set W[i, those indices] = 0
6.  Return W (now sparse)
────────────────────────────────────────────────────────────────

This is remarkably simple: the entire algorithm is essentially 5 lines of code. No Hessian computation, no Cholesky decomposition, no iterative updates. The only computation beyond the weight matrix itself is a single pass over the calibration data to compute \(|X_j|_2\).

Speed Comparison

#

Wanda is approximately 20-30x faster than SparseGPT while achieving comparable quality at moderate sparsity levels.

N:M Sparsity Support

#

Wanda naturally supports N:M structured sparsity. For 2:4 sparsity, within each group of 4 consecutive weights in a row, the 2 with the lowest Wanda scores are pruned. The only change to the algorithm is replacing global per-row sorting with local within-group sorting:

For each row i:
    For g = 0, 4, 8, ..., d_col - 4:          // groups of 4
        scores = S[i, g:g+4]
        prune the 2 entries with the lowest scores

Results Comparison: Wanda vs. SparseGPT

#

Key findings:

1. Wanda is within 1-2% PPL of SparseGPT at 50% sparsity across all model sizes.
2. Simple magnitude pruning is catastrophically bad (17.29 vs. 5.68 for LLaMA-7B), demonstrating that activation awareness is essential.
3. The gap between Wanda and SparseGPT widens at higher sparsity (60%+), where the Hessian-based weight updates of SparseGPT become more valuable.
4. Both methods dramatically outperform magnitude pruning, confirming that activation outliers dominate pruning behavior in LLMs.

Magnitude Pruning Revisited for LLMs

#

Despite the clear superiority of activation-aware methods, it is instructive to understand exactly when and why simple magnitude pruning fails for LLMs, and how it can be partially rescued.

Global vs. Per-Layer vs. Per-Row Thresholds

#

There are three granularities for setting the pruning threshold:

1. Global: Compute a single threshold across all weights in the model. Prune all weights below it. This leads to highly non-uniform sparsity — sensitive layers may be over-pruned while redundant layers are under-pruned.

2. Per-layer (uniform): Apply the same sparsity ratio to every layer. This is the most common approach but ignores the fact that different layers have vastly different sensitivity. Empirically, the first and last layers are much more sensitive — applying 60% sparsity uniformly often means the first layer is catastrophically damaged.

3. Per-row (per-output): Apply the target sparsity independently within each row of each weight matrix. This ensures balanced sparsity across output neurons and is the approach used by both SparseGPT and Wanda.

Why Per-Layer Uniform Sparsity Fails

#

Different layers in an LLM have dramatically different weight distributions and sensitivity to pruning. Consider the weight magnitude statistics for LLaMA-7B:

Layer Sensitivity Analysis (LLaMA-7B, 50% unstructured pruning)
─────────────────────────────────────────────────────────────
Layer    | Avg |W| (×10⁻³) | Outlier Ratio | PPL if pruned alone
─────────────────────────────────────────────────────────────
Layer 0  |     8.2          |    0.3%        |   6.42 (+13.0%)
Layer 1  |     5.1          |    0.5%        |   5.82 (+2.5%)
Layer 2  |     4.8          |    0.7%        |   5.79 (+1.9%)
  ...    |     ...          |    ...         |   ...
Layer 15 |     3.9          |    1.2%        |   5.72 (+0.7%)
Layer 16 |     3.8          |    1.4%        |   5.71 (+0.5%)
  ...    |     ...          |    ...         |   ...
Layer 30 |     4.5          |    0.8%        |   5.75 (+1.2%)
Layer 31 |     6.1          |    0.4%        |   6.15 (+8.3%)
─────────────────────────────────────────────────────────────

The first layer (Layer 0) and last layer (Layer 31) are significantly more sensitive. An intelligent strategy would prune these layers less aggressively and allocate the saved budget to middle layers.

Outlier-Aware Magnitude Pruning

#

A simple rescue for magnitude pruning is to protect outlier channels. If we know that feature dimensions with high activation norms are critical, we can:

1. Identify the top-\(k\) features by activation norm (e.g., \(k = 0.01 \times d\)).
2. Mark all weights connected to these features as unprunable.
3. Apply magnitude pruning only to the remaining weights.

This hybrid approach (OWL — Outlier Weighed Layerwise sparsity, Yin et al., 2023) significantly improves magnitude pruning but still falls short of Wanda and SparseGPT because it only protects a binary set of features rather than smoothly weighting all features by their activation norms.

Structural Pruning for LLMs

#

Unstructured sparsity (arbitrary individual weights set to zero) offers maximum flexibility but requires sparse matrix formats and specialized kernels to achieve actual speedups. Structural pruning removes entire computational units — heads, neurons, or layers — producing a smaller dense model that runs faster on standard hardware without any sparse acceleration support.

Attention Head Pruning

#

A multi-head attention layer with \(n_h\) heads computes:

where each head is:

Removing head \(h\) is equivalent to zeroing out the \(h\)-th block of \(d_h\) columns in \(W_Q\), \(W_K\), \(W_V\) and the corresponding \(d_h\) rows in \(W_O\). The parameter savings per removed head are:

For LLaMA-7B (\(d = 4096\), \(n_h = 32\), \(d_h = 128\)): removing one head saves \(4 \times 4096 \times 128 = 2{,}097{,}152\) parameters per layer, or 67.1M across 32 layers.

Head Importance Scoring Methods

#

Gradient-based importance: Measure how much the loss changes when a head’s output is perturbed:

This is a first-order Taylor approximation of the loss change from removing the head. The expectation is taken over calibration samples. Heads where the gradient-activation product is large are critical; those where it is small contribute little to the loss.

Full derivation: The loss change from scaling head \(h\)’s output by \((1 - \epsilon)\) is:

Setting \(\epsilon = 1\) (complete removal) and taking the absolute value gives the importance score. The absolute value handles the sign ambiguity (removing a head might increase or decrease the loss, and we care about the magnitude of the change).

Taylor expansion importance: A slightly different formulation uses the element-wise product:

This is equivalent to the gradient-based method but expressed as an inner product, making it a scalar importance score per head.

Entropy-based importance: Measure the entropy of each head’s attention distribution:

where \(A_h\) is the attention matrix for head \(h\). Heads with very low entropy (attending to 1-2 positions) are often critical (they implement specific retrieval or positional functions). Heads with high entropy (near-uniform attention) are often redundant.

Effects on Different Capabilities

#

Empirical studies show that different types of heads affect different capabilities:

Typically, 25-50% of attention heads can be removed with minimal impact on perplexity, but the effect on specific downstream tasks (especially factual QA and in-context learning) can be more severe than perplexity alone suggests.

Width Pruning (Neuron/Channel Removal)

#

Width pruning removes entire neurons (rows or columns) from weight matrices, reducing the hidden dimension of MLP or attention layers.

SliceGPT (Ashkboos et al., 2024)

#

SliceGPT introduces a mathematically elegant approach to structural pruning based on orthogonal transformations that make weight matrices more amenable to column/row removal.

Key Idea: Before pruning columns from a weight matrix, apply an orthogonal rotation that concentrates the important information into a subset of columns. This is analogous to PCA — project into a basis where most of the variance is captured by a few dimensions, then discard the rest.

Mathematical Framework:

Consider a linear layer \(y = Wx\). We can insert an orthogonal matrix \(Q\) (where \(Q^T Q = I\)) without changing the computation:

where \(W’ = WQ\) and \(x’ = Q^T x\). The rotated weight matrix \(W’\) has the same Frobenius norm as \(W\) (orthogonal transformations preserve norms) but its columns may have very different magnitudes.

If we choose \(Q\) such that the columns of \(W’\) are sorted by importance (e.g., by their contribution to the output variance), we can prune the least important columns of \(W’\) with minimal error.

Choosing Q via PCA: Compute the covariance of the layer’s output:

where \(\Sigma_x = XX^T / N\) is the input covariance. The eigendecomposition of \(\Sigma_x\) gives:

where \(\Lambda = \text{diag}(\lambda_1, \lambda_2, \ldots)\) with eigenvalues sorted in decreasing order. Using this \(Q\) as our rotation, the rotated input \(x’ = Q^T x\) has covariance \(\Lambda\) — a diagonal matrix. The rotated weight \(W’ = WQ\) applied to \(x’\) gives:

The contribution of feature \(j\) to the output variance is proportional to \(\lambda_j |W’_{:,j}|^2\). Since eigenvalues decay (often rapidly for LLMs), features corresponding to small eigenvalues can be pruned.

Absorbing the rotation: The beauty of SliceGPT is that the rotation can be absorbed into adjacent layers. For consecutive layers \(y = W_2(W_1 x)\), inserting \(QQ^T\) between them:

After absorbing \(Q\) into both weight matrices, we can slice (prune) the shared dimension. The model’s computational graph is unchanged — no rotation is needed at inference time.

Algorithm:

Algorithm: SliceGPT
────────────────────────────────────────────────────────────────
1. For each layer l = 1, ..., L:
   a. Collect input activations X_l from calibration data
   b. Compute input covariance: Σ_l = X_l X_l^T / N
   c. Eigendecompose: Σ_l = Q_l Λ_l Q_l^T
   d. Sort eigenvectors by eigenvalue (descending)

2. For each pair of adjacent layers:
   a. Absorb Q_l into W_{l} and W_{l+1}:
      W_l ← W_l Q_l
      W_{l+1} ← Q_l^T W_{l+1}

3. Choose slicing fraction s (e.g., 25%)
4. Remove the last s fraction of columns from each W_l
   and corresponding rows from W_{l+1}
5. The result is a smaller, dense model
────────────────────────────────────────────────────────────────

Complexity: The PCA step requires \(O(d^2 N + d^3)\) per layer for the covariance and eigendecomposition. The rotation absorption is \(O(d^2 d_{\text{ffn}})\). Total: \(O(L \times d^3)\), similar to SparseGPT.

Results: SliceGPT can remove 20-30% of the embedding dimension of LLaMA-7B/13B/70B while maintaining competitive perplexity. Unlike unstructured pruning, the resulting model is dense and runs faster on standard hardware without any sparse kernel support.

Layer Pruning for LLMs

#

The most aggressive form of structural pruning removes entire transformer layers. This yields maximum speedup (each removed layer eliminates all its parameters and computation) but risks significant quality degradation.

Block Importance Metric: A simple and effective metric for layer importance is based on the observation that transformer layers learn residual functions. Layer \(l\) computes:

where \(f_l\) is the layer’s transformation (attention + MLP). If \(f_l(h_l) \approx 0\), the layer is doing little and can be skipped. This motivates the Block Influence (BI) metric:

Layers with low BI scores contribute little to the residual stream and are candidates for removal.

Shortened LLaMA (Kim et al., 2024) uses this approach, combined with optional light fine-tuning after layer removal:

Before Layer Pruning (32-layer LLaMA-7B):
┌───────────────────────────┐
│  Layer 0  (BI = 0.42)     │  ← Keep (high importance)
├───────────────────────────┤
│  Layer 1  (BI = 0.31)     │  ← Keep
├───────────────────────────┤
│  Layer 2  (BI = 0.28)     │  ← Keep
├───────────────────────────┤
│       ...                 │
├───────────────────────────┤
│  Layer 15 (BI = 0.08)     │  ← REMOVE (low importance)
├───────────────────────────┤
│  Layer 16 (BI = 0.07)     │  ← REMOVE (low importance)
├───────────────────────────┤
│  Layer 17 (BI = 0.06)     │  ← REMOVE (low importance)
├───────────────────────────┤
│       ...                 │
├───────────────────────────┤
│  Layer 31 (BI = 0.35)     │  ← Keep (high importance)
└───────────────────────────┘

After Pruning (29-layer model):
┌───────────────────────────┐
│  Layer 0  (BI = 0.42)     │
├───────────────────────────┤
│  Layer 1  (BI = 0.31)     │
├───────────────────────────┤
│       ...                 │
├───────────────────────────┤
│  Layer 14 (BI = 0.12)     │  ← Connected directly to
├───────────────────────────┤     former Layer 18
│  Layer 18 (BI = 0.15)     │
├───────────────────────────┤
│       ...                 │
├───────────────────────────┤
│  Layer 31 (BI = 0.35)     │
└───────────────────────────┘

Saved: 3 layers × 202M params = ~606M parameters (9% reduction)
Speed: ~10% faster inference (fewer sequential steps)

The key finding is that middle layers (especially around layers 15-20 in a 32-layer model) often have the lowest BI scores. These layers tend to learn highly similar representations — a phenomenon called layer redundancy. Removing 2-5 layers from a 32-layer model typically increases PPL by 0.5-2.0 points, which is often acceptable.

Depth Pruning (Dynamic Depth)

#

Rather than permanently removing layers, dynamic depth methods decide at inference time whether to skip a layer based on the current input. This is especially useful because not all inputs need the same depth of processing — simple continuations (e.g., completing “The capital of France is”) may need fewer layers than complex reasoning tasks.

Early exit is a form of depth pruning: if the model’s hidden state at layer \(l\) is already confident enough, skip all remaining layers and project directly to the vocabulary:

Confidence can be measured by the entropy of the predicted distribution or by the norm of the residual \(|f_l(h_l)|\).

Compounding: Pruning + Quantization for LLMs

#

Pruning and quantization are complementary compression techniques that can be applied together for multiplicative benefits. Pruning reduces the number of parameters; quantization reduces the bits per parameter. Combined:

where \(s\) is the sparsity ratio, \(b_{\text{original}}\) is the original bitwidth, and \(b_{\text{quantized}}\) is the quantized bitwidth.

SparseGPT + GPTQ Pipeline

#

The standard pipeline for maximum compression:

1. Prune with SparseGPT to 50% unstructured sparsity (or 2:4 structured).
2. Quantize with GPTQ to 4-bit weights.
3. Store the sparse-quantized model in a compressed format.

The compression calculation for LLaMA-70B:

At 8x compression, LLaMA-70B fits in 17.5 GB — within the memory of a single consumer GPU (RTX 4090 with 24 GB). This is a transformative reduction: from requiring multiple enterprise GPUs to running on consumer hardware.

Wanda + AWQ Combination

#

Activation-Weighted Quantization (AWQ) and Wanda share the same core insight: activation magnitudes determine importance. They can be combined naturally:

1. Compute activation statistics (shared between both methods).
2. Prune with Wanda using \(|W_{ij}| \cdot |X_j|_2\).
3. Quantize with AWQ using activation-aware scaling factors.
4. Both methods use the same calibration data, so the overhead of combining them is minimal.

2:4 Sparsity + INT4 Quantization

#

NVIDIA’s Ampere and Hopper architectures provide hardware acceleration for 2:4 structured sparsity through the Sparse Tensor Cores. When combined with INT4 quantization:

• Sparsity gain: 2:4 pattern gives 2x throughput on Sparse Tensor Cores.
• Quantization gain: INT4 gives 2x throughput over INT8 on Tensor Cores.
• Combined throughput: up to 4x over dense INT8, or 8x over dense FP16.

Memory calculation for LLaMA-70B with 2:4 + INT4:

Dense FP16 parameters: \(70 \times 10^9 \times 2 = 140\) GB

With 2:4 sparsity, we store 50% of weights plus a 2-bit index per group of 4:

• Weight storage: \(70 \times 10^9 \times 0.5 \times 0.5 = 17.5\) GB (50% weights at 4 bits = 0.5 bytes)
• Sparsity metadata: \(70 \times 10^9 \times 2 / (8 \times 4) = 4.375\) GB (2 bits per group of 4)
• Total: approximately 22 GB

Accuracy-Compression Pareto Frontier

#

The sweet spot for most deployments is either (a) quantization only (GPTQ/AWQ INT4) for 4x compression with minimal quality loss, or (b) 2:4 sparsity + INT4 for 8x compression when hardware supports sparse acceleration.

Semi-Structured and Pattern-Based Pruning for LLMs

#

2:4 Sparsity on LLMs

#

NVIDIA’s 2:4 (or N:M) sparsity constraint requires that exactly 2 out of every 4 consecutive weights are zero. This provides a regular pattern that Sparse Tensor Cores can exploit for 2x throughput.

Applying SparseGPT with 2:4 constraint: Replace the pruning decision step in SparseGPT with a constrained selection. Instead of choosing the globally least salient weights, within each group of 4 consecutive columns, prune the 2 with the lowest saliency. The OBS weight update still applies — remaining weights are compensated.

Applying Wanda with 2:4 constraint: Similarly, within each group of 4, prune the 2 with the lowest Wanda scores (\(|W_{ij}| \cdot |X_j|_2\)). This is the simplest possible implementation of hardware-friendly LLM pruning.

Training LLMs with N:M from Scratch vs. Post-Training N:M

#

Training with N:M sparsity from scratch uses the SR-STE (Straight-Through Estimator for Structured Sparsity) approach:

1. Maintain a dense weight matrix \(W_{\text{dense}}\).
2. At each forward pass, apply the N:M mask to get \(W_{\text{sparse}}\).
3. Compute loss with \(W_{\text{sparse}}\).
4. In the backward pass, compute gradients with respect to \(W_{\text{sparse}}\) but apply them to \(W_{\text{dense}}\) (straight-through).
5. Periodically update the mask based on current weight magnitudes.

For LLMs, training from scratch with N:M is extremely expensive (it requires the same resources as dense training). Post-training N:M pruning (SparseGPT or Wanda with the 2:4 constraint) is therefore the practical choice, even though it yields slightly worse results.

Performance on Actual Hardware

#

Measured on NVIDIA A100 (Ampere) for LLaMA-7B inference, batch size 1, sequence length 2048:

The 2:4 sparse INT4 configuration achieves 3.5x speedup over dense FP16 with 6x memory reduction, while typically maintaining acceptable quality (PPL increase of 2-4 points on WikiText-2).

Pruning for Efficient LLM Inference

#

KV-Cache Pruning

#

During autoregressive generation, LLMs cache the key-value (KV) tensors for all previous tokens to avoid redundant computation. This KV-cache grows linearly with sequence length and becomes the dominant memory consumer for long sequences.

KV-cache memory formula:

where:

• Factor 2 = keys + values
• \(L\) = number of layers
• \(n_h\) = number of attention heads (or KV heads in GQA)
• \(d_h\) = head dimension
• \(\text{seq_len}\) = sequence length
• \(\text{batch}\) = batch size
• \(b\) = bytes per element

Numerical example for LLaMA-7B (\(L=32, n_h=32, d_h=128\)):

At sequence length 4096, FP16:

At sequence length 128K (long-context models):

This is larger than the model weights themselves (13.5 GB for LLaMA-7B). For long-context applications, KV-cache memory is the primary bottleneck.

KV-cache pruning removes tokens from the cache that are unlikely to be attended to in the future, keeping the cache at a manageable size.

KV-Cache Pruning Strategies:
─────────────────────────────────────────────────────────────

Full KV-Cache (no pruning):
┌────────────────────────────────────────────────────┐
│ t1 │ t2 │ t3 │ t4 │ t5 │ t6 │ ... │ t_n │ NEW    │
└────────────────────────────────────────────────────┘
  Memory: O(n) — grows without bound

Sliding Window (fixed):
┌────────────────────────────────────────────────────┐
│ ██ │ ██ │ ██ │ ██ │ t_{n-3} │ t_{n-2} │ t_{n-1} │ NEW │
└────────────────────────────────────────────────────┘
  ██ = evicted       Memory: O(window_size)
  Problem: loses long-range dependencies

H2O (Heavy-Hitter Oracle):
┌────────────────────────────────────────────────────┐
│ t1 │ ██ │ t3 │ ██ │ ██ │ t6 │ t_{n-2} │ t_{n-1} │ NEW │
└────────────────────────────────────────────────────┘
  Keep: high-attention tokens + recent tokens
  Memory: O(budget) — fixed budget

StreamingLLM (sink + window):
┌────────────────────────────────────────────────────┐
│ t1 │ t2 │ ██ │ ██ │ ██ │ ██ │ t_{n-2} │ t_{n-1} │ NEW │
└────────────────────────────────────────────────────┘
  Keep: first few tokens (sinks) + recent window
  Memory: O(sinks + window_size)
─────────────────────────────────────────────────────────────

H2O (Heavy-Hitter Oracle) — Zhang et al., 2023

#

H2O observes that attention patterns in LLMs follow a power-law distribution: a small fraction of tokens receive a disproportionately large share of attention across all heads and layers. These “heavy hitter” tokens carry critical information.

Algorithm:

1. Maintain a cumulative attention score for each cached token:

2. Set a fixed budget \(B\) for the KV-cache.
3. When the cache exceeds \(B\), evict the token with the lowest cumulative attention score (excluding the most recent \(W\) tokens, which are always kept).

The combination of heavy hitters (globally important tokens) + recent tokens (locally important for coherent generation) preserves both long-range factual knowledge and short-range fluency.

Results: H2O can reduce KV-cache by 5-10x (keeping only 10-20% of tokens) with minimal perplexity increase on long-context benchmarks. This enables LLaMA-7B to handle 100K+ token contexts within the memory budget originally needed for 10K tokens.

StreamingLLM — Xiao et al., 2023

#

StreamingLLM makes a simpler but powerful observation: the first few tokens in the sequence (the “attention sinks”) receive abnormally high attention from all subsequent tokens, regardless of their semantic content. This is an artifact of the softmax function — it must assign attention mass somewhere, and the first positions serve as a default sink.

Strategy: Keep the first \(S\) tokens (sinks, typically \(S = 4\)) and a sliding window of the most recent \(W\) tokens. Evict everything in between.

Memory: \(O(S + W)\) — constant regardless of total sequence length. This enables truly infinite-length streaming generation.

Limitation: Information between the sinks and the window is lost. StreamingLLM is best suited for streaming applications (chatbots, real-time transcription) where maintaining a complete history is not required.

Activation Pruning / Dynamic Sparsity

#

A complementary approach to weight pruning is activation pruning: dynamically skipping computations based on the input at inference time. This exploits the fact that for any given input token, a large fraction of MLP neurons produce negligibly small outputs.

ReLU-Based Sparsity

#

Models using ReLU activation (rather than GeLU or SiLU) naturally produce sparse activations — roughly 90%+ of neurons output zero for any given input. This sparsity can be exploited by only computing the non-zero neurons.

Some recent work has explored replacing GeLU/SiLU with ReLU in LLMs specifically to unlock this activation sparsity. The quality impact is small (< 0.5% PPL increase) but the inference speedup can be significant (2-3x for MLP layers).

Deja Vu: Contextual Sparsity Prediction (Liu et al., 2023)

#

Deja Vu takes activation pruning further by training a small predictor network that, given the current hidden state, predicts which neurons and attention heads will be active (have significant output) for the current input.

Architecture:

┌──────────────────────────────────────────────────┐
│            Deja Vu: Contextual Sparsity          │
│                                                  │
│  Input hidden state h_l                          │
│        │                                         │
│        ▼                                         │
│  ┌─────────────────┐                             │
│  │  Small Predictor │  (2-layer MLP, ~0.5% of    │
│  │  Network         │   main model size)          │
│  └────────┬────────┘                             │
│           │                                      │
│           ▼                                      │
│  Predicted active set: {neuron_3, neuron_7,      │
│    neuron_15, ...}  (~25% of total neurons)      │
│           │                                      │
│           ▼                                      │
│  ┌─────────────────┐                             │
│  │  Execute ONLY    │  (75% computation saved     │
│  │  active neurons  │   in this layer)            │
│  └────────┬────────┘                             │
│           │                                      │
│           ▼                                      │
│  Output h_{l+1}  (approximately same as full)    │
└──────────────────────────────────────────────────┘

Results: Deja Vu demonstrates that 75%+ of neurons can be skipped per token with less than 0.1% accuracy loss. For MLP layers (which dominate computation), this translates to a 2-4x speedup. The predictor adds negligible overhead because it is much smaller than the main model.

Latency analysis: For a LLaMA-7B MLP layer with \(d = 4096\), \(d_{\text{ffn}} = 11008\):

• Full MLP: \(3 \times d \times d_{\text{ffn}} = 3 \times 4096 \times 11008 \approx 135M\) MACs
• With 75% sparsity prediction: \(3 \times 4096 \times 2752 \approx 34M\) MACs + predictor overhead (~0.5M MACs)
• Speedup: \(135M / 34.5M \approx 3.9\times\)

However, the practical speedup is lower (typically 2-3x) because: (1) sparse computation has lower hardware utilization than dense, (2) the predictor introduces a serial dependency (must run before the main layer), and (3) memory access patterns become irregular.

Pruning Emerging Architectures

#

Mamba/State Space Models Pruning

#

State Space Models (SSMs) like Mamba replace attention with a linear recurrence, eliminating the KV-cache entirely. Pruning SSMs involves:

• Width pruning: Reduce the state dimension \(N\) or the expansion factor \(E\). SSMs typically use \(N = 16\) and \(E = 2\), so there is less room to prune than in transformers.
• Channel pruning: Remove input/output channels from the SSM block. Similar to MLP neuron pruning.
• Selective state removal: Some state dimensions contribute more than others. Importance can be measured by the magnitude of the \(C\) matrix (output projection).

SSM pruning is less studied than transformer pruning because SSMs are inherently more parameter-efficient. A Mamba model with equivalent quality to a transformer is typically 2-3x smaller, reducing the need for aggressive pruning.

Mixture of Experts (MoE) Pruning

#

MoE models (Mixtral, Switch Transformer) have a unique structure: each MLP layer is replaced by \(E\) expert MLPs, and a router selects the top-\(k\) experts per token. The total parameter count is \(E \times\) larger than a dense model, but only \(k/E\) fraction is activated per token.

Expert merging: Identify pairs of experts that learn similar functions (measured by weight cosine similarity or output correlation on calibration data). Merge them by averaging their weights:

This reduces the number of experts (and total parameters) while maintaining the routing structure. Mixtral-8x7B (8 experts) can often be compressed to 4-6 experts with modest quality loss.

Expert dropping: Simply remove the least-used experts (those routed to least frequently). The router is adjusted to distribute their probability mass among remaining experts. This is the simplest approach but can cause load imbalance.

Multi-Modal LLM Pruning

#

Multi-modal LLMs (LLaVA, GPT-4V) combine a vision encoder with a language model. Pruning these models requires separate strategies:

• Vision encoder: Typically a ViT, which can be pruned using standard CNN/ViT pruning techniques (token merging, attention head pruning, patch embedding pruning).
• Language model: Uses the LLM pruning techniques discussed throughout this post.
• Cross-modal projector: Usually a small MLP connecting vision and language spaces. This is typically not pruned because it is small and critical for alignment.

The key question is how to balance pruning between the vision and language components. Empirically, the language model tolerates pruning better because it is much larger and more redundant. A common strategy is to prune the language model to 50% sparsity while leaving the vision encoder at 20-30% sparsity.

Evaluation and Benchmarking

#

Standard Benchmarks

#

LLM pruning methods are evaluated on:

1. Perplexity (PPL) on held-out text:

   • WikiText-2: ~250K tokens of Wikipedia articles. The standard benchmark for language modeling quality.
   • C4: Colossal Clean Crawled Corpus. Larger and more diverse than WikiText-2.

2. Zero-shot downstream tasks:

   • ARC (AI2 Reasoning Challenge): Science exam questions. Tests reasoning.
   • HellaSwag: Sentence completion. Tests common sense.
   • WinoGrande: Pronoun resolution. Tests commonsense reasoning.
   • MMLU: Massive Multitask Language Understanding. 57 subjects, tests broad knowledge.
   • TruthfulQA: Tests tendency to generate truthful answers.

3. Generation quality: Human evaluation or GPT-4-based evaluation of generation fluency, coherence, and factual accuracy.

Comprehensive Comparison Table

#

Key takeaways from the table:

1. Magnitude pruning catastrophically fails at 50% sparsity for LLMs.
2. SparseGPT and Wanda achieve comparable quality, with SparseGPT slightly better.
3. Structured pruning (SliceGPT, head pruning) yields real dense-model speedups at the cost of slightly worse quality than unstructured methods at the same compression ratio.
4. Layer pruning is the most aggressive but can maintain reasonable quality when removing only a few layers.

Throughput Measurements

#

Actual inference throughput depends heavily on the sparsity pattern, hardware, and software stack:

Critical observation: unstructured sparsity provides no speedup on standard hardware. The 50% sparse model runs at the same speed as the dense model because standard dense GEMM kernels process all matrix entries regardless of sparsity. Only specialized sparse kernels (available on NVIDIA Ampere+ for 2:4 patterns) provide actual acceleration.

Practical Guide

#

Decision Flowchart: Which Method to Use?

#

Start: I want to compress my LLM
│
├── Q: Do you need actual inference speedup on current hardware?
│   │
│   ├── YES ──► Q: Does your hardware support 2:4 sparsity?
│   │           │
│   │           ├── YES ──► SparseGPT or Wanda with 2:4 constraint
│   │           │           + INT4 quantization (GPTQ/AWQ)
│   │           │           Expected: ~4x speedup, ~8x compression
│   │           │
│   │           └── NO ──► Quantization alone (GPTQ/AWQ INT4)
│   │                       is your best bet for speedup.
│   │                       Consider SliceGPT for structural reduction.
│   │                       Expected: ~2-3x speedup, ~4x compression
│   │
│   └── NO (just want smaller model for storage/transfer)
│       │
│       ├── Q: How much quality loss is acceptable?
│       │   │
│       │   ├── Minimal (<5% PPL increase)
│       │   │   └── Wanda or SparseGPT at 50% unstructured
│       │   │       + GPTQ INT4
│       │   │       Expected: ~8x size reduction
│       │   │
│       │   └── Moderate (5-15% PPL increase)
│       │       └── SparseGPT at 60% + INT4
│       │           or 2:4 sparsity + INT4
│       │           Expected: ~10-16x size reduction
│       │
│       └── Q: How much time can you spend on pruning?
│           │
│           ├── Minutes ──► Wanda (30 sec for 7B model)
│           │
│           └── Hours OK ──► SparseGPT (better quality)
│
└── Q: Do you need to fit model in very limited memory?
    │
    ├── < 4 GB ──► Aggressive: 2:4 + INT4 + layer pruning
    │              May need to accept significant quality loss
    │
    └── < 8 GB ──► 50% sparse + INT4, or INT4 alone
                   for 7B models this is quite achievable

Calibration Data Selection Tips

#

1. Use diverse data: 128 sequences from C4 is the standard. Do not use only code, only dialogue, or only one domain.
2. Match the deployment domain (partially): If the model will primarily be used for code generation, include some code in the calibration set (but not exclusively).
3. Sequence length matters: Use the same context length for calibration as you expect during inference. Short calibration sequences (512 tokens) may not capture long-range patterns needed for long-context inference (4096+ tokens).
4. More is not always better: Beyond 128-256 sequences, additional calibration data provides diminishing returns for both SparseGPT and Wanda. The Hessian / activation statistics converge quickly.
5. Avoid degenerate samples: Remove sequences that are mostly whitespace, repetitive characters, or non-natural-language content. These can skew activation statistics.

Common Failures and Debugging

#

Tools and Frameworks

#

1. SparseML (Neural Magic): Production-ready framework for pruning and sparse inference. Supports SparseGPT and Wanda with optimized CPU sparse kernels.
2. llama.cpp: Supports GGML format with sparse weight matrices. Wanda-pruned models can be exported directly.
3. NVIDIA TensorRT-LLM: Supports 2:4 structured sparsity with Sparse Tensor Core acceleration on Ampere and Hopper GPUs.
4. torch.ao (Architecture Optimization): PyTorch’s built-in support for structured pruning and semi-structured sparsity.
5. Hugging Face PEFT + Pruning: Integration of pruning with parameter-efficient fine-tuning for combined compression and adaptation.

Current State and Future Directions (2024-2025)

#

Trends

#

1. From post-training to training-aware LLM pruning: As more organizations train LLMs from scratch, there is increasing interest in incorporating sparsity during pre-training. Cerebras and Neural Magic have demonstrated that training with 80% unstructured sparsity from the start can match dense model quality while reducing training compute by 2-5x.

2. Scaling laws for sparse models: Extending the Chinchilla scaling laws to sparse models. Early results suggest that sparse models follow different scaling laws — with optimal sparsity increasing as model size grows. A 70B model may achieve its best quality-efficiency tradeoff at 60-70% sparsity, while a 7B model peaks at 40-50%.

3. Hardware co-design for sparsity: Next-generation AI accelerators are being designed with native support for flexible sparsity patterns (beyond 2:4). NVIDIA’s Blackwell architecture extends sparse support, and startups like Cerebras and Groq build sparsity into their fundamental architecture.

4. Compositional compression: Combining pruning + quantization + distillation + architecture search in a unified framework. Rather than applying each technique independently, joint optimization finds better Pareto-optimal points.

5. Pruning for inference-time efficiency: KV-cache pruning, dynamic depth, and speculative decoding are becoming standard components of production LLM serving stacks. These techniques reduce per-query cost without modifying the model weights.

Open Problems and Research Directions

#

1. Pruning with guaranteed capability preservation: Current methods optimize for perplexity, which does not capture all capabilities (reasoning, factual knowledge, instruction following). How can we prune while provably preserving specific capabilities?

2. Structured sparsity beyond 2:4: Can we design hardware-friendly sparsity patterns (e.g., 4:8, block sparsity) that offer higher compression while maintaining real speedups?

3. Pruning for fine-tuned/RLHF models: Most pruning research focuses on base models. How does pruning interact with instruction tuning and RLHF? Preliminary evidence suggests that RLHF-tuned models are more sensitive to pruning.

4. Data-free pruning: Eliminating the need for calibration data entirely. Some methods explore using the weight statistics alone (e.g., weight distribution shape) to guide pruning decisions.

5. Pruning as a regularizer: Some evidence suggests that mild pruning (10-20%) can actually improve generalization on out-of-distribution tasks by removing spurious correlations learned during training.

Summary

#

Master Comparison of All LLM Pruning Methods

#

Key Takeaways

#

1. LLM pruning requires fundamentally different approaches than CNN pruning because retraining is infeasible. One-shot methods (SparseGPT, Wanda) are the practical standard.

2. Activation awareness is essential: Simple magnitude pruning fails catastrophically for LLMs because of activation outliers. Both SparseGPT (via the full Hessian) and Wanda (via activation norms) address this, with Wanda offering a remarkably simple and effective solution.

3. The OBS framework (SparseGPT) provides the theoretical gold standard: The formula \(\delta_w = -\frac{w_q}{[H^{-1}]{qq}} \cdot (H^{-1}){:,q}\) optimally compensates for each pruned weight. Wanda approximates this with a diagonal Hessian assumption, trading some accuracy for dramatic simplicity.

4. Structured pruning is necessary for real speedups: Unstructured sparsity reduces model size but does not reduce inference latency on standard hardware. 2:4 structured sparsity (with hardware support), head pruning, layer pruning, and width reduction (SliceGPT) provide actual deployment benefits.

5. Pruning and quantization are complementary: The combination of 50% sparsity + INT4 quantization achieves 8x compression, potentially fitting LLaMA-70B in under 20 GB of memory.

6. KV-cache pruning is a separate but equally important frontier: For long-context applications, KV-cache memory dominates. H2O and StreamingLLM provide practical solutions that enable 5-10x longer contexts within the same memory budget.

7. The field is moving toward inference-time adaptive sparsity: Rather than static pruning, methods like Deja Vu and dynamic depth adapt the computation per input, exploiting the fact that not all tokens need the same amount of processing.

Series Wrap-Up: Pruning and Quantization Together

#

This post concludes the AI Accelerator series on model compression. Across the series, we have covered:

• Quantization fundamentals: Number representation, uniform/non-uniform schemes, calibration strategies.
• Post-training quantization (PTQ): GPTQ, AWQ, and their application to LLMs.
• Quantization-aware training (QAT): Training with simulated quantization for maximum quality.
• Extreme and mixed-precision quantization: Sub-4-bit methods, mixed precision allocation.
• Pruning fundamentals: Magnitude pruning, OBD, OBS, theoretical foundations.
• Structured vs. unstructured pruning: Filter pruning, channel pruning, hardware implications.
• Advanced pruning methods: Lottery ticket hypothesis, progressive pruning, neural architecture search.
• Pruning for LLMs (this post): SparseGPT, Wanda, KV-cache pruning, and the unique challenges of billion-parameter models.

The central lesson across the entire series is that model compression is not a single technique but a toolbox. The best deployment strategy depends on the specific constraints: target hardware, latency requirements, memory budget, quality tolerance, and engineering resources. The table below summarizes the overall compression landscape:

As hardware evolves to better support sparsity and low-precision arithmetic, and as algorithms improve to minimize quality loss at extreme compression ratios, the gap between a 70-billion-parameter model and its 10x-compressed counterpart will continue to narrow. The goal is not merely to make models smaller, but to make the full power of frontier language models accessible on the hardware that people actually have.

References

• Frantar, E., & Alistarh, D. (2023). SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. ICML 2023.
• Sun, M., et al. (2023). A Simple and Effective Pruning Approach for Large Language Models. ICLR 2024.
• Ashkboos, S., et al. (2024). SliceGPT: Compress Large Language Models by Deleting Rows and Columns. ICLR 2024.
• Zhang, Z., et al. (2023). H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. NeurIPS 2023.
• Xiao, G., et al. (2023). Efficient Streaming Language Models with Attention Sinks. ICLR 2024.
• Liu, Z., et al. (2023). Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML 2023.
• Kim, B., et al. (2024). Shortened LLaMA: A Simple Depth Pruning for Large Language Models. ICLR 2024 Workshop.
• Yin, L., et al. (2023). Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity. ICML 2024.
• Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022.
• LeCun, Y., Denker, J., & Solla, S. (1989). Optimal Brain Damage. NeurIPS 1989.
• Hasselmo, M., et al. (1993). Optimal Brain Surgeon and General Network Pruning. IEEE ICNN 1993.