Structured vs Unstructured Pruning: A Complete Guide with Math, Diagrams, and Real-World Analysis

Overview

#

Neural network pruning removes redundant parameters to produce smaller, faster models. The central question is not whether to prune, but how to prune – and the answer determines whether you get a real-world speedup or merely a theoretical one.

The Fundamental Tradeoff: Flexibility vs Hardware Efficiency

#

Pruning methods span a spectrum defined by two opposing forces:

• Flexibility: the freedom to remove any individual weight regardless of its position in a tensor. More flexibility means higher sparsity at the same accuracy, because the optimizer can cherry-pick the least important parameters wherever they are.
• Hardware efficiency: the ability of actual processors to exploit the resulting sparsity. Modern GPUs, CPUs, and accelerators are optimized for dense, regular memory access patterns. The more structured the sparsity pattern, the easier it is for hardware to convert zero parameters into saved computation cycles.

This tradeoff is the single most important concept in pruning research:

Flexibility                                      Hardware Efficiency
(Accuracy)                                       (Real Speedup)
    |                                                    |
    |  Unstructured   N:M      Block    Channel  Filter  |
    |  (individual)  (2:4)   (4x1,2x4)          Layer   |
    |  =============================================>    |
    |                                                    |
    |  Fine-grained <-----------> Coarse-grained         |

Why Structure Matters for Real-World Speedup

#

Consider a dense matrix multiply \(Y = XW\) where \(W \in \mathbb{R}^{m \times n}\). If we prune 90% of the weights in \(W\) at random positions:

1. The number of nonzero multiply-accumulate operations drops to 10%.
2. But \(X\) and \(W\) cannot be stored as contiguous dense blocks.
3. The GPU still allocates the full \(m \times n\) tensor in memory.
4. Sparse indexing introduces overhead for every nonzero access.
5. The Tensor Cores cannot be used because the operands are not dense tiles.

The result: theoretical 10x FLOPs reduction, actual 1.0-1.2x speedup on a standard GPU.

Now consider pruning 50% of the filters (entire rows of \(W\)). The resulting weight matrix is \(W’ \in \mathbb{R}^{m/2 \times n}\). This is a perfectly dense matrix – just smaller. The matrix multiply becomes \(Y’ = XW’\), which runs at full Tensor Core efficiency on a smaller problem. The result: theoretical 2x FLOPs reduction, actual 1.8-1.95x speedup.

Taxonomy of Pruning Granularity

#

We classify pruning granularity from finest to coarsest:

Unstructured (Fine-Grained) Pruning

#

Definition and Formulation

#

Unstructured pruning removes individual weights from a parameter tensor. Given a weight matrix \(W \in \mathbb{R}^{m \times n}\), we compute a binary mask \(M \in {0, 1}^{m \times n}\) and apply it element-wise:

where \(\odot\) denotes the Hadamard (element-wise) product. The mask \(M\) is determined by some importance criterion. The simplest and most widely used is magnitude pruning:

where \(\theta\) is a threshold chosen to achieve the desired sparsity level \(s\):

Here \(||M||_0\) counts the number of nonzero entries in \(M\).

Why Unstructured Pruning Achieves the Highest Sparsity

#

The key insight is degrees of freedom. With \(m \times n\) independent binary decisions, the optimizer can pick the globally least important weights. For a given accuracy target, unstructured pruning always achieves equal or higher sparsity than any structured method, because structured methods impose additional constraints on which weights must be removed together.

Formally, let \(\mathcal{M}_u\) be the set of all possible unstructured masks and \(\mathcal{M}_s \subset \mathcal{M}_u\) be the set of structured masks. The optimal unstructured mask solves:

Since \(\mathcal{M}_s \subset \mathcal{M}_u\), we have \(\mathcal{L}(W \odot M^_u) \leq \mathcal{L}(W \odot M^_s)\) for any sparsity budget \(k\). In practice, unstructured methods routinely reach 90-98% sparsity with less than 1% accuracy loss, while structured methods often struggle beyond 50-70%.

The Sparsity Illusion: 95% Sparse but No Speedup on GPU

#

This is the most common trap in pruning research. A paper reports “95% sparsity with only 0.5% accuracy drop” and claims massive compression. But when deployed:

Dense Model:
  W = [0.3  0.0  0.7  0.1]     Memory: 4 x 4 x 4B = 64 bytes
      [0.0  0.5  0.0  0.0]     GEMM: Dense 4x4, fully pipelined
      [0.2  0.0  0.0  0.8]     Tensor Cores: YES
      [0.0  0.0  0.6  0.0]     Actual speed: baseline

After 75% Unstructured Pruning:
  W = [0.3  0.0  0.7  0.0]     Memory: still 64 bytes (or more with index)
      [0.0  0.5  0.0  0.0]     GEMM: Sparse, irregular access
      [0.0  0.0  0.0  0.8]     Tensor Cores: NO
      [0.0  0.0  0.6  0.0]     Actual speed: ~1.0x (no speedup!)

The problem is architectural. Modern GPUs execute matrix multiplications by:

1. Loading tiles (e.g., 16x16) from global memory into shared memory.
2. Computing the tile product using Tensor Cores (dense fused multiply-add).
3. Writing the output tile back.

Every step assumes dense, contiguous data. A sparse matrix with 95% zeros still occupies the same memory footprint unless converted to a sparse format. Even in sparse formats, the irregular access patterns prevent coalesced memory reads and Tensor Core utilization.

Irregular Memory Access Patterns

#

Consider a simple sparse matrix-vector multiply \(y = Wx\) with \(W\) stored in Compressed Sparse Row (CSR) format:

Dense W:                   CSR Representation:
[0.3  0.0  0.7  0.0]      values:    [0.3, 0.7, 0.5, 0.8, 0.6]
[0.0  0.5  0.0  0.0]      col_idx:   [0,   2,   1,   3,   2  ]
[0.0  0.0  0.0  0.8]      row_ptr:   [0,   2,   3,   4,   5  ]
[0.0  0.0  0.6  0.0]

Memory access for row 0: x[0], x[2]     (stride=2, non-contiguous)
Memory access for row 1: x[1]            (single element)
Memory access for row 2: x[3]            (single element)
Memory access for row 3: x[2]            (single element)

Each row accesses different, unpredictable locations in \(x\). This is the opposite of what GPUs need (coalesced, predictable access). Cache lines are loaded but only partially used, wasting memory bandwidth.

Sparse Matrix Storage Overhead Analysis

#

Let us quantify the storage overhead. For a dense matrix \(W \in \mathbb{R}^{m \times n}\) with sparsity \(s\) (fraction of zeros):

Dense storage: \(m \times n \times b\) bytes, where \(b\) is bytes per element (4 for FP32, 2 for FP16).

CSR storage:

• Values array: \((1-s) \cdot m \cdot n \cdot b\) bytes
• Column indices: \((1-s) \cdot m \cdot n \cdot 4\) bytes (INT32)
• Row pointers: \((m+1) \cdot 4\) bytes

Total CSR: \((1-s) \cdot m \cdot n \cdot (b + 4) + (m+1) \cdot 4\)

The break-even point where CSR becomes smaller than dense occurs at:

Solving for \(s\) (ignoring the row pointer term for large matrices):

For FP32 (\(b=4\)): \(s > 0.5\) (50% sparsity needed just to break even on storage). For FP16 (\(b=2\)): \(s > 0.667\) (67% sparsity needed). For INT8 (\(b=1\)): \(s > 0.8\) (80% sparsity needed).

This shows that sparse formats become less attractive as element size decreases – precisely when quantization is also applied.

Numerical Example: Pruning a 4x4 Matrix

#

Consider a fully connected layer with \(W \in \mathbb{R}^{4 \times 4}\):

Step 1: Compute magnitudes:

Step 2: Sort all 16 magnitudes: 0.01, 0.02, 0.03, 0.05, 0.07, 0.09, 0.11, 0.15, 0.23, 0.38, 0.44, 0.56, 0.68, 0.77, 0.82, 0.91

Step 3: For 50% sparsity, prune the 8 smallest. Threshold \(\theta = 0.15\) (the 8th value). Everything with magnitude \(< 0.15\) is pruned:

Notice the irregular pattern: nonzeros are scattered with no spatial regularity. This matrix cannot be represented as a smaller dense matrix.

When Unstructured Pruning Works: Specialized Hardware

#

Unstructured pruning becomes practical on hardware designed for sparsity:

• Cerebras WSE-2/3: The wafer-scale engine has a dataflow architecture where each processing element can skip zero operands natively. Unstructured sparsity directly reduces compute.
• NVIDIA Sparse Tensor Cores (Ampere+): Support N:M structured sparsity (a constrained form), not fully unstructured.
• Graphcore IPU: Can exploit some levels of sparsity through its bulk synchronous parallel model.
• CPUs with branch-based kernels: For small models, CPU inference can use conditional branches to skip zero multiplications, though branch misprediction limits the benefit.

Structured Pruning – Detailed Taxonomy

#

Filter/Kernel Pruning (Coarse-Grained)

#

Filter pruning is the most widely used form of structured pruning for convolutional networks. It removes entire 3D filters from a convolutional layer, resulting in a smaller but fully dense layer.

Setup: Consider a convolutional layer with weight tensor \(W \in \mathbb{R}^{C_{out} \times C_{in} \times k_h \times k_w}\), where:

• \(C_{out}\): number of output channels (filters)
• \(C_{in}\): number of input channels
• \(k_h \times k_w\): kernel spatial dimensions

Each filter \(F_i \in \mathbb{R}^{C_{in} \times k_h \times k_w}\) for \(i = 1, \ldots, C_{out}\) produces one output feature map.

L1-Norm Filter Pruning (Li et al., 2017)

#

The importance of filter \(i\) is measured by the sum of absolute values of all its parameters:

Derivation of why L1-norm is a reasonable proxy for importance:

The output of filter \(i\) at spatial location \((x, y)\) is:

where \(A\) is the input activation tensor. If the input activations have roughly unit variance and zero mean (which BatchNorm ensures), then the expected magnitude of \(Z_i(x,y)\) scales with:

A filter with smaller L1-norm produces activations with smaller expected magnitude, contributing less to the network’s representational capacity. Removing it should therefore cause less damage to the output.

Algorithm:

1. For each layer \(l\), compute \(\text{score}(F_i^{(l)})\) for all \(i\).
2. Sort filters by score within each layer (or globally).
3. Remove the bottom \(p%\) of filters per layer (or global threshold).
4. Remove corresponding structures in subsequent layers.
5. Fine-tune the pruned network.

Geometric Median Filter Pruning (He et al., 2019)

#

Instead of pruning the smallest filters, this method prunes filters that are most replaceable – those closest to the geometric median of all filters. The geometric median minimizes the sum of distances:

The most replaceable filter \(j\) is:

Intuition: If filter \(i\) is close to many other filters, its function can be approximated by a linear combination of the remaining filters. Removing it causes minimal information loss. This is particularly useful when many filters have similar L1-norms but encode redundant features.

Effect on Layer Dimensions

#

Removing filter \(i\) from layer \(l\) has cascading effects:

BEFORE PRUNING:
Layer l:     W^(l) in R^{C_out x C_in x k x k}
             bias^(l) in R^{C_out}
             BN^(l): gamma, beta, running_mean, running_var in R^{C_out}

Layer l+1:   W^(l+1) in R^{C_out' x C_out x k' x k'}

AFTER REMOVING FILTER i FROM LAYER l:
Layer l:     W^(l) in R^{(C_out-1) x C_in x k x k}     [row i removed]
             bias^(l) in R^{(C_out-1)}                   [element i removed]
             BN^(l): all params in R^{(C_out-1)}          [element i removed]

Layer l+1:   W^(l+1) in R^{C_out' x (C_out-1) x k' x k'} [channel i removed]

This is the fundamental property of structured pruning: removing a filter from layer \(l\) changes the shape of two layers (\(l\) and \(l+1\)), but both remain fully dense tensors.

ASCII Diagram: Before and After Filter Pruning

#

BEFORE: Conv Layer l (C_out=4, C_in=3, k=3x3)

Filter 0:  [3x3x3]  ----+
Filter 1:  [3x3x3]  ----+----> Output: [4 x H' x W']
Filter 2:  [3x3x3]  ----+       (4 output channels)
Filter 3:  [3x3x3]  ----+

Next Layer l+1 expects 4 input channels:
W^(l+1) shape: [C_out' x 4 x k' x k']

---------- Prune Filter 1 and Filter 3 (50% pruning) ----------

AFTER: Conv Layer l (C_out=2, C_in=3, k=3x3)

Filter 0:  [3x3x3]  ----+
                          +----> Output: [2 x H' x W']
Filter 2:  [3x3x3]  ----+       (2 output channels)

Next Layer l+1 now expects 2 input channels:
W^(l+1) shape: [C_out' x 2 x k' x k']

Result: Layer l is 50% smaller, Layer l+1 input dimension halved
        Both remain DENSE tensors -> full hardware utilization

Full Numerical Example with a Small Conv Layer

#

Consider a tiny conv layer: \(C_{out}=3, C_{in}=2, k=2\times 2\).

Filter 0:

Filter 1:

Filter 2:

Compute L1-norm scores:

Ranking: \(F_1 (0.24) < F_0 (2.9) < F_2 (4.4)\)

Pruning: Remove \(F_1\) (lowest L1-norm). The pruned layer has \(C_{out}=2\), \(C_{in}=2\), \(k=2\times 2\). This is a standard dense conv layer that any framework can execute efficiently.

Channel Pruning

#

Channel pruning removes input channels rather than output filters. While filter pruning operates on the output dimension, channel pruning operates on the input dimension of a weight tensor.

Channel Pruning via LASSO Regression (He et al., 2017)

#

The goal is to select a subset of input channels that best reconstruct the output feature maps. For a layer with input \(X \in \mathbb{R}^{N \times C_{in} \times H \times W}\) (batch of activations) and filters \(W \in \mathbb{R}^{C_{out} \times C_{in} \times k \times k}\):

The output is \(Y = \sum_{c=1}^{C_{in}} X_c * W_c\) where \(*\) denotes convolution and \(X_c, W_c\) are the \(c\)-th channel slices.

Channel pruning introduces a channel selection vector \(\beta \in {0,1}^{C_{in}}\):

The optimization problem is:

Since the \(\ell_0\) constraint is NP-hard, it is relaxed to an \(\ell_1\) penalty (LASSO):

Derivation of the LASSO solution (for fixed \(W’ = W\)):

Reformulate as a standard LASSO. Let \(Z_c = X_c * W_c \in \mathbb{R}^{N \times H’ \times W’}\) be the contribution of channel \(c\). Flatten \(Y\) and each \(Z_c\) into vectors, forming the matrix \(Z = [z_1, z_2, \ldots, z_{C_{in}}]\):

Taking the subgradient and setting to zero:

For each coordinate \(c\), the solution is the soft-thresholding operator:

where \(r_c = Z_c^T (y - Z_{-c}\beta_{-c}) / ||Z_c||_2^2\) is the partial residual.

Channels with \(\hat{\beta}_c = 0\) are pruned.

ThiNet: Pruning Channels Based on Next Layer’s Statistics

#

ThiNet (Luo et al., 2017) takes a different approach: instead of analyzing the current layer, it selects channels to prune based on how well the next layer’s output can be reconstructed.

For layer \(l+1\) with weights \(W^{(l+1)}\), the output at a single spatial position is:

ThiNet uses a greedy algorithm to find the subset \(S \subset {1, \ldots, C}\) with \(|S| = C \cdot (1-s)\) that minimizes:

Relationship Between Filter and Channel Pruning (Duality)

#

Filter pruning on layer \(l\) removes rows from \(W^{(l)}\) and columns from \(W^{(l+1)}\). Channel pruning on layer \(l\) removes columns from \(W^{(l)}\) and rows from \(W^{(l-1)}\). They are dual operations:

Layer l-1                Layer l                Layer l+1
[C_out^{l-1} x C_in^{l-1}]  [C_out^l x C_in^l]   [C_out^{l+1} x C_in^{l+1}]

Filter pruning on l:   removes row of W^l    ->  removes col of W^{l+1}
Channel pruning on l:  removes col of W^l    <-  removes row of W^{l-1}

               Filter pruning on layer l
            =  Channel pruning on layer l+1
                   (in terms of effect on W^{l+1})

Neuron/Head Pruning

#

FC Layer Neuron Pruning

#

For a fully connected layer \(y = Wx + b\) with \(W \in \mathbb{R}^{n \times m}\), pruning neuron \(i\) means removing row \(i\) of \(W\), element \(i\) of \(b\), and column \(i\) of the next layer’s weight matrix. This is mathematically identical to filter pruning but for FC layers.

Attention Head Pruning in Transformers (Michel et al., 2019)

#

A multi-head attention layer computes:

where \(\text{head}_h = \text{Attention}(QW^Q_h, KW^K_h, VW^V_h)\).

Head importance score derivation:

Define a mask variable \(\xi_h \in {0, 1}\) for each head:

The importance of head \(h\) is the expected sensitivity of the loss to masking it:

By the chain rule:

where \(\text{Attn}_h = \text{head}_h \cdot W^O_h\) is the contribution of head \(h\) to the output. Therefore:

In practice, the expectation is estimated over a validation set. Michel et al. found that in BERT-base (12 layers, 12 heads each = 144 heads), up to 40% of heads can be pruned with less than 1% accuracy drop on many NLP benchmarks.

ASCII Diagram: Transformer Block Before/After Head Pruning

#

BEFORE: Multi-Head Attention (H=8 heads, d_model=512, d_k=64)

Input (512-dim)
    |
    +--[W_Q0, W_K0, W_V0]---> Head 0 (d_k=64)---+
    +--[W_Q1, W_K1, W_V1]---> Head 1 (d_k=64)---+
    +--[W_Q2, W_K2, W_V2]---> Head 2 (d_k=64)---+
    +--[W_Q3, W_K3, W_V3]---> Head 3 (d_k=64)---+--Concat--> W_O --> 512-dim
    +--[W_Q4, W_K4, W_V4]---> Head 4 (d_k=64)---+   (512x512)
    +--[W_Q5, W_K5, W_V5]---> Head 5 (d_k=64)---+
    +--[W_Q6, W_K6, W_V6]---> Head 6 (d_k=64)---+
    +--[W_Q7, W_K7, W_V7]---> Head 7 (d_k=64)---+

Parameters: 3 * (512*64) * 8 + 512*512 = 786,432 + 262,144 = 1,048,576

---------- Prune heads {1, 3, 5, 7} (50% head pruning) ----------

AFTER: Multi-Head Attention (H=4 heads, d_model=512, d_k=64)

Input (512-dim)
    |
    +--[W_Q0, W_K0, W_V0]---> Head 0 (d_k=64)---+
    +--[W_Q2, W_K2, W_V2]---> Head 2 (d_k=64)---+--Concat--> W_O --> 512-dim
    +--[W_Q4, W_K4, W_V4]---> Head 4 (d_k=64)---+   (256x512)
    +--[W_Q6, W_K6, W_V6]---> Head 6 (d_k=64)---+

Parameters: 3 * (512*64) * 4 + 256*512 = 393,216 + 131,072 = 524,288
Reduction: 50% of attention parameters

Layer Pruning

#

Layer pruning removes entire layers from deep networks. This is the coarsest form of structured pruning.

Layer Importance Estimation

#

For a network \(f = f_L \circ f_{L-1} \circ \cdots \circ f_1\), the importance of layer \(l\) can be estimated as the increase in loss when the layer is bypassed:

For networks with residual connections, “without \(f_l\)” means replacing \(x_{l+1} = x_l + f_l(x_l)\) with \(x_{l+1} = x_l\) (identity shortcut).

ResNet Layer Removal Studies

#

Veit et al. (2016) showed that individual residual blocks in ResNet can be removed at test time with surprisingly small accuracy drops. For ResNet-110 on CIFAR-10:

• Removing 1 block from the middle: ~0.2% accuracy drop
• Removing 5 blocks: ~1.5% accuracy drop
• Removing 10 blocks: ~4% accuracy drop

This works because residual connections ensure that \(x_{l+1} = x_l + f_l(x_l)\), and if \(f_l(x_l)\) is small (which BN regularization encourages), then skipping the block has minimal effect.

When Is Layer Pruning Viable

#

Layer pruning requires skip connections (residual, dense, or highway connections). Without them, removing a layer completely disconnects the forward pass. This is why layer pruning is primarily studied in:

• ResNets and ResNeXt (residual connections)
• DenseNets (dense connections provide alternative paths)
• Transformers (residual connections around attention and FFN)

Block/Group Pruning

#

Block pruning removes rectangular groups of weights, providing a middle ground between unstructured and fully structured pruning.

Common Block Patterns

#

1x1 (unstructured):    4x1 (vector):        2x4 (block):
[x . . .]             [x . . .]             [x x x x]
[. . x .]             [x . . .]             [x x x x]
[. x . .]             [x . . .]             [. . . .]
[. . . x]             [x . . .]             [. . . .]

1x4 (row vector):      4x4 (tile):
[. . . .]             [x x x x]
[x x x x]             [x x x x]
[. . . .]             [x x x x]
[. . . .]             [x x x x]

Legend: x = nonzero, . = zero (pruned)

Bank-Balanced Sparsity

#

Bank-balanced sparsity (Cao et al., 2019) partitions the weight matrix into banks (groups of consecutive rows or columns) and enforces the same number of nonzeros per bank. This enables balanced workload distribution across parallel hardware units.

For a matrix \(W \in \mathbb{R}^{m \times n}\) with bank size \(B\):

• Partition rows into \(m/B\) banks, each with \(B\) rows.
• Within each bank, maintain exactly \(k\) nonzero columns (out of \(n\)).
• Every bank has identical compute workload: \(B \times k\) multiply-accumulates.

This ensures no hardware unit is idle, achieving near-theoretical speedup.

N:M Structured Sparsity (NVIDIA)

#

Definition

#

N:M sparsity requires exactly \(N\) nonzero values in every group of \(M\) consecutive elements along a specific dimension of the weight matrix. The most important instance is 2:4 sparsity: exactly 2 nonzero values per group of 4.

2:4 Sparsity on NVIDIA Ampere/Hopper

#

NVIDIA introduced hardware support for 2:4 sparsity in the Ampere architecture (A100, 2020), continued in Hopper (H100, 2022) and Blackwell (B100/B200, 2024).

Key properties:

• Exactly 50% of weights are zero (2 out of every 4).
• The Sparse Tensor Core achieves 2x throughput compared to the Dense Tensor Core for the same matrix dimensions.
• The sparsity pattern is stored as a compact 2-bit index per group.

Hardware Sparse Tensor Core Operation

#

The sparse matrix multiply works as follows:

Dense A (16x8, FP16)  x  Sparse B (8x16, 2:4 pattern)  =  C (16x16, FP32)

Sparse B storage:
  Original B (8x16):
  [0.5  0.0  0.3  0.0 | 0.0  0.7  0.0  0.1 | ...]
  [0.0  0.2  0.0  0.8 | 0.4  0.0  0.0  0.6 | ...]
  ...

  Compressed B (8x8) + metadata:
  [0.5  0.3  0.7  0.1  ...]     <- nonzero values only (half the columns)
  [0.2  0.8  0.4  0.6  ...]
  ...

  Metadata (2-bit indices per group of 4):
  [00 10 | 01 11 | ...]          <- positions: (0,2) and (1,3) in each group
  [01 11 | 00 11 | ...]

Hardware operation:
  1. Load dense tile of A (16x8)
  2. Load compressed tile of B (8x8) + metadata
  3. Use metadata to select which columns of A to multiply
  4. Execute dense 16x8 x 8x8 multiply (half the original 16x8 x 8x16)
  5. Accumulate into C (16x16)

Result: Same output as dense multiply, but 2x throughput

The key insight is that the hardware uses the metadata to dynamically gather the appropriate elements of \(A\), then performs a dense multiply on the compressed operands. This avoids the irregular access patterns of general sparse formats.

ASCII Diagram of 2:4 Pattern in a Weight Matrix

#

Original Dense Weight Matrix W (8x8):
+------+------+------+------+------+------+------+------+
| 0.82 |-0.15 | 0.91 | 0.03 | 0.44 | 0.02 |-0.68 | 0.11 |
| 0.07 | 0.68 |-0.11 | 0.44 |-0.38 | 0.56 | 0.01 |-0.09 |
| 0.23 |-0.02 | 0.05 | 0.77 | 0.90 |-0.34 | 0.12 | 0.67 |
| 0.45 | 0.31 |-0.88 | 0.04 | 0.19 | 0.73 |-0.55 | 0.08 |
+------+------+------+------+------+------+------+------+

Apply 2:4 sparsity (keep 2 largest per group of 4):

Group boundaries:     [----group 1----] [----group 2----]

Row 0: [0.82, -0.15, 0.91, 0.03] -> keep 0.82, 0.91   (idx 0,2)
       [0.44,  0.02,-0.68, 0.11] -> keep 0.44,-0.68    (idx 0,2)

Row 1: [0.07,  0.68,-0.11, 0.44] -> keep 0.68, 0.44    (idx 1,3)
       [-0.38, 0.56, 0.01,-0.09] -> keep -0.38, 0.56   (idx 0,1)

2:4 Sparse Matrix:
+------+------+------+------+------+------+------+------+
| 0.82 |  0   | 0.91 |  0   | 0.44 |  0   |-0.68 |  0   |
|  0   | 0.68 |  0   | 0.44 |-0.38 | 0.56 |  0   |  0   |
| 0.23 |  0   |  0   | 0.77 | 0.90 |  0   |  0   | 0.67 |
| 0.45 |  0   |-0.88 |  0   |  0   | 0.73 |-0.55 |  0   |
+------+------+------+------+------+------+------+------+

Compressed Storage (nonzeros only):
+------+------+------+------+
| 0.82 | 0.91 | 0.44 |-0.68 |   Metadata: [00,10 | 00,10]
| 0.68 | 0.44 |-0.38 | 0.56 |   Metadata: [01,11 | 00,01]
| 0.23 | 0.77 | 0.90 | 0.67 |   Metadata: [00,11 | 00,11]
| 0.45 |-0.88 | 0.73 |-0.55 |   Metadata: [00,10 | 01,10]
+------+------+------+------+
(50% memory for values + small metadata overhead)

How 2:4 Is Enforced

#

The simplest enforcement: for each group of 4 consecutive weights, keep the 2 with largest magnitude and zero the rest.

Algorithm:

for each row in W:
    for g in range(0, n_cols, 4):
        group = W[row, g:g+4]
        magnitudes = abs(group)
        # Find indices of 2 smallest
        sorted_idx = argsort(magnitudes)
        # Zero the 2 smallest
        W[row, g + sorted_idx[0]] = 0
        W[row, g + sorted_idx[1]] = 0

Numerical example:

Group: \([0.45, 0.31, -0.88, 0.04]\)

Magnitudes: \([0.45, 0.31, 0.88, 0.04]\)

Sorted indices by magnitude: \([3, 1, 0, 2]\) (0.04, 0.31, 0.45, 0.88)

Zero indices 3 and 1: \([0.45, 0, -0.88, 0]\) – kept the two largest magnitudes.

Training with N:M Sparsity: SR-STE

#

Straight-Through Estimator (STE) is the standard approach for training with discrete constraints. SR-STE (Sparse-Refined STE) by Zhou et al. (2021) refines this for N:M sparsity.

Standard STE for N:M sparsity:

Forward pass uses the sparse weights:

where \(M(W)\) is the mask selecting the top-\(N\) magnitudes per group of \(M\).

Backward pass ignores the pruning (straight-through):

Problem: STE allows pruned weights to grow large during training because they never participate in the forward pass but receive gradient updates. When the mask is recomputed, large formerly-pruned weights may suddenly appear, causing instability.

SR-STE solution: Decay the pruned weights toward zero:

where:

• \(M^{(t)} \odot \frac{\partial \mathcal{L}}{\partial W_s^{(t)}}\): standard gradient update for non-pruned weights
• \(\lambda (1 - M^{(t)}) \odot W^{(t)}\): weight decay on pruned weights, pushing them toward zero

This ensures pruned weights stay small, making the mask more stable across training iterations.

Full training algorithm:

1. Initialize dense weights \(W^{(0)}\).
2. For each training step \(t\):
   • a. Compute mask: \(M^{(t)} = \text{TopN:M}(|W^{(t)}|)\)
   • b. Forward: \(W_s^{(t)} = W^{(t)} \odot M^{(t)}\), compute \(\mathcal{L}\)
   • c. Backward: compute \(g = \partial \mathcal{L} / \partial W_s^{(t)}\)
   • d. Update: \(W^{(t+1)} = W^{(t)} - \eta [M^{(t)} \odot g + \lambda(1-M^{(t)}) \odot W^{(t)}]\)
3. Final model uses \(W_s = W \odot M\).

N:M Beyond 2:4

#

Mathematical Analysis: Why 2:4 Is the Sweet Spot

#

The quality of an N:M pattern depends on the expected approximation error. For a group of \(M\) weights drawn i.i.d. from a symmetric distribution with variance \(\sigma^2\), the error from zeroing the \(M-N\) smallest is:

where \(W_{(k)}\) denotes the \(k\)-th order statistic.

For Gaussian weights, the expected squared magnitude of the \(k\)-th smallest out of \(M\) is:

(This is an approximation; exact expressions involve incomplete beta functions.)

For 2:4: We remove the 2 smallest out of 4, each with expected squared magnitude roughly \(0.32\sigma^2\) and \(0.68\sigma^2\). Total error: \(\approx 1.0\sigma^2\) per group. Fraction of total energy pruned: \(1.0 / (4 \cdot 1.0) = 25%\).

For 1:4: We remove 3 out of 4, pruning about \(\approx 2.18\sigma^2\) per group. Fraction pruned: \(\approx 55%\). Much more destructive.

The 2:4 pattern hits the sweet spot: 50% sparsity (which the hardware can double the throughput for) with only ~25% of the weight energy removed (which fine-tuning easily recovers).

Pruning Criteria for Structured Pruning

#

Norm-Based Criteria

#

L1-Norm of Filters

#

Derivation: The L1-norm is the tightest convex relaxation of the L0-norm (number of nonzeros). Minimizing \(||W||_1\) encourages sparsity. Conversely, filters with large L1-norm carry more “weight” in the computation. The expected output magnitude of filter \(i\) is proportional to its L1-norm when inputs have symmetric distributions.

Advantages: Simple, fast to compute, no data needed. Disadvantages: Does not account for correlations between filters, or the actual data distribution.

L2-Norm of Filters

#

The L2-norm measures the energy of the filter. It is related to the expected output variance:

Filters with small L2-norm produce low-variance activations, contributing less to downstream discrimination.

Batch-Norm Scaling Factor (Network Slimming, Liu et al., 2017)

#

This elegant method repurposes the BatchNorm scaling factor \(\gamma\) as a built-in importance indicator.

Background: BatchNorm normalizes activations channel-wise:

where \(\gamma_c\) and \(\beta_c\) are learned per-channel parameters. If \(\gamma_c \to 0\), then channel \(c\) is effectively zeroed out regardless of the filter weights.

Method: Add L1 regularization on \(\gamma\) during training:

Full training procedure:

1. Train the network with the modified loss \(\mathcal{L}_{\text{total}}\).
2. The L1 penalty on \(\gamma\) drives unimportant channels’ \(\gamma_c\) toward zero.
3. After training, rank all \(\gamma_c\) across the network.
4. Set a global threshold \(\theta\) such that a fraction \(s\) of channels have \(|\gamma_c| < \theta\).
5. Prune those channels (and corresponding filters, BN parameters in adjacent layers).
6. Fine-tune the pruned network.

Derivation of the L1 proximal gradient step: Since \(|\gamma|\) is non-smooth at zero, standard SGD cannot be directly applied. Instead, use the proximal gradient:

where the proximal operator for L1 is the soft-thresholding function:

In practice, PyTorch’s SGD with weight decay on \(\gamma\) approximates this (though not exactly, since weight decay is L2, not L1). Correct implementation requires a custom optimizer step.

Reconstruction-Based Criteria

#

The idea is to prune structures that cause the minimal change in the layer’s output.

Formulation: Given input activations \(X \in \mathbb{R}^{N \times d_{in}}\) and current output \(Y = XW\), find a subset \(S\) of columns to keep (i.e., input features/channels) and new weights \(W’\) such that:

where \(p\) is the number of channels/features to prune.

Derivation for the optimal W’ given S:

This is a standard least-squares problem. Partition \(X = [X_S, X_{\bar{S}}]\) and \(W = [W_S; W_{\bar{S}}]\). The original output is:

The reconstruction target is \(Y\) and the model is \(X_S W’\). Setting the derivative to zero:

This is the ordinary least-squares solution. The reconstruction error for subset \(S\) is:

where \(P_S = X_S (X_S^T X_S)^{-1} X_S^T\) is the projection matrix onto the column space of \(X_S\). The optimal \(S\) minimizes this projection residual.

Gradient/Taylor-Based Criteria

#

First-Order Taylor Expansion

#

The importance of a pruning group \(g\) (filter, channel, head) can be estimated via a first-order Taylor expansion of the loss around the current parameters:

Derivation: Let \(\delta W\) be the change in weights when group \(g\) is removed: \(\delta W_i = -W_i\) for \(i \in g\), \(\delta W_i = 0\) otherwise. By Taylor expansion:

The importance of group \(g\) is therefore:

Activation-based variant (Molchanov et al., 2017): Instead of weight gradients, use activation gradients. For filter \(i\) producing activation \(a_i\):

This is mathematically equivalent (by chain rule) but numerically more stable and easier to compute in practice, since activation gradients are readily available during backpropagation.

Second-Order (Hessian) Criteria

#

The second-order Taylor expansion gives a more accurate importance estimate:

where \(g_i = \partial \mathcal{L}/\partial W_i\) and \(H_{ij} = \partial^2 \mathcal{L}/\partial W_i \partial W_j\).

Computing the full Hessian \(H\) is \(O(n^2)\) in parameters, which is infeasible for large networks. Approximations include:

• Diagonal Hessian: \(H_{ij} \approx 0\) for \(i \neq j\), giving \(I_g = |\sum_{i \in g} W_i g_i - \frac{1}{2} H_{ii} W_i^2|\)
• Fisher Information Matrix: \(H \approx F = \mathbb{E}[gg^T]\), which can be estimated from gradient samples
• Hessian trace: \(\text{tr}(H_g) = \sum_{i \in g} H_{ii}\), estimated via Hutchinson’s trace estimator

Hutchinson’s trace estimator derivation:

For any square matrix \(A\), if \(v\) is a random vector with \(\mathbb{E}[vv^T] = I\) (e.g., Rademacher \(\pm 1\) entries):

The Hessian-vector product \(Hv\) can be computed efficiently via automatic differentiation (one extra backward pass), so the trace can be estimated without materializing \(H\).

Accumulated Gradient Information

#

In practice, importance scores computed from a single minibatch are noisy. The standard approach is to accumulate importance scores over multiple batches:

Some methods use exponential moving averages for online estimation:

Learning-Based Criteria

#

Learnable Pruning Masks with Gumbel-Softmax

#

Instead of using a heuristic criterion, learn the pruning mask end-to-end. The mask \(M \in {0,1}^G\) (one bit per group) is a discrete variable, which is non-differentiable. The Gumbel-Softmax trick provides a continuous relaxation.

Derivation: For a binary mask variable \(m_g\) (keep or prune group \(g\)):

where \(\sigma\) is the sigmoid function and \(\alpha_g\) is a learnable logit.

The Gumbel-Softmax relaxation replaces the discrete sample with:

where \(u \sim \text{Uniform}(0,1)\) and \(\tau\) is a temperature parameter. As \(\tau \to 0\), \(\tilde{m}_g \to m_g\) (discrete). During training, \(\tau\) is annealed from a high value (smooth, easy to optimize) to a low value (near-discrete).

The loss becomes:

where the regularization term encourages sparsity by penalizing the probability of keeping each group.

AMC: AutoML for Model Compression

#

AMC (He et al., 2018) uses reinforcement learning to find per-layer pruning ratios automatically.

State: For layer \(l\), the state vector includes:

• Layer type (conv, FC, etc.)
• Layer dimensions (\(C_{in}, C_{out}, k, H, W\))
• Current FLOPs and parameter count
• Remaining FLOPs budget
• Layer index

Action: Continuous action \(a_l \in [0, 1]\) specifying the pruning ratio for layer \(l\). (E.g., \(a_l = 0.3\) means prune 30% of filters in layer \(l\).)

Reward: After pruning all layers with the chosen ratios and brief fine-tuning:

If the constraint is violated, a large negative penalty is applied.

Policy: A DDPG (Deep Deterministic Policy Gradient) agent learns a policy \(\pi(a_l | s_l)\) that maps layer states to pruning ratios. The agent processes layers sequentially, observing the updated state after each pruning decision.

Key finding: AMC consistently outperforms hand-crafted uniform pruning ratios. The learned policies tend to prune more aggressively in redundant layers (early conv layers with many similar filters) and preserve layers that are bottlenecks.

Structured Pruning Algorithms (Deep Dive)

#

DepGraph (Dependency Graph-Based Pruning, 2023)

#

Modern architectures have complex topologies (residual connections, concatenation, split, group convolutions) that make structured pruning non-trivial. Pruning a filter in one layer may require simultaneously pruning corresponding structures in multiple other layers.

The problem: In a ResNet block:

x ---> Conv1 ---> BN1 ---> ReLU ---> Conv2 ---> BN2 ---> (+) ---> out
|                                                          ^
+----------------------------------------------------------+

If we prune filter \(i\) from Conv1, we must also prune:

• BN1’s \(\gamma_i, \beta_i\), running mean/var index \(i\)
• Input channel \(i\) of Conv2

But if there is a residual connection, the output of Conv2 is added to \(x\). If Conv2’s output channels are pruned, the addition dimensions no longer match unless the same channels are pruned from \(x\) (which means pruning the same channels from the preceding layer).

DepGraph solution: Build a dependency graph where nodes are parameter groups and edges represent “must prune together” relationships.

Algorithm:

1. Parse the computational graph of the network.
2. For each layer, identify which dimension of its parameters corresponds to “output features” and “input features.”
3. Create dependency edges:
   • Conv output channels ↔ next layer’s input channels
   • BN parameters ↔ corresponding conv output channels
   • Residual addition: all inputs must have matching channel counts → coupled pruning groups
   • Concatenation: each branch can be pruned independently
4. Group all transitively connected parameters into “pruning groups.”
5. Assign importance scores to each group.
6. Prune the least important groups.

ASCII Diagram: Dependency Graph for ResNet Block

#

Dependency Graph for ResNet Basic Block:

[Conv1 output ch]---dep---[BN1 channels]---dep---[Conv2 input ch]
                                                        |
                                                       dep
                                                        |
[Conv2 output ch]---dep---[BN2 channels]---dep---[Add input ch]
                                                        |
                                                       dep (residual)
                                                        |
                                              [Previous block output ch]
                                                        |
                                                       dep
                                                        |
                                              [Previous BN output ch]
                                                       ...

Pruning Group Example (if we want to prune channel i):
{Conv1.weight[i,:,:,:], BN1.gamma[i], BN1.beta[i],
 Conv2.weight[:,i,:,:]}

For residual-connected channels:
{Conv2.weight[j,:,:,:], BN2.gamma[j], BN2.beta[j],
 Prev_Conv.weight[j,:,:,:], Prev_BN.gamma[j], Prev_BN.beta[j], ...}
 ^--- all layers in the residual chain must prune channel j together

This automatic dependency resolution is what makes DepGraph applicable to arbitrary architectures (EfficientNet, ConvNeXt, Vision Transformers, etc.) without manual per-architecture pruning code.

Group Sparsity Regularization

#

Instead of pruning after training, we can encourage structured sparsity during training through group sparsity regularization.

Group LASSO (L2,1 Norm)

#

Partition the weight matrix into groups \(g_1, g_2, \ldots, g_G\) (e.g., each group is one filter). The group LASSO penalty is:

The training loss becomes:

Why group LASSO induces group sparsity (derivation):

The subdifferential of \(||W_g||_2\) with respect to \(W_g\) is:

At the optimum, for group \(g\):

If \(||\frac{\partial \mathcal{L}_{\text{task}}}{\partial W_g}||_2 \leq \lambda\), then \(W_g = 0\) is optimal (the entire group is zeroed). This is the mechanism by which group LASSO drives entire groups to zero, unlike L2 regularization (weight decay) which shrinks all weights but never exactly zeros them.

Proximal Gradient Descent

#

Since \(||W_g||_2\) is non-smooth at \(W_g = 0\), we use the proximal gradient method:

Derivation of the proximal operator for group LASSO:

The proximal operator is defined as:

This separates across groups. For a single group with parameter \(v_g\):

Case 1: \(u_g = 0\). Objective = \(\frac{1}{2}||v_g||_2^2\).

Case 2: \(u_g \neq 0\). Take derivative and set to zero:

Since both \(u_g\) and \(v_g\) point in the same direction:

Taking norms: \(||u_g|| = ||v_g|| \cdot \frac{||u_g||}{||u_g|| + \eta\lambda}\), so \(||u_g|| + \eta\lambda = ||v_g||\), giving \(||u_g|| = ||v_g|| - \eta\lambda\).

This is valid only when \(||v_g|| > \eta\lambda\). Otherwise, \(u_g = 0\).

Final proximal operator (group soft-thresholding):

This is the block soft-thresholding operator. When \(||v_g||_2 \leq \eta\lambda\), the entire group is set to zero in a single step.

Soft Pruning vs Hard Pruning

#

Hard pruning: Once a structure is pruned, it is permanently removed from the architecture. The pruned network has fewer parameters and cannot recover the pruned capacity.

Soft pruning (He et al., 2018): Set pruned structures to zero but keep them in the architecture. During fine-tuning, pruned weights can be updated (potentially becoming nonzero again). The pruning mask is periodically recomputed.

Hard Pruning Cycle:
Train -> Score -> Prune (permanent) -> Fine-tune -> Done
                    |
          Removed from architecture

Soft Pruning Cycle:
Train -> Score -> Mask (temporary) -> Fine-tune -> Re-score -> Re-mask -> ...
                    |                                 |
          Set to zero, but kept           May unmask previously pruned

Comparison:

Real-World Speedup Analysis

#

Theoretical FLOPs Reduction vs Actual Wall-Clock Speedup

#

The gap between theoretical and actual speedup is the most important practical consideration in pruning. We now analyze why this gap exists and how structured pruning closes it.

Why structured pruning gets real speedup:

1. Dense GEMM on smaller matrices: A pruned convolution with 50% of filters removed is simply a convolution with half the output channels. The hardware runs the same dense operation, just on a smaller tensor. All existing optimizations (tiling, vectorization, Tensor Core utilization) apply perfectly.

2. No sparse format overhead: No index arrays, no indirect memory access, no metadata. The weight tensor is a standard contiguous block.

3. Better memory bandwidth utilization: Smaller tensors mean less data transfer between DRAM, L2 cache, and compute units. For memory-bandwidth-bound operations (small batch sizes, depthwise convolutions), this is the dominant factor.

Speedup Measurements on Different Hardware

#

Key observations:

• Unstructured pruning shows almost no speedup on GPUs even at 90%+ sparsity.
• CPUs fare slightly better with unstructured pruning due to branch-based sparse kernels.
• Structured pruning achieves near-theoretical speedup across all platforms.
• 2:4 sparsity is excellent on NVIDIA GPUs but useless on other hardware.
• Mobile platforms benefit most from structured pruning (memory-bandwidth bound).

Roofline Model Analysis

#

The roofline model relates computational performance to arithmetic intensity (FLOPs per byte of memory transferred).

Performance                    Roofline Model: Sparse vs Dense
(TFLOPS)
    |
  8 |                            xxxxxxxxxxxxxxxxxx  Peak Compute (Dense)
    |                          x
  6 |                        x
    |                      x     oooooooooooooooo  Peak Compute (2:4 Sparse)
  4 |                    x     o
    |         Dense:   x     o
  3 |                x     o
    |              x     o       Structured pruning shifts
  2 |         ___x_____o         operations LEFT (less data)
    |        / x     o           AND stays on dense roofline
  1 |      /x     o
    |    /x    o
    |  /x   o    Unstructured: below roofline due to
    | /  o       irregular access (cache misses, no vectorization)
    |o..........u.u..u...u..u   Unstructured sparse
    +---+----+----+----+----+---> Arithmetic Intensity
        1    2    4    8   16     (FLOPs / Byte)

    x = Dense/Structured (on the roofline)
    o = 2:4 Sparse (on a lower but real roofline)
    u = Unstructured (below any roofline due to overhead)

Key: Structured pruning reduces problem size while staying on the optimal
     roofline. Unstructured pruning falls off the roofline entirely.

Analysis: Dense and structured-pruned operations ride the roofline – they achieve peak performance for their arithmetic intensity. Reducing the tensor size (structured pruning) moves the operating point left on the x-axis (less data to transfer) but stays on the roofline. The actual throughput equals \(\min(\text{peak compute}, \text{bandwidth} \times \text{arithmetic intensity})\).

Unstructured sparse operations fall below the roofline because:

• Cache misses from irregular access reduce effective bandwidth.
• No SIMD/Tensor Core utilization reduces effective peak compute.
• Index overhead increases bytes transferred without adding useful FLOPs.

Pruning + Other Compression Techniques

#

Pruning + Quantization: Compound Compression

#

Pruning and quantization are complementary: pruning reduces the number of parameters, quantization reduces the bits per parameter. The compound compression ratio multiplies:

Example: 50% structured pruning (2x) + INT8 quantization (4x from FP32) = 8x total compression. Plus 50% pruning (2x) + INT4 quantization (8x) = 16x compression.

Order matters: There are two strategies:

Prune first, then quantize:

1. Train dense model.
2. Prune to target sparsity, fine-tune.
3. Quantize the pruned model (PTQ or QAT).

This is simpler but may lose accuracy at the quantization step because the pruned model has fewer parameters to absorb quantization error.

Joint pruning and quantization:

1. Train with both pruning masks and quantization-aware fake quantization.
2. The model adapts to both constraints simultaneously.
3. Generally achieves better accuracy but is more complex to implement.

N:M sparsity + INT8 quantization is the most deployment-friendly combination:

• 2:4 sparsity: 2x Tensor Core throughput
• INT8: 2x throughput over FP16 on Tensor Cores
• Combined: theoretically 4x throughput over dense FP16
• Memory: 50% weights x 50% bits = 25% of original FP16 storage

Pruning + Knowledge Distillation

#

Knowledge distillation uses a large teacher model to guide the training of a smaller student model. When combined with pruning:

Standard distillation loss:

where \(\hat{z}_T, \hat{z}_S\) are teacher and student logits, \(T\) is temperature, and \(\sigma\) is softmax.

Feature-level distillation for structured pruning: When filter/channel pruning changes intermediate feature map dimensions, a linear projection aligns teacher and student feature maps:

where \(P_l\) is a learnable projection matrix that maps the student’s (smaller) feature maps to the teacher’s dimension. This is particularly important for structured pruning because entire feature channels are missing from the student.

Pruning + NAS

#

Neural Architecture Search and pruning share a deep connection: pruning can be viewed as searching for optimal sub-architectures within a larger network.

Once-for-All (OFA) Networks (Cai et al., 2020):

1. Train a single large network that supports elastic depth, width, and kernel size.
2. At deployment time, extract a sub-network matching the target hardware constraints.
3. The sub-network extraction is equivalent to structured pruning (removing layers, channels, reducing kernels).
4. No fine-tuning needed because the large network was trained to support all sub-networks.

This unifies NAS and pruning into a single paradigm: train once, deploy many configurations.

Practical Implementation Guide

#

PyTorch Pruning API (torch.nn.utils.prune)

#

PyTorch provides built-in pruning utilities. Here is how the key functions work:

Unstructured pruning:

import torch.nn.utils.prune as prune

# L1 unstructured: prune 30% of weights by magnitude
prune.l1_unstructured(module, name='weight', amount=0.3)
# Creates: module.weight_mask (binary), module.weight_orig (original)
# module.weight is now a property: weight_orig * weight_mask

# Random unstructured
prune.random_unstructured(module, name='weight', amount=0.3)

# Global unstructured: prune 20% globally across multiple layers
parameters_to_prune = [
    (model.conv1, 'weight'),
    (model.conv2, 'weight'),
    (model.fc1, 'weight'),
]
prune.global_unstructured(
    parameters_to_prune,
    pruning_method=prune.L1Unstructured,
    amount=0.2,
)

Structured pruning:

# Ln structured: prune 40% of filters by L2-norm (dim=0 = output channels)
prune.ln_structured(module, name='weight', amount=0.4, n=2, dim=0)
# This zeros out entire filters (output channels)

# Prune by L1-norm along input channel dimension
prune.ln_structured(module, name='weight', amount=0.3, n=1, dim=1)

Making pruning permanent (removing the reparametrization):

prune.remove(module, 'weight')
# Now module.weight is a regular parameter with zeros baked in
# For structured pruning, you still need to manually resize the tensor
# and adjust adjacent layers

Important caveat: PyTorch’s built-in pruning API only applies masks; it does not physically remove structures. For actual speedup from structured pruning, you must manually reconstruct the network with smaller layers. Libraries like DepGraph, torch-pruning, and NNI handle this automatically.

NVIDIA ASP (Automatic SParsity) for 2:4

#

NVIDIA’s Automatic SParsity library applies 2:4 sparsity to PyTorch models with minimal code:

from apex.contrib.sparsity import ASP

# Prepare model for sparse training
ASP.prune_trained_model(model, optimizer)
# This applies 2:4 masks to all supported layers (Linear, Conv2d)

# Training loop runs normally; masks are maintained
for epoch in range(num_epochs):
    for batch in dataloader:
        loss = model(batch)
        loss.backward()
        optimizer.step()
        # ASP automatically re-applies 2:4 masks after each step

# Export sparse model for inference
# The 2:4 pattern is automatically detected by TensorRT for
# Sparse Tensor Core acceleration

Comparison of Pruning Tools

#

Choosing the Right Tool

#

Decision criteria:

1. If targeting NVIDIA GPU inference with TensorRT: Use NVIDIA ASP for 2:4 sparsity. This is the path of least resistance for guaranteed 2x Tensor Core speedup.

2. If targeting mobile/edge (TFLite, Core ML): Use structured pruning via torch-pruning or TF Model Optimization. Physical tensor size reduction translates directly to latency reduction.

3. If targeting CPU inference: Structured pruning (filter/channel) with Intel Neural Compressor. CPU benefits from both smaller tensors and reduced memory bandwidth.

4. If targeting maximum compression (storage, not latency): Unstructured pruning at high sparsity + quantization. Store in sparse format. Accept no inference speedup on standard hardware.

5. If working with complex architectures: Use DepGraph (torch-pruning) for automatic dependency resolution.

Summary

#

Structured vs Unstructured Decision Matrix

#

                     Need real speedup    Need max compression    Have sparse HW
                     on standard HW?      (storage only)?         (Ampere+)?
                          |                     |                      |
                         YES                   YES                    YES
                          |                     |                      |
                    Structured              Unstructured            N:M (2:4)
                    Pruning                 Pruning                 Sparsity
                          |                     |                      |
                  Filter/Channel            Magnitude              NVIDIA ASP
                  + DepGraph                + Sparse Format        + Fine-tune
                          |                     |                      |
                  Actual 1.5-3x            Theoretical 10-20x      Actual 2x
                  speedup                   (1.0x on GPU)          on Tensor Cores

Complete Comparison Table

#

Key Takeaways

#

1. Unstructured pruning is a compression technique, not an acceleration technique (on commodity hardware). Use it when storage size matters more than inference speed, or when deploying on sparsity-aware hardware like Cerebras.

2. Structured pruning is the only way to get real speedup on GPUs, CPUs, and mobile devices without specialized hardware. It produces smaller dense tensors that exploit all existing hardware optimizations.

3. 2:4 sparsity is the current best compromise for NVIDIA GPU deployment: 50% sparsity with hardware-guaranteed 2x Tensor Core throughput, minimal accuracy loss, and easy implementation via ASP.

4. Dependency-aware pruning (DepGraph) is essential for modern architectures. Manual structured pruning is error-prone and architecture-specific; automatic dependency resolution makes it applicable to any model.

5. Compound compression (pruning + quantization + distillation) yields the best real-world results. The compression ratios multiply, and knowledge distillation recovers accuracy lost to aggressive pruning.

6. The pruning criterion matters less than the pruning granularity for speedup. A simple L1-norm filter pruning with proper fine-tuning often matches sophisticated criteria in accuracy, and the speedup is determined by the structure, not the selection method.

7. Always measure wall-clock time, not FLOPs. A method claiming 10x FLOPs reduction with no wall-clock improvement is not useful for deployment.

Next post: We will explore advanced pruning methods including lottery ticket hypothesis, pruning at initialization, gradual magnitude pruning schedules, and iterative pruning strategies that push the boundaries of how much we can prune while maintaining accuracy.