SignMuon: Matrix Directions, Majority Votes, and the Actual Bytes Sent

September 2026 · 4 min read

Before you start: Muon and polar directions and distributed collectives.

In distributed training, workers must agree on a parameter update. Sending a full float32 matrix costs four bytes per entry before collective overhead. A sign matrix carries much less information, but the communication primitive and aggregation rule determine whether that information becomes fewer transmitted bytes. SignMuon combines local matrix transformations with a majority vote over signs.

Keep local and shared operations separate

Each worker computes its own momentum and polar direction before communicating signs. The shared majority matrix supplies the update, with scaling according to the chosen variant. An optional polar transformation after aggregation is local because every worker already has the same aggregate. Specify how zeros and tied votes are handled so replicas cannot diverge.

For one entry, worker directions [100,−1,−1] have a positive arithmetic mean but a negative majority sign. Majority vote deliberately discards magnitude. It is not a quantized implementation of exact gradient averaging. This distinction affects optimization behavior as well as communication.

A tiny aggregation calculation

import numpy as np

directions = np.array([
    [[ 2., -1.], [ 1.,  3.]],
    [[-1., -2.], [ 2., -1.]],
    [[-1.,  1.], [ 4.,  2.]],
])
signs = np.sign(directions).astype(np.int32)
vote_sum = signs.sum(axis=0, dtype=np.int32)
majority = np.sign(vote_sum)
assert majority.tolist() == [[-1, -1], [1, 1]]
# Here a tied vote maps to zero. Other policies must be explicit.

One bit of information is not automatically a one-bit all-reduce

For d entries, a float32 vector has 4d bytes of payload. Packed signs have d/8 bytes, an ideal 32-fold payload reduction. An int8 representation has d bytes, only a fourfold reduction. Accumulating P signs also requires enough range for sums from −P to P; an int8 sum can overflow when worker counts are too large. The collective’s actual accumulator type matters.

A ring all-reduce’s bandwidth term is roughly 2(P−1)/P times the uncompressed buffer size per worker. A packed-sign all-gather transmits each worker’s signs to every other worker, costing roughly (P−1)d/8 bytes per worker. Its scaling differs from a reduction. Compare end-to-end collectives at the actual worker count instead of multiplying an ideal compression ratio into a claimed training speedup.

Match a guarantee to its assumptions and direction

The paper discusses spectral geometry and sign-reliability assumptions, including distinctions between normalized sign directions and polar-transformed directions. A theorem for one direction is not automatically a theorem for every composition of polar and sign. Majority-vote benefits also depend on worker noise and agreement assumptions; correlated worker errors can survive the vote.

Treat optional transforms, normalization, momentum, and communication implementations as named variants. Record which variant generated each curve. In particular, a practical local-polar-then-sign direction should not inherit a descent claim merely because raw-gradient sign descent has one under another set of assumptions.

The experiment needs three clocks

Measure gradient computation, local matrix-transformation time, and communication time separately, then measure the total step. Overlap can make their sum exceed wall-clock duration, so keep traces as well as totals. Compare matched effective batch sizes and data order when evaluating distributed scaling; increasing workers while changing the optimization problem obscures attribution.

Report validation quality versus examples processed and wall time, actual network bytes, collective implementation, worker count, and accumulator dtype. Check replica agreement after every shared update in a small deterministic run. A communication-efficient representation is useful only if the chosen implementation and optimization behavior preserve the intended advantage.

Sources and further reading

• Mishra, Trivedi & Kumar: SignMuon, arXiv v1 — The specific majority-vote matrix optimizer discussed here; similarly named methods can use different algorithms.
• Bernstein et al.: signSGD — Sign-based communication and majority voting.
• Keller Jordan: Muon implementation — Matrix-shaped updates, Newton–Schulz approximation, and parameter grouping.