Skip to main content

physics.communication

physics.communication

Collective communication time models (α–β).

All collectives here are pure communication models: the local reduction compute term (γ in Thakur et al. 2005) is deliberately omitted, matching the book’s α–β pedagogical treatment.

Functions

Name Description
calc_all_to_all_time Estimate an equal-partition ring-style All-to-All exchange.
calc_alpha_beta_crossover Find the message size where startup and transfer costs are equal.
calc_bisection_bandwidth Sum effective link capacity across a specified partition cut.
calc_double_binary_tree_allreduce_time Approximate a double-tree collective with declared overhead factors.
calc_fat_tree_hosts Count hosts in the modeled nonblocking fat-tree.
calc_fat_tree_path_switches Count switch traversals on the longest fat-tree path.
calc_fat_tree_switches Count switches in the modeled full fat-tree.
calc_hierarchical_allreduce_time Compose local scatter, inter-node reduction, and local gather.
calc_hop_latency Sum fixed latency over identical sequential hops.
calc_oversubscription_effect Estimate throughput after stretching the communication portion of a step.
calc_point_to_point_time Estimate startup plus serialization time for one transfer.
calc_ring_allreduce_data_factor Return per-rank ring AllReduce traffic in message equivalents.
calc_ring_allreduce_latency_steps Count startup terms in ring AllReduce.
calc_ring_allreduce_latency_time Compute the startup-only component of ring AllReduce.
calc_ring_allreduce_time Estimate ring AllReduce communication time.
calc_ring_attention_min_chunk Estimate the token chunk needed to hide a ring-attention transfer.
calc_ring_collective_data_factor Return a single ring-phase traffic multiplier.
calc_ring_tree_crossover_size Estimate a large-rank ring/tree crossover payload.
calc_torus_bisection_links Count links in a planar cut through a regular torus.
calc_torus_diameter Count hops in the longest shortest path of a regular torus.
calc_tree_allreduce_time Estimate reduction and broadcast over a binary tree.
ring_allreduce_data_factor_latex Format the ring AllReduce data factor as LaTeX.

calc_all_to_all_time

physics.communication.calc_all_to_all_time(
    message_bytes,
    n_gpus,
    bandwidth_bytes_s,
    latency_s,
)

Estimate an equal-partition ring-style All-to-All exchange.

Parameters

Name Type Description Default
message_bytes Quantity or float Nonnegative message size M; bare values are bytes. required
n_gpus int Participant count N, at least one; intended to be integral. required
bandwidth_bytes_s Quantity or float Positive unidirectional bandwidth B; bare values are bytes/second. required
latency_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required

Returns

Name Type Description
Quantity Exchange time in seconds; zero for one participant.

Notes

Equation: T = (N - 1) M/(N B) + (N - 1) alpha.

Assumptions: M is the per-rank total send buffer, divided into N equal destination chunks; one chunk stays local.

Excludes: Unequal expert routing, fabric contention, local packing, and reduction compute.

References

Derivation: Each rank sends N - 1 chunks of size M/N in sequential exchange rounds.

Examples

>>> calc_all_to_all_time(1024, 1, 1024, 0).m_as("s") == 0
True

calc_alpha_beta_crossover

physics.communication.calc_alpha_beta_crossover(alpha_s, bandwidth_bytes_s)

Find the message size where startup and transfer costs are equal.

Parameters

Name Type Description Default
alpha_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required
bandwidth_bytes_s Quantity or float Positive unidirectional bandwidth B; bare values are bytes/second. required

Returns

Name Type Description
Quantity Crossover message size in bytes.

Notes

Equation: M_cross = alpha B.

Assumptions: Constant startup latency and size-independent bandwidth.

Excludes: Protocol thresholds, contention, and collective round counts.

References

Derivation: Solve alpha = M/B for M in the point-to-point alpha-beta model.

Examples

>>> m = calc_alpha_beta_crossover(Q_("1 us"), Q_("1 GB/s"))
>>> (m / Q_("1 GB/s")).to("us") == Q_("1 us")
True

calc_bisection_bandwidth

physics.communication.calc_bisection_bandwidth(
    num_ports,
    link_bandwidth,
    oversubscription_ratio=1.0,
)

Sum effective link capacity across a specified partition cut.

Parameters

Name Type Description Default
num_ports int Count of links crossing the cut, at least one; not all fabric ports. required
link_bandwidth Quantity or float Positive capacity per crossing link; bare values are bytes/second. required
oversubscription_ratio float Positive capacity divisor; default one means no reduction. 1.0

Returns

Name Type Description
Quantity Effective cut bandwidth in bytes/second.

Notes

Equation: B_cut = num_ports B_link / r.

Assumptions: Identical crossing links; the specified ratio captures the shared upstream capacity.

Excludes: Routing imbalance and traffic not crossing the selected cut.

For a nonblocking fabric with N endpoints, N/2 links may cross the chosen cut; counting all ports would overstate this cut capacity.

References

Derivation: Add parallel crossing-link capacities and divide by the supplied sharing factor.

Examples

>>> calc_bisection_bandwidth(4, Q_("1 GB/s"), 2) == 2 * Q_("1 GB/s")
True

calc_double_binary_tree_allreduce_time

physics.communication.calc_double_binary_tree_allreduce_time(
    message_bytes,
    n_gpus,
    bandwidth_bytes_s,
    latency_s,
    latency_factor=1.2,
    bandwidth_factor=1.05,
)

Approximate a double-tree collective with declared overhead factors.

Parameters

Name Type Description Default
message_bytes Quantity or float Nonnegative message size M; bare values are bytes. required
n_gpus int Participant count N, at least one; intended to be integral. required
bandwidth_bytes_s Quantity or float Positive unidirectional bandwidth B; bare values are bytes/second. required
latency_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required
latency_factor float Nonnegative startup multiplier; package assumption defaults to 1.2. 1.2
bandwidth_factor float Nonnegative payload multiplier; package assumption defaults to 1.05. 1.05

Returns

Name Type Description
Quantity Approximate collective time in seconds; zero for one participant.

Notes

Equation: T = f_latency 2 log2(N) alpha + f_bandwidth 2(N - 1) M/(N B).

Assumptions: Logarithmic startup and near-ring traffic volume; the factors are modeling assumptions, not measurements.

Excludes: Exact NCCL scheduling, reduction compute, and topology-specific routing.

The startup term uses fractional log2(N), unlike the discrete level count in calc_tree_allreduce_time. Defaults do not establish implementation accuracy.

References

Derivation: Combine a smooth logarithmic startup term with the ring-optimal traffic term; the user-supplied factors perturb these idealized costs.

Examples

>>> calc_double_binary_tree_allreduce_time(1024, 1, 1024, 0).m_as("s") == 0
True

calc_fat_tree_hosts

physics.communication.calc_fat_tree_hosts(radix, tiers)

Count hosts in the modeled nonblocking fat-tree.

Parameters

Name Type Description Default
radix int Switch port count k, at least two; intended to be even. Odd inputs use floor(k/2). required
tiers int Positive number of switching tiers L. required

Returns

Name Type Description
int Host count under the equal up/down port split.

Notes

Equation: H = 2 floor(k/2)^L.

Assumptions: Identical even-radix switches, full population, and half the non-top ports facing each direction.

Excludes: Oversubscription, spare ports, and incomplete pods.

References

Derivation: Starting with a two-way split, multiply host reach by k/2 for each tier. For odd radix the implementation explicitly uses integer division.

Al-Fares, Loukissas, and Vahdat (2008), “A Scalable, Commodity Data Center Network Architecture.” https://doi.org/10.1145/1402958.1402967 (bibliography key: al-fares2008). The tier-general formula above extends the symmetric port-count construction.

Examples

>>> calc_fat_tree_hosts(4, 3) == calc_fat_tree_hosts(4, 2) * (4 // 2)
True

calc_fat_tree_path_switches

physics.communication.calc_fat_tree_path_switches(tiers)

Count switch traversals on the longest fat-tree path.

Parameters

Name Type Description Default
tiers int Positive number of switching tiers L. required

Returns

Name Type Description
int Number of switches traversed between hosts in different top-level branches.

Notes

Equation: S_path = 2L - 1.

Assumptions: Route ascends from the source leaf to the top tier and descends to the destination leaf.

Excludes: Host NICs, retries, and alternate routing detours.

References

Derivation: Count L ascending switches and L - 1 descending switches without counting the top switch twice.

Examples

>>> calc_fat_tree_path_switches(3) == 2 * 3 - 1
True

calc_fat_tree_switches

physics.communication.calc_fat_tree_switches(radix, tiers)

Count switches in the modeled full fat-tree.

Parameters

Name Type Description Default
radix int Switch port count k, at least two; intended to be even. Odd inputs use floor(k/2). required
tiers int Positive number of switching tiers L. required

Returns

Name Type Description
int Total switch count for the assumed topology.

Notes

Equation: S = (2L - 1) floor(k/2)^(L - 1).

Assumptions: Full symmetric fabric with even radix and equal up/down splits.

Excludes: Management switches, spares, and partially populated tiers.

References

Derivation: Count one top-tier group and two groups per remaining tier, each with (k/2)^(L - 1) switches under this regular construction.

Al-Fares, Loukissas, and Vahdat (2008), “A Scalable, Commodity Data Center Network Architecture.” https://doi.org/10.1145/1402958.1402967 (bibliography key: al-fares2008). The tier-general expression follows the regular construction described above.

Examples

>>> calc_fat_tree_switches(4, 1) == 1
True

calc_hierarchical_allreduce_time

physics.communication.calc_hierarchical_allreduce_time(
    message_bytes,
    n_nodes,
    gpus_per_node,
    intra_node_bw,
    inter_node_bw,
    intra_node_lat=Q_('500 ns'),
    inter_node_lat=Q_('5 us'),
)

Compose local scatter, inter-node reduction, and local gather.

Parameters

Name Type Description Default
message_bytes Quantity or float Nonnegative message size M; bare values are bytes. required
n_nodes int Positive physical node count n. required
gpus_per_node int Positive ranks per node g. required
intra_node_bw Quantity or float Positive unidirectional local bandwidth; bare values are bytes/second. required
inter_node_bw Quantity or float Positive per-GPU NIC bandwidth; bare values are bytes/second. required
intra_node_lat Quantity or float Nonnegative local round startup; default 500 ns; bare values are seconds. Q_('500 ns')
inter_node_lat Quantity or float Nonnegative inter-node round startup; default 5 us; bare values are seconds. Q_('5 us')

Returns

Name Type Description
Quantity Sum of all three sequential phase costs in seconds.

Notes

Equation: T = 2[(g - 1) M/(g B_local) + (g - 1) alpha_local] + T_ring(M/g, n, B_network, alpha_network).

Assumptions: Local ReduceScatter and AllGather surround g concurrent inter-node rings; each local GPU has its own NIC capacity.

Excludes: Shared-NIC contention, phase overlap, packing, and local reduction arithmetic.

If all local GPUs share one NIC, the inter-node traffic term needs an aggregate-capacity treatment; this independent-NIC model understates that contention.

References

Derivation: Each local ring phase moves (g - 1)/g of M. Each GPU then reduces its M/g shard across nodes before local gathering.

Examples

>>> calc_hierarchical_allreduce_time(1024, 1, 1, 1024, 1024).m_as("s") == 0
True

calc_hop_latency

physics.communication.calc_hop_latency(hops, hop_latency_s)

Sum fixed latency over identical sequential hops.

Parameters

Name Type Description Default
hops int Nonnegative number of sequential hops. required
hop_latency_s Quantity or float Nonnegative per-hop latency; bare values are seconds. required

Returns

Name Type Description
Quantity Total hop latency in seconds.

Notes

Equation: T = hops alpha.

Assumptions: Each hop contributes the same serial delay.

Excludes: Packet serialization, queueing, and overlapping hop processing.

References

Derivation: Add one identical latency contribution per traversed hop.

Examples

>>> calc_hop_latency(0, Q_("1 us")).m_as("s") == 0
True

calc_oversubscription_effect

physics.communication.calc_oversubscription_effect(
    comm_fraction,
    oversubscription_ratio,
)

Estimate throughput after stretching the communication portion of a step.

Parameters

Name Type Description Default
comm_fraction float Baseline communication time fraction f in [0, 1]. required
oversubscription_ratio float Positive stretch ratio r; oversubscription normally means r >= 1. required

Returns

Name Type Description
tuple of float Relative throughput and throughput loss, both dimensionless; r < 1 produces a gain and negative loss.

Notes

Equation: relative = 1/((1 - f) + f r); loss = 1 - relative.

Assumptions: Only communication time changes; compute and communication are serial.

Excludes: Overlap changes, congestion nonlinearities, and altered workload size.

References

Derivation: Normalize baseline step time to one, stretch its communication share by r, then invert the time ratio.

Examples

>>> relative, loss = calc_oversubscription_effect(0.5, 2)
>>> relative + loss == 1
True

calc_point_to_point_time

physics.communication.calc_point_to_point_time(
    message_bytes,
    alpha_s,
    bandwidth_bytes_s,
)

Estimate startup plus serialization time for one transfer.

Parameters

Name Type Description Default
message_bytes Quantity or float Nonnegative message size M; bare values are bytes. required
alpha_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required
bandwidth_bytes_s Quantity or float Positive unidirectional bandwidth B; bare values are bytes/second. required

Returns

Name Type Description
Quantity Transfer time in seconds, including startup even for zero bytes.

Notes

Equation: T = alpha + M/B.

Assumptions: One transfer on an otherwise idle link at the supplied effective bandwidth.

Excludes: Retransmission, queueing, and receiver processing.

References

Derivation: Serialization takes bytes divided by bytes per second; add the fixed startup delay.

Examples

>>> calc_point_to_point_time(0, Q_("2 us"), Q_("1 GB/s")) == Q_("2 us")
True

calc_ring_allreduce_data_factor

physics.communication.calc_ring_allreduce_data_factor(n_gpus)

Return per-rank ring AllReduce traffic in message equivalents.

Parameters

Name Type Description Default
n_gpus int Participant count N, at least one; intended to be integral. required

Returns

Name Type Description
float Dimensionless total traffic multiplier; zero for N = 1.

Notes

Equation: f = 2(N - 1)/N.

Assumptions: Equal chunks with two complete ring phases.

Excludes: Startup latency and local reduction work.

References

Derivation: Each phase moves N - 1 chunks of size M/N; divide two-phase traffic by M.

Examples

>>> calc_ring_allreduce_data_factor(4) == 2 * calc_ring_collective_data_factor(4)
True

calc_ring_allreduce_latency_steps

physics.communication.calc_ring_allreduce_latency_steps(n_gpus)

Count startup terms in ring AllReduce.

Parameters

Name Type Description Default
n_gpus int Participant count N, at least one; intended to be integral. required

Returns

Name Type Description
int Startup count for integral participant input.

Notes

Equation: s = 2(N - 1).

Assumptions: Two sequential ring phases with N - 1 rounds each.

Excludes: Pipelining across rounds and per-byte transmission time.

References

Derivation: Add the ReduceScatter and AllGather round counts.

Examples

>>> calc_ring_allreduce_latency_steps(4) == 2 * (4 - 1)
True

calc_ring_allreduce_latency_time

physics.communication.calc_ring_allreduce_latency_time(n_gpus, latency_s)

Compute the startup-only component of ring AllReduce.

Parameters

Name Type Description Default
n_gpus int Participant count N, at least one; intended to be integral. required
latency_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required

Returns

Name Type Description
Quantity Aggregate startup time in seconds.

Notes

Equation: T_startup = 2(N - 1) alpha.

Assumptions: Every ring round incurs the same startup cost.

Excludes: Payload serialization and reduction compute.

References

Derivation: Multiply the two-phase round count by the per-round startup latency.

Examples

>>> t = calc_ring_allreduce_latency_time(4, Q_("2 us"))
>>> t == calc_ring_allreduce_latency_steps(4) * Q_("2 us")
True

calc_ring_allreduce_time

physics.communication.calc_ring_allreduce_time(
    message_bytes,
    n_gpus,
    bandwidth_bytes_s,
    latency_s,
)

Estimate ring AllReduce communication time.

Parameters

Name Type Description Default
message_bytes Quantity or float Nonnegative message size M; bare values are bytes. required
n_gpus int Participant count N, at least one; intended to be integral. required
bandwidth_bytes_s Quantity or float Positive unidirectional bandwidth B; bare values are bytes/second. required
latency_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required

Returns

Name Type Description
Quantity Collective time in seconds; zero for one participant.

Notes

Equation: T = 2(N - 1) M / (N B) + 2(N - 1) alpha.

Assumptions: Equal chunks and identical links; ReduceScatter followed by AllGather.

Excludes: Local reduction compute, congestion, overlap with computation, and protocol costs.

Bandwidth is one direction, not the sum of both directions of a full-duplex link.

References

Derivation: Each of the two phases sends M/N bytes in each of N - 1 rounds; sum serialization and startup costs.

Examples

>>> calc_ring_allreduce_time(1024, 1, 1024, 0).m_as("s") == 0
True

calc_ring_attention_min_chunk

physics.communication.calc_ring_attention_min_chunk(
    element_bytes,
    sustained_flops,
    link_bandwidth,
)

Estimate the token chunk needed to hide a ring-attention transfer.

Parameters

Name Type Description Default
element_bytes float Positive byte count per key/value element s. required
sustained_flops Quantity or float Positive sustained attention compute rate R; bare values are FLOP/second. required
link_bandwidth Quantity or float Positive unidirectional link bandwidth B; bare values are bytes/second. required

Returns

Name Type Description
float Continuous chunk-size estimate in tokens; not rounded to an integer.

Notes

Equation: n_min = s R / (2 B).

Assumptions: Tile compute is 4 n^2 d FLOPs and key/value transfer is 2 n d s bytes, with perfect overlap; d cancels.

Excludes: Kernel launch costs, softmax overhead, link startup, and discrete tile constraints.

References

Derivation: Set tile compute time 4 n^2 d/R greater than or equal to tile transfer time 2 n d s/B and cancel n d.

Liu et al. (2023), “Ring Attention with Blockwise Transformers for Near-Infinite Context.” https://arxiv.org/abs/2310.01889 (bibliography key: liu2023ringAttention). The threshold above is the stated ideal-overlap tile accounting, not an empirical latency fit.

Examples

>>> base = calc_ring_attention_min_chunk(2, Q_("1 TFLOP/s"), Q_("1 GB/s"))
>>> calc_ring_attention_min_chunk(2, Q_("2 TFLOP/s"), Q_("1 GB/s")) == 2 * base
True

calc_ring_collective_data_factor

physics.communication.calc_ring_collective_data_factor(n_gpus)

Return a single ring-phase traffic multiplier.

Parameters

Name Type Description Default
n_gpus int Participant count N, at least one; intended to be integral. required

Returns

Name Type Description
float Dimensionless fraction of a per-rank message transferred.

Notes

Equation: f = (N - 1)/N.

Assumptions: One ring AllGather, ReduceScatter, or equal-partition exchange phase.

Excludes: Startup latency and repeated phases.

References

Derivation: The rank transfers N - 1 of the N equal message chunks.

Examples

>>> calc_ring_collective_data_factor(1) == 0
True

calc_ring_tree_crossover_size

physics.communication.calc_ring_tree_crossover_size(
    n_gpus,
    alpha_s,
    bandwidth_bytes_s,
)

Estimate a large-rank ring/tree crossover payload.

Parameters

Name Type Description Default
n_gpus int Participant count N, at least one; intended to be integral. required
alpha_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required
bandwidth_bytes_s Quantity or float Positive unidirectional bandwidth B; bare values are bytes/second. required

Returns

Name Type Description
Quantity Heuristic crossover size in bytes; zero for one participant.

Notes

Equation: M_est = N alpha B / log2(N).

Assumptions: Large N; this is an approximation to the crossover of the ring and full-message tree models.

Excludes: Exact crossover solving and small-N guarantees; N = 2 returns a finite estimate although those two cost models have no positive crossover.

References

Derivation: For the exact intersection, the coefficient is ((N - 1) - ceil(log2 N))/(ceil(log2 N) - (N - 1)/N). Replacing its leading terms by N/log2(N) yields this estimate.

Examples

>>> calc_ring_tree_crossover_size(1, Q_("1 us"), Q_("1 GB/s")).m_as("byte") == 0
True

calc_torus_diameter

physics.communication.calc_torus_diameter(side, dims=3)

Count hops in the longest shortest path of a regular torus.

Parameters

Name Type Description Default
side int Nodes per dimension, at least two for a bisection and at least one for diameter. required
dims int Positive dimension count; default three. 3

Returns

Name Type Description
int Maximum shortest-path hop count.

Notes

Equation: diameter = dims floor(side/2).

Assumptions: Bidirectional wraparound links with identical lengths in each dimension.

Excludes: Routing restrictions, failed links, and hop timing.

References

Derivation: The farthest coordinate is floor(side/2) hops away in each dimension; add the independent coordinate distances.

Examples

>>> calc_torus_diameter(4, 3) == 3 * (4 // 2)
True

calc_tree_allreduce_time

physics.communication.calc_tree_allreduce_time(
    message_bytes,
    n_gpus,
    bandwidth_bytes_s,
    latency_s,
)

Estimate reduction and broadcast over a binary tree.

Parameters

Name Type Description Default
message_bytes Quantity or float Nonnegative message size M; bare values are bytes. required
n_gpus int Participant count N, at least one; intended to be integral. required
bandwidth_bytes_s Quantity or float Positive unidirectional bandwidth B; bare values are bytes/second. required
latency_s Quantity or float Nonnegative startup latency alpha; bare values are seconds. required

Returns

Name Type Description
Quantity Two-phase tree communication time in seconds; zero for one participant.

Notes

Equation: T = 2 ceil(log2 N) (M/B + alpha).

Assumptions: Each level serializes the full message; balanced tree with enough levels for N ranks.

Excludes: Pipelining, local reduction compute, contention, and bandwidth-optimal double trees.

References

Derivation: A reduction and a broadcast each traverse ceil(log2 N) levels, paying startup and full-message transfer at every level.

Examples

>>> calc_tree_allreduce_time(1024, 1, 1024, 0).m_as("s") == 0
True

ring_allreduce_data_factor_latex

physics.communication.ring_allreduce_data_factor_latex(n_gpus_symbol='N')

Format the ring AllReduce data factor as LaTeX.

Parameters

Name Type Description Default
n_gpus_symbol str Symbol substituted for participant count; default N. 'N'

Returns

Name Type Description
str LaTeX expression for the dimensionless traffic multiplier.

Notes

Equation: The displayed expression is 2(N - 1)/N.

Assumptions: The supplied symbol is already suitable for LaTeX.

Excludes: Numeric evaluation and LaTeX escaping of arbitrary input.

References

Derivation: ReduceScatter and AllGather each move (N - 1)/N message equivalents per rank.

Examples

>>> "P" in ring_allreduce_data_factor_latex("P")
True
Back to top