Skip to main content

TokenizedBatchFeedScenario

Training scenarios · Implementation: mlsysim/engine/scenario_definitions/training.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

A host that tokenizes each batch and copies it to the accelerator, against the device’s step.

Without prefetching the host and the device run back to back, so the device is busy compute_time out of every prep_time + compute_time.

Notes

Equation: tokenization_time is the worked quantity; each property below states its equation and units.

Assumptions: Preparation and compute are serial; multiple workers scale tokenization ideally in the separate parallel estimate.

Excludes: Overlap, tokenizer imbalance, and parallel workers in busy_fraction, which uses single-core prep_time.

References

Derivation: The equations below follow the stated counts, geometric relations, or accounting definitions; scenario inputs are supplied assumptions or observations, not fitted predictions.

Examples

Execute the registered worked inputs; this is a modeled calculation.

>>> from mlsysim import Scenarios
>>> case = Scenarios.Gpt2XlSingleCoreTokenizerFeed
>>> value = case.tokenization_time
>>> print(f"{value:.3f~P}")
0.655 s

Attributes

Name Description
batch_bytes Token-ID storage in bytes for one batch transfer.
busy_fraction Dimensionless accelerator busy share with serial single-core preparation and compute.
bytes_per_token
compute_time
link_bandwidth
parallel_tokenization_time Ideal parallel tokenization duration in seconds, dividing by worker count.
prep_time Sequential single-core preparation duration in seconds, tokenization plus transfer.
token_rate
tokenization_time Single-core tokenization duration in seconds for one batch.
tokens_per_batch
transfer_time Host-device transfer duration in seconds at the supplied bandwidth.
workers

compute_time

Type: Quantity.

Declared default: required.

tokens_per_batch

Type: int.

Declared default: required.

token_rate

Type: Quantity.

Declared default: required.

bytes_per_token

Type: Quantity.

Declared default: required.

workers

Type: int.

Declared default: 1.

tokenization_time

engine.scenario_types.TokenizedBatchFeedScenario.tokenization_time

Single-core tokenization duration in seconds for one batch.

Equation: return (self.tokens_per_batch / self.token_rate).to(‘s’)

parallel_tokenization_time

engine.scenario_types.TokenizedBatchFeedScenario.parallel_tokenization_time

Ideal parallel tokenization duration in seconds, dividing by worker count.

Equation: return (self.tokenization_time / self.workers).to(‘s’)

batch_bytes

engine.scenario_types.TokenizedBatchFeedScenario.batch_bytes

Token-ID storage in bytes for one batch transfer.

Equation: return (self.tokens_per_batch * self.bytes_per_token).to(‘byte’)

transfer_time

engine.scenario_types.TokenizedBatchFeedScenario.transfer_time

Host-device transfer duration in seconds at the supplied bandwidth.

Equation: return (self.batch_bytes / self.link_bandwidth).to(‘s’)

prep_time

engine.scenario_types.TokenizedBatchFeedScenario.prep_time

Sequential single-core preparation duration in seconds, tokenization plus transfer.

Single-core preparation: tokenization plus the copy.

Equation: return (self.tokenization_time + self.transfer_time).to(‘s’)

busy_fraction

engine.scenario_types.TokenizedBatchFeedScenario.busy_fraction

Dimensionless accelerator busy share with serial single-core preparation and compute.

Share of time the device computes when preparation and compute run in sequence.

Equation: return (self.compute_time / (self.prep_time + self.compute_time)).to(’’).magnitude

Back to top