ShardPackagingScenario
Data scenarios · Implementation: mlsysim/engine/scenario_definitions/data.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
files samples packaged into shards large files on a file system rated at rated_bandwidth.
Notes
Equation: Use the dimensional products, ratios, or selection rules documented on the calculated properties below.
Assumptions: Every sample belongs to one shard; files-per-shard is an average, not an integer packing guarantee.
Excludes: Metadata overhead and file-system throughput scaling are not calculated by this ratio.
References
Derivation: the calculated properties below state their defining arithmetic and units. They are scenario accounting or input-selection rules, not independently measured relationships.
Examples
>>> from mlsysim import Scenarios
>>> case = Scenarios.ImageFileMetadataMeltdown
>>> round(case.files_per_shard, 4)
4000.0Attributes
| Name | Description |
|---|---|
| files | |
| files_per_shard | Average small-file count represented by one shard. Equation: files / shards. Units: |
| rated_bandwidth | |
| shards |
files
Type: int.
Declared default: required.
shards
Type: int.
Declared default: required.
rated_bandwidth
Type: Quantity.
Declared default: required.
files_per_shard
engine.scenario_types.ShardPackagingScenario.files_per_shard
Average small-file count represented by one shard. Equation: files / shards. Units: files per shard; uneven shard sizes are summarized by an average.