DuplicateRemovalScenario

Data scenarios · Implementation: mlsysim/engine/scenario_definitions/data.py.

Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.

Near-duplicate shares removed from run’s corpus: the training time and energy each share returns, against one read of the corpus and the MinHash signatures of its documents.

Notes

Equation: read_time is the worked quantity; each property below states its equation and units.

Assumptions: Removed data proportionally reduces training work and signatures use fixed hashes per document.

Excludes: Deduplication quality, selection computation, index overhead, and nonlinear training effects.

References

Derivation: The equations below follow the stated counts, geometric relations, or accounting definitions; scenario inputs are supplied assumptions or observations, not fitted predictions.

Examples

Execute the registered worked inputs; this is a modeled calculation.

>>> from mlsysim import Scenarios
>>> case = Scenarios.Gpt2WebTextNearDuplicates
>>> value = case.read_time
>>> print(f"{value:.3f~P}")
5.714 s

Attributes

Name	Description
all_pairs	Unordered document-pair count for exhaustive comparison.
documents
duplicate_shares
hash_bytes
read_bandwidth
read_time	One full corpus read in seconds at the specified sustained read bandwidth.
run
signature_bytes	Raw MinHash signature storage in GB, documents times hashes times bytes per hash.
signature_hashes

Methods

Name	Description
energy_saved	Kilowatt-hours avoided for the named corpus at the run’s board-power estimate.
time_saved	Hours avoided for the named corpus, removed share times full-run duration.

energy_saved

engine.scenario_types.DuplicateRemovalScenario.energy_saved(corpus)

Kilowatt-hours avoided for the named corpus at the run’s board-power estimate.

Equation: return (self.duplicate_shares[corpus] * self.run.run_energy).to(‘kWh’)

time_saved

engine.scenario_types.DuplicateRemovalScenario.time_saved(corpus)

Hours avoided for the named corpus, removed share times full-run duration.

Equation: return (self.duplicate_shares[corpus] * self.run.run_time).to(‘hour’)