CrossEntropyScenario
Training scenarios · Implementation: mlsysim/engine/scenario_definitions/training.py.
Construct this scenario with keyword inputs. The fields below show the types and defaults declared by this class.
Cross-entropy losses for named predicted probabilities of the correct class, among classes.
misread gives the probabilities of a confident mistake, keyed "true" (the correct class) and "wrong" (the class predicted); the gradient on each score is its probability minus its target.
Notes
Equation: Correct-class loss is -ln(p); softmax-logit gradients are predicted probability minus one-hot target.
Assumptions: Valid nonzero correct-class probabilities and one categorical target per example.
Excludes: Calibration quality, label smoothing, and class weighting.
References
Derivation: Uses mlsysim.physics.neural.calc_cross_entropy for -ln(p) of the correct class. Differentiating softmax cross-entropy with respect to a logit gives p-y, so the true-class target is one and a wrong-class target is zero.
Examples
>>> from mlsysim import Scenarios
>>> s = Scenarios.DigitCrossEntropyExamples
>>> s.uniform_loss > 0
TrueAttributes
| Name | Description |
|---|---|
| classes | |
| misread | |
| misread_gradient_true | Dimensionless true-class softmax-logit gradient, probability minus its one-hot target. |
| misread_gradient_wrong | Dimensionless wrong-class softmax-logit gradient, probability minus zero target. |
| misread_loss | Natural-log loss in nats at the supplied confident-error probability for the true class. |
| probabilities | |
| uniform_loss | Natural-log loss in nats when probability is shared equally among classes. |
Methods
| Name | Description |
|---|---|
| loss | Natural-log cross-entropy in nats for the named correct-class probability. |
loss
engine.scenario_types.CrossEntropyScenario.loss(key)Natural-log cross-entropy in nats for the named correct-class probability.
Equation: calc_cross_entropy(probabilities[key]).
classes
Type: int.
Declared default: required.
probabilities
Type: Dict[str, float].
Declared default: required.
misread
Type: Dict[str, float].
Declared default: required.
uniform_loss
engine.scenario_types.CrossEntropyScenario.uniform_loss
Natural-log loss in nats when probability is shared equally among classes.
Equation: calc_cross_entropy(1 / classes).
misread_loss
engine.scenario_types.CrossEntropyScenario.misread_loss
Natural-log loss in nats at the supplied confident-error probability for the true class.
Equation: calc_cross_entropy(misread['true']).
misread_gradient_true
engine.scenario_types.CrossEntropyScenario.misread_gradient_true
Dimensionless true-class softmax-logit gradient, probability minus its one-hot target.
Equation: misread['true'] - 1.0.
misread_gradient_wrong
engine.scenario_types.CrossEntropyScenario.misread_gradient_wrong
Dimensionless wrong-class softmax-logit gradient, probability minus zero target.
Equation: misread['wrong'] - 0.0.