sciunit.scores package¶
Submodules¶
sciunit.scores.base module¶
Base class for SciUnit scores.
- class sciunit.scores.base.ErrorScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A score returned when an error occurs during testing.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.base'¶
- property norm_score¶
A floating point version of the score used for sorting. If normalized = True, this must be in the range 0.0 to 1.0, where larger is better (used for sorting and coloring tables).
- property summary¶
Summarize the performance of a model on a test.
- class sciunit.scores.base.Score(score, related_data=None)[source]¶
Bases:
sciunit.base.SciUnit
Abstract base class for scores.
- __annotations__ = {}¶
- __hash__ = None¶
- __init__(score, related_data=None)[source]¶
Abstract base class for scores.
- Args:
score (int, float, bool): A raw value to wrap in a Score class. related_data (dict, optional): Artifacts to store with the score.
- __module__ = 'sciunit.scores.base'¶
- _allowed_types = None¶
List of allowed types for the score argument
- _allowed_types_message = 'Score of type %s is not an instance of one of the allowed types: %s'¶
Error message when score argument is not one of these types
- _best = None¶
The best possible score of this type
- _check_score(score)[source]¶
A method for each Score subclass to impose additional constraints on the score, e.g. the range of the allowed score
- _description = ''¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- _raw = None¶
A raw number arising in a test’s compute_score, used to determine this score. Can be set for reporting a raw value determined in Test.compute_score before any transformation, e.g. by a Converter
- _worst = None¶
The best possible score of this type
- classmethod compute(observation, prediction)[source]¶
Compute whether the observation equals the prediction.
- description = ''¶
A description of this score, i.e. how to interpret it. For the user to set in bind_score
- classmethod extract_mean_or_value(obs_or_pred, key=None)[source]¶
Extracts the mean, value, or user-provided key from an observation or prediction dictionary.
- classmethod extract_means_or_values(observation, prediction, key=None)[source]¶
Extracts the mean, value, or user-provided key from the observation and prediction dictionaries.
- property log10_norm_score¶
The logarithm base 10 of the norm_score. This is useful for guaranteeing convexity in an error surface
- property log2_norm_score¶
The logarithm base 2 of the norm_score. This is useful for guaranteeing convexity in an error surface
- property log_norm_score¶
The natural logarithm of the norm_score. This is useful for guaranteeing convexity in an error surface
- model = None¶
The model judged. Set automatically by Test.judge.
- property norm_score¶
A floating point version of the score used for sorting. If normalized = True, this must be in the range 0.0 to 1.0, where larger is better (used for sorting and coloring tables).
- property raw¶
Data specific to the result of a test run on a model.
- score = None¶
The score itself.
- property score_type¶
- property summary¶
Summarize the performance of a model on a test.
- test = None¶
The test taken. Set automatically by Test.judge.
sciunit.scores.collections module¶
SciUnit score collections, such as arrays and matrices.
These collections allow scores to be organized and visualized by model, test, or both.
- class sciunit.scores.collections.ScoreArray(tests_or_models, scores=None, weights=None)[source]¶
Bases:
pandas.core.series.Series
,sciunit.base.SciUnit
,sciunit.base.TestWeighted
Represents an array of scores derived from a test suite.
Extends the pandas Series such that items are either models subject to a test or tests taken by a model. Also displays and compute score summaries in sciunit-specific ways.
Can use like this, assuming n tests and m models:
>>> sm[test]
>>> sm[test] (score_1, ..., score_m) >>> sm[model] (score_1, ..., score_n)
- __annotations__ = {}¶
- __getattr__(name)[source]¶
After regular attribute access, try looking up the name This allows simpler access to columns for interactive use.
- __init__(tests_or_models, scores=None, weights=None)[source]¶
Initialize self. See help(type(self)) for accurate signature.
- __module__ = 'sciunit.scores.collections'¶
- direct_attrs = ['score', 'norm_scores', 'related_data']¶
- mean()[source]¶
Compute a total score for each model over all the tests.
Uses the norm_score attribute, since otherwise direct comparison across different kinds of scores would not be possible.
- property norm_scores¶
Return the norm_score for each test.
- class sciunit.scores.collections.ScoreMatrix(tests, models, scores=None, weights=None, transpose=False)[source]¶
Bases:
pandas.core.frame.DataFrame
,sciunit.base.SciUnit
,sciunit.base.TestWeighted
Represents a matrix of scores derived from a test suite. Extends the pandas DataFrame such that tests are columns and models are the index. Also displays and compute score summaries in sciunit-specific ways.
Can use like this, assuming n tests and m models:
>>> sm[test]
>>> sm[test] (score_1, ..., score_m) >>> sm[model] (score_1, ..., score_n)
- property T¶
Get transpose of this ScoreMatrix.
- __annotations__ = {}¶
- __getattr__(name)[source]¶
After regular attribute access, try looking up the name This allows simpler access to columns for interactive use.
- __init__(tests, models, scores=None, weights=None, transpose=False)[source]¶
Initialize self. See help(type(self)) for accurate signature.
- __module__ = 'sciunit.scores.collections'¶
- direct_attrs = ['score', 'norm_scores', 'related_data']¶
- property norm_scores¶
- show_mean = False¶
- sortable = False¶
sciunit.scores.collections_m2m module¶
Score collections for direct comparison of models against other models.
- class sciunit.scores.collections_m2m.ScoreArrayM2M(test, models, scores)[source]¶
Bases:
pandas.core.series.Series
Represents an array of scores derived from TestM2M. Extends the pandas Series such that items are either models subject to a test or the test itself.
- __annotations__ = {}¶
- __getattr__(name)[source]¶
After regular attribute access, try looking up the name This allows simpler access to columns for interactive use.
- __init__(test, models, scores)[source]¶
Initialize self. See help(type(self)) for accurate signature.
- __module__ = 'sciunit.scores.collections_m2m'¶
- property norm_scores¶
- class sciunit.scores.collections_m2m.ScoreMatrixM2M(test, models, scores)[source]¶
Bases:
pandas.core.frame.DataFrame
Represents a matrix of scores derived from TestM2M. Extends the pandas DataFrame such that models/observation are both columns and the index.
- __annotations__ = {}¶
- __getattr__(name)[source]¶
After regular attribute access, try looking up the name This allows simpler access to columns for interactive use.
- __init__(test, models, scores)[source]¶
Initialize self. See help(type(self)) for accurate signature.
- __module__ = 'sciunit.scores.collections_m2m'¶
- property norm_scores¶
sciunit.scores.complete module¶
Score types for tests that completed successfully.
These include various representations of goodness-of-fit.
- class sciunit.scores.complete.BooleanScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A boolean score, which must be True or False.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.complete'¶
- _allowed_types = (<class 'bool'>,)¶
List of allowed types for the score argument
- _description = 'True if the observation and prediction were sufficiently similar; False otherwise'¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- classmethod compute(observation, prediction)[source]¶
Compute whether the observation equals the prediction.
- property norm_score¶
Return 1.0 for a True score and 0.0 for False score.
- class sciunit.scores.complete.CohenDScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.complete.ZScore
A Cohen’s D score.
A float indicating difference between two means normalized by the pooled standard deviation.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.complete'¶
- _description = "The Cohen's D between the prediction and the observation"¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- class sciunit.scores.complete.FloatScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A float score.
A float with any value.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.complete'¶
- _allowed_types = (<class 'float'>, <class 'quantities.quantity.Quantity'>)¶
List of allowed types for the score argument
- _check_score(score)[source]¶
A method for each Score subclass to impose additional constraints on the score, e.g. the range of the allowed score
- _description = 'There is no canonical mapping between this score type and a measure of agreement between the observation and the prediction'¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- class sciunit.scores.complete.PercentScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A percent score.
A float in the range [0,0,100.0] where higher is better.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.complete'¶
- _check_score(score)[source]¶
A method for each Score subclass to impose additional constraints on the score, e.g. the range of the allowed score
- _description = '100.0 is considered perfect agreement between the observation and the prediction. 0.0 is the worst possible agreement'¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- property norm_score¶
Return 1.0 for a percent score of 100, and 0.0 for 0.
- class sciunit.scores.complete.RandomScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A random score in [0,1].
This has no scientific value and should only be used for debugging purposes. For example, one might assign a random score under some error condition to move forward with an application that requires a numeric score, and use the presence of a RandomScore in the output as an indication of an internal error.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.complete'¶
- _allowed_types = (<class 'float'>,)¶
List of allowed types for the score argument
- _description = 'There is a random number in [0,1] and has no relation to the prediction or the observation'¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- class sciunit.scores.complete.RatioScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A ratio of two numbers.
Usually the prediction divided by the observation.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.complete'¶
- _allowed_types = (<class 'float'>,)¶
List of allowed types for the score argument
- _best = 1.0¶
The best possible score of this type
- _check_score(score)[source]¶
A method for each Score subclass to impose additional constraints on the score, e.g. the range of the allowed score
- _description = 'The ratio between the prediction and the observation'¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- classmethod compute(observation, prediction, key=None)[source]¶
Compute a ratio from an observation and a prediction.
- property norm_score¶
Return 1.0 for a ratio of 1, falling to 0.0 for extremely small or large values.
- class sciunit.scores.complete.ZScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A Z score.
A float indicating standardized difference from a reference mean.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.complete'¶
- _allowed_types = (<class 'float'>,)¶
List of allowed types for the score argument
- _best = 0.0¶
The best possible score of this type
- _description = 'The difference between the means of the observation and prediction divided by the standard deviation of the observation'¶
A description of this score, i.e. how to interpret it. Provided in the score definition
- _worst = inf¶
The best possible score of this type
- classmethod compute(observation, prediction)[source]¶
Compute a z-score from an observation and a prediction.
- property norm_score¶
Return the normalized score.
Equals 1.0 for a z-score of 0, falling to 0.0 for extremely positive or negative values.
sciunit.scores.incomplete module¶
Score types for tests that did not complete successfully.
These include details about the various possible reasons that a particular combination of model and test could not be completed.
- class sciunit.scores.incomplete.InsufficientDataScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.incomplete.NoneScore
A score returned when the model or test data is insufficient to score the test.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.incomplete'¶
- description = 'Insufficient Data'¶
A description of this score, i.e. how to interpret it. For the user to set in bind_score
- class sciunit.scores.incomplete.NAScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.incomplete.NoneScore
A N/A (not applicable) score.
Indicates that the model doesn’t have the capabilities that the test requires.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.incomplete'¶
- description = 'N/A'¶
A description of this score, i.e. how to interpret it. For the user to set in bind_score
- class sciunit.scores.incomplete.NoneScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.base.Score
A None score.
Usually indicates that the model has not been checked to see if it has the capabilities required by the test.
- __annotations__ = {}¶
- __init__(score, related_data=None)[source]¶
Abstract base class for scores.
- Args:
score (int, float, bool): A raw value to wrap in a Score class. related_data (dict, optional): Artifacts to store with the score.
- __module__ = 'sciunit.scores.incomplete'¶
- property norm_score¶
A floating point version of the score used for sorting. If normalized = True, this must be in the range 0.0 to 1.0, where larger is better (used for sorting and coloring tables).
- class sciunit.scores.incomplete.TBDScore(score, related_data=None)[source]¶
Bases:
sciunit.scores.incomplete.NoneScore
A TBD (to be determined) score. Indicates that the model has capabilities required by the test but has not yet taken it.
- __annotations__ = {}¶
- __module__ = 'sciunit.scores.incomplete'¶
- description = 'None'¶
A description of this score, i.e. how to interpret it. For the user to set in bind_score
Module contents¶
Contains classes for different representations of test scores.
It also contains score collections such as arrays and matrices.