data_juicer.analysis.measure module

class data_juicer.analysis.measure.Measure

Bases: object

Base class for Measure distribution.

name = 'base'

measure(*args, **kwargs)

class data_juicer.analysis.measure.KLDivMeasure

Bases: Measure

Measure Kullback-Leibler divergence.

name = 'kl_divergence'

measure(p, q)

class data_juicer.analysis.measure.JSDivMeasure

Bases: Measure

Measure Jensen-Shannon divergence.

name = 'js_divergence'

measure(p, q)

class data_juicer.analysis.measure.CrossEntropyMeasure

Bases: Measure

Measure Cross-Entropy.

name = 'cross_entropy'

measure(p, q)

class data_juicer.analysis.measure.EntropyMeasure

Bases: Measure

Measure Entropy.

name = 'entropy'

measure(p)

class data_juicer.analysis.measure.RelatedTTestMeasure

Bases: Measure

Measure T-Test for two related distributions on their histogram of the same bins.

Ref: https://en.wikipedia.org/wiki/Student%27s_t-test

For continuous features or distributions, the input could be dataset stats list. For discrete features or distributions, the input could be the tags or the categories list.

name = 't-test'

static stats_to_hist(p, q)

static category_to_hist(p, q)

measure(p, q)

Parameters:

• p – the first feature or distribution. (stats/tags/categories)

• q – the second feature or distribution. (stats/tags/categories)

Returns:

the T-Test results object – ([ref](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats._result_classes.TtestResult.html#scipy.stats._result_classes.TtestResult)) # noqa: E501