data_juicer.analysis¶

class data_juicer.analysis.ColumnWiseAnalysis(dataset, output_path, overall_result=None, save_stats_in_one_file=True)[源代码]¶

基类：object

Apply analysis on each column of stats respectively.

__init__(dataset, output_path, overall_result=None, save_stats_in_one_file=True)[源代码]¶

Initialization method

参数:

dataset -- the dataset to be analyzed
output_path -- path to store the analysis results
overall_result -- optional precomputed overall stats result
save_stats_in_one_file -- whether save all analysis figures of all stats into one image file

analyze(show_percentiles=False, show=False, skip_export=False)[源代码]¶

Apply analysis and draw the analysis figure for stats.

参数:

show_percentiles -- whether to show the percentile line in each sub-figure. If it's true, there will be several red lines to indicate the quantiles of the stats distributions
show -- whether to show in a single window after drawing
skip_export -- whether save the results into disk

返回:

draw_box(ax, data, save_path, percentiles=None, show=False)[源代码]¶

Draw the box plot for the data.

参数:

ax -- the axes to draw
data -- data to draw
save_path -- the path to save the box figure
percentiles -- the overall analysis result of the data including percentile information
show -- whether to show in a single window after drawing

返回:

draw_hist(ax, data, save_path, percentiles=None, show=False)[源代码]¶

Draw the histogram for the data.

参数:

ax -- the axes to draw
data -- data to draw
save_path -- the path to save the histogram figure
percentiles -- the overall analysis result of the data including percentile information
show -- whether to show in a single window after drawing

返回:

draw_wordcloud(ax, data, save_path, show=False)[源代码]¶

class data_juicer.analysis.CorrelationAnalysis(dataset, output_path)[源代码]¶

基类：object

Analyze the correlations among different stats. Only for numerical stats.

__init__(dataset, output_path)[源代码]¶

Initialization method.

参数:

dataset -- the dataset to be analyzed
output_path -- path to store the analysis results

analyze(method='pearson', show=False, skip_export=False)[源代码]¶

class data_juicer.analysis.DiversityAnalysis(dataset, output_path, lang_or_model='en')[源代码]¶

基类：object

Apply diversity analysis for each sample and get an overall analysis result.

__init__(dataset, output_path, lang_or_model='en')[源代码]¶: Initialization method :param dataset: the dataset to be analyzed :param output_path: path to store the analysis results :param lang_or_model: the diversity model or a specific language used to load the diversity model.

analyze(lang_or_model=None, column_name='text', postproc_func=<function get_diversity>, **postproc_kwarg)[源代码]¶

Apply diversity analysis on the whole dataset.

参数:

lang_or_model -- the diversity model or a specific language used to load the diversity model
column_name -- the name of column to be analyzed
postproc_func -- function to analyze diversity. In default, it's function get_diversity
postproc_kwarg -- arguments of the postproc_func

返回:

compute(lang_or_model=None, column_name='text')[源代码]¶

Apply lexical tree analysis on each sample.

参数:

lang_or_model -- the diversity model or a specific language used to load the diversity model
column_name -- the name of column to be analyzed

返回:

the analysis result.

class data_juicer.analysis.OverallAnalysis(dataset, output_path)[源代码]¶

基类：object

Apply analysis on the overall stats, including mean, std, quantiles, etc.

__init__(dataset, output_path)[源代码]¶

Initialization method.

参数:

dataset -- the dataset to be analyzed
output_path -- path to store the analysis results.

analyze(percentiles=[], num_proc=1, skip_export=False)[源代码]¶

Apply overall analysis on the whole dataset based on the describe method of pandas.