data_juicer.analysis¶
- class data_juicer.analysis.ColumnWiseAnalysis(dataset, output_path, overall_result=None, save_stats_in_one_file=True)[source]¶
Bases:
object
Apply analysis on each column of stats respectively.
- __init__(dataset, output_path, overall_result=None, save_stats_in_one_file=True)[source]¶
Initialization method
- Parameters:
dataset – the dataset to be analyzed
output_path – path to store the analysis results
overall_result – optional precomputed overall stats result
save_stats_in_one_file – whether save all analysis figures of all stats into one image file
- analyze(show_percentiles=False, show=False, skip_export=False)[source]¶
Apply analysis and draw the analysis figure for stats.
- Parameters:
show_percentiles – whether to show the percentile line in each sub-figure. If it’s true, there will be several red lines to indicate the quantiles of the stats distributions
show – whether to show in a single window after drawing
skip_export – whether save the results into disk
- Returns:
- draw_hist(ax, data, save_path, percentiles=None, show=False)[source]¶
Draw the histogram for the data.
- Parameters:
ax – the axes to draw
data – data to draw
save_path – the path to save the histogram figure
percentiles – the overall analysis result of the data including percentile information
show – whether to show in a single window after drawing
- Returns:
- draw_box(ax, data, save_path, percentiles=None, show=False)[source]¶
Draw the box plot for the data.
- Parameters:
ax – the axes to draw
data – data to draw
save_path – the path to save the box figure
percentiles – the overall analysis result of the data including percentile information
show – whether to show in a single window after drawing
- Returns:
- class data_juicer.analysis.DiversityAnalysis(dataset, output_path, lang_or_model='en')[source]¶
Bases:
object
Apply diversity analysis for each sample and get an overall analysis result.
- __init__(dataset, output_path, lang_or_model='en')[source]¶
Initialization method :param dataset: the dataset to be analyzed :param output_path: path to store the analysis results :param lang_or_model: the diversity model or a specific language used to load the diversity model.
- compute(lang_or_model=None, column_name='text')[source]¶
Apply lexical tree analysis on each sample.
- Parameters:
lang_or_model – the diversity model or a specific language used to load the diversity model
column_name – the name of column to be analyzed
- Returns:
the analysis result.
- analyze(lang_or_model=None, column_name='text', postproc_func=<function get_diversity>, **postproc_kwarg)[source]¶
Apply diversity analysis on the whole dataset.
- Parameters:
lang_or_model – the diversity model or a specific language used to load the diversity model
column_name – the name of column to be analyzed
postproc_func – function to analyze diversity. In default, it’s function get_diversity
postproc_kwarg – arguments of the postproc_func
- Returns:
- class data_juicer.analysis.OverallAnalysis(dataset, output_path)[source]¶
Bases:
object
Apply analysis on the overall stats, including mean, std, quantiles, etc.
- __init__(dataset, output_path)[source]¶
Initialization method.
- Parameters:
dataset – the dataset to be analyzed
output_path – path to store the analysis results.
- analyze(percentiles=[], num_proc=1, skip_export=False)[source]¶
Apply overall analysis on the whole dataset based on the describe method of pandas.
- Parameters:
percentiles – percentiles to analyze
num_proc – number of processes to analyze the dataset
skip_export – whether export the results to disk
- Returns:
the overall analysis result.