data_juicer.utils.logger_utils module

data_juicer.utils.logger_utils.get_caller_name(depth=0)

Get caller name by depth.

Parameters:

depth – depth of caller context, use 0 for caller depth.

Returns:

module name of the caller

class data_juicer.utils.logger_utils.StreamToLoguru(level='INFO', caller_names=('datasets', 'logging'))

Bases: object

Stream object that redirects writes to a logger instance.

__init__(level='INFO', caller_names=('datasets', 'logging'))

Initialization method.

Parameters:

• level – log level string of loguru. Default value: “INFO”.

• caller_names – caller names of redirected module. Default value: (apex, pycocotools).

fileno()

write(buf)

getvalue()

flush()

isatty()

data_juicer.utils.logger_utils.redirect_sys_output(log_level='INFO')

Redirect stdout/stderr to loguru with log level.

Parameters:

log_level – log level string of loguru. Default value: “INFO”.

data_juicer.utils.logger_utils.get_log_file_path()

Get the path to the location of the log file.

Returns:

a location of log file.

data_juicer.utils.logger_utils.setup_logger(save_dir, distributed_rank=0, filename='log.txt', mode='o', level='INFO', redirect=True)

Setup logger for training and testing.

Parameters:

• save_dir – location to save log file

• distributed_rank – device rank when multi-gpu environment

• filename – log file name to save

• mode – log file write mode, append or override. default is o.

• level – log severity level. It’s “INFO” in default.

• redirect – whether to redirect system output

Returns:

logger instance.

data_juicer.utils.logger_utils.make_log_summarization(max_show_item=10)

class data_juicer.utils.logger_utils.HiddenPrints

Bases: object

Define a range that hide the outputs within this range.