data_juicer.utils.process_utils module¶

data_juicer.utils.process_utils.setup_mp(method=None)[源代码]¶

data_juicer.utils.process_utils.get_min_cuda_memory()[源代码]¶

data_juicer.utils.process_utils.calculate_np(name, mem_required, cpu_required, num_proc=None, use_cuda=False)[源代码]¶: Calculate the optimum number of processes for the given OP