Welcome to data-juicer’s documentation!

Tutorial

We will give a tutorial on KDD’24, Multi-modal Data Processing for Foundation Models: Practical Guidances and Use Cases, see more details here!

API Reference

• data_juicer.core
• data_juicer.ops
• data_juicer.ops.filter
• data_juicer.ops.mapper
• data_juicer.ops.deduplicator
• data_juicer.ops.selector
• data_juicer.ops.common
• data_juicer.analysis
• data_juicer.config
  • init_configs()
  • get_init_configs()
  • export_config()
  • merge_config()
  • prepare_side_configs()
• data_juicer.format
  • load_formatter()
  • JsonFormatter
  • LocalFormatter
  • RemoteFormatter
  • TextFormatter
  • ParquetFormatter
  • CsvFormatter
  • TsvFormatter
  • MixtureFormatter
  • EmptyFormatter
  • RayEmptyFormatter

Indices and Tables

• Index

• Module Index

• Search Page