Welcome to data-juicer’s documentation!

Tutorial

We will give a tutorial on KDD’24, Multi-modal Data Processing for Foundation Models: Practical Guidances and Use Cases, see more details here!

API Reference

• data_juicer.core
  • Analyser
  • NestedDataset
  • Executor
  • Exporter
  • Tracer
• data_juicer.ops
  • load_ops()
  • Filter
  • Mapper
  • Deduplicator
  • Selector
• data_juicer.ops.filter
  • ImageTextSimilarityFilter
  • VideoAspectRatioFilter
  • ImageTextMatchingFilter
  • ImageNSFWFilter
  • TokenNumFilter
  • TextLengthFilter
  • SpecifiedNumericFieldFilter
  • AudioNMFSNRFilter
  • VideoAestheticsFilter
  • PerplexityFilter
  • PhraseGroundingRecallFilter
  • MaximumLineLengthFilter
  • AverageLineLengthFilter
  • SpecifiedFieldFilter
  • VideoTaggingFromFramesFilter
  • TextEntityDependencyFilter
  • VideoResolutionFilter
  • AlphanumericFilter
  • ImageWatermarkFilter
  • ImageAestheticsFilter
  • AudioSizeFilter
  • StopWordsFilter
  • CharacterRepetitionFilter
  • ImageShapeFilter
  • VideoDurationFilter
  • TextActionFilter
  • VideoOcrAreaRatioFilter
  • VideoNSFWFilter
  • SpecialCharactersFilter
  • VideoFramesTextSimilarityFilter
  • ImageAspectRatioFilter
  • AudioDurationFilter
  • LanguageIDScoreFilter
  • SuffixFilter
  • ImageSizeFilter
  • VideoWatermarkFilter
  • WordNumFilter
  • ImageFaceRatioFilter
  • FlaggedWordFilter
  • WordRepetitionFilter
  • VideoMotionScoreFilter
• data_juicer.ops.mapper
  • VideoCaptioningFromAudioMapper
  • VideoTaggingFromAudioMapper
  • ImageCaptioningFromGPT4VMapper
  • PunctuationNormalizationMapper
  • RemoveBibliographyMapper
  • SentenceSplitMapper
  • VideoSplitBySceneMapper
  • CleanIpMapper
  • CleanLinksMapper
  • RemoveHeaderMapper
  • RemoveTableTextMapper
  • VideoRemoveWatermarkMapper
  • RemoveRepeatSentencesMapper
  • ImageDiffusionMapper
  • ImageFaceBlurMapper
  • VideoFFmpegWrappedMapper
  • ChineseConvertMapper
  • NlpcdaZhMapper
  • ImageBlurMapper
  • CleanCopyrightMapper
  • RemoveNonChineseCharacterlMapper
  • VideoSplitByKeyFrameMapper
  • RemoveSpecificCharsMapper
  • VideoResizeAspectRatioMapper
  • CleanHtmlMapper
  • WhitespaceNormalizationMapper
  • VideoTaggingFromFramesMapper
  • RemoveCommentsMapper
  • ExpandMacroMapper
  • ImageCaptioningMapper
  • RemoveWordsWithIncorrectSubstringsMapper
  • VideoCaptioningFromVideoMapper
  • VideoCaptioningFromSummarizerMapper
  • FixUnicodeMapper
  • NlpaugEnMapper
  • VideoCaptioningFromFramesMapper
  • RemoveLongWordsMapper
  • VideoResizeResolutionMapper
  • CleanEmailMapper
  • ReplaceContentMapper
  • AudioFFmpegWrappedMapper
  • VideoSplitByDurationMapper
  • VideoFaceBlurMapper
• data_juicer.ops.deduplicator
  • VideoDeduplicator
  • RayBasicDeduplicator
  • DocumentMinhashDeduplicator
  • RayImageDeduplicator
  • RayDocumentDeduplicator
  • DocumentDeduplicator
  • ImageDeduplicator
  • DocumentSimhashDeduplicator
  • RayVideoDeduplicator
• data_juicer.ops.selector
  • FrequencySpecifiedFieldSelector
  • TopkSpecifiedFieldSelector
• data_juicer.ops.common
  • get_sentences_from_document()
  • get_words_from_document()
  • merge_on_whitespace_tab_newline()
  • split_on_newline_tab_whitespace()
  • split_on_whitespace()
  • strip()
  • words_augmentation()
  • words_refinement()
• data_juicer.analysis
  • ColumnWiseAnalysis
  • DiversityAnalysis
  • OverallAnalysis
• data_juicer.config
  • init_configs()
  • export_config()
  • merge_config()
• data_juicer.format
  • load_formatter()
  • JsonFormatter
  • LocalFormatter
  • RemoteFormatter
  • TextFormatter
  • ParquetFormatter
  • CsvFormatter
  • TsvFormatter
  • MixtureFormatter

Indices and Tables

• Index

• Module Index

• Search Page