Welcome to data-juicer’s documentation!¶
Tutorial¶
We will give a tutorial on KDD’24, Multi-modal Data Processing for Foundation Models: Practical Guidances and Use Cases, see more details here!
API Reference
- data_juicer.core
- data_juicer.ops
- data_juicer.ops.filter
AlphanumericFilterAudioDurationFilterAudioNMFSNRFilterAudioSizeFilterAverageLineLengthFilterCharacterRepetitionFilterFlaggedWordFilterImageAestheticsFilterImageAspectRatioFilterImageFaceCountFilterImageFaceRatioFilterImageNSFWFilterImagePairSimilarityFilterImageShapeFilterImageSizeFilterImageTextMatchingFilterImageTextSimilarityFilterImageWatermarkFilterLanguageIDScoreFilterLLMQualityScoreFilterLLMDifficultyScoreFilterMaximumLineLengthFilterPerplexityFilterPhraseGroundingRecallFilterSpecialCharactersFilterSpecifiedFieldFilterSpecifiedNumericFieldFilterStopWordsFilterSuffixFilterTextActionFilterTextEntityDependencyFilterTextLengthFilterTextPairSimilarityFilterTokenNumFilterVideoAestheticsFilterVideoAspectRatioFilterVideoDurationFilterVideoFramesTextSimilarityFilterVideoMotionScoreFilterVideoMotionScoreRaftFilterVideoNSFWFilterVideoOcrAreaRatioFilterVideoResolutionFilterVideoTaggingFromFramesFilterVideoWatermarkFilterWordRepetitionFilterWordsNumFilter
- data_juicer.ops.mapper
AudioFFmpegWrappedMapperCalibrateQAMapperCalibrateQueryMapperCalibrateResponseMapperChineseConvertMapperCleanCopyrightMapperCleanEmailMapperCleanHtmlMapperCleanIpMapperCleanLinksMapperDialogIntentDetectionMapperDialogSentimentDetectionMapperDialogSentimentIntensityMapperDialogTopicDetectionMapperExpandMacroMapperExtractEntityAttributeMapperExtractEntityRelationMapperExtractEventMapperExtractKeywordMapperExtractNicknameMapperExtractSupportTextMapperFixUnicodeMapperGenerateQAFromExamplesMapperGenerateQAFromTextMapperImageBlurMapperImageCaptioningFromGPT4VMapperImageCaptioningMapperImageDiffusionMapperImageFaceBlurMapperImageRemoveBackgroundMapperImageSegmentMapperImageTaggingMapperMllmMapperNlpaugEnMapperNlpcdaZhMapperOptimizeQAMapperOptimizeQueryMapperOptimizeResponseMapperPairPreferenceMapperPunctuationNormalizationMapperPythonFileMapperPythonLambdaMapperQuerySentimentDetectionMapperQueryIntentDetectionMapperQueryTopicDetectionMapperRelationIdentityMapperRemoveBibliographyMapperRemoveCommentsMapperRemoveHeaderMapperRemoveLongWordsMapperRemoveNonChineseCharacterlMapperRemoveRepeatSentencesMapperRemoveSpecificCharsMapperRemoveTableTextMapperRemoveWordsWithIncorrectSubstringsMapperReplaceContentMapperSDXLPrompt2PromptMapperSentenceAugmentationMapperSentenceSplitMapperTextChunkMapperVideoCaptioningFromAudioMapperVideoCaptioningFromFramesMapperVideoCaptioningFromSummarizerMapperVideoCaptioningFromVideoMapperVideoExtractFramesMapperVideoFFmpegWrappedMapperVideoFaceBlurMapperVideoRemoveWatermarkMapperVideoResizeAspectRatioMapperVideoResizeResolutionMapperVideoSplitByDurationMapperVideoSplitByKeyFrameMapperVideoSplitBySceneMapperVideoTaggingFromAudioMapperVideoTaggingFromFramesMapperWhitespaceNormalizationMapper
- data_juicer.ops.deduplicator
- data_juicer.ops.selector
- data_juicer.ops.common
- data_juicer.analysis
- data_juicer.config
- data_juicer.format