imgdiff_difference_caption_generator_mapper¶
Generates difference captions for bounding box regions in two images.
This operator processes pairs of images and generates captions for the differences in their bounding box regions. It uses a multi-step process:
Describes the content of each bounding box region using a Hugging Face model.
Crops the bounding box regions from both images.
Checks if the cropped regions match the generated captions.
Determines if there are differences between the two captions.
Marks the difference area with a red box.
Generates difference captions for the marked areas.
The key metric is the similarity score between the captions, computed using a CLIP model.
If no valid bounding boxes or differences are found, it returns empty captions and zeroed bounding boxes.
Uses ‘cuda’ as the accelerator if any of the fused operations support it.
Caches temporary images during processing and clears them afterward.
为两幅图像的边界框区域生成差异描述。
此算子处理成对的图像并为其边界框区域的差异生成描述。它使用多步骤过程：
使用 Hugging Face 模型描述每个边界框区域的内容。
从两幅图像中裁剪出边界框区域。
检查裁剪区域是否与生成的描述匹配。
确定两个描述之间是否存在差异。
用红色框标记差异区域。
为标记区域生成差异描述。
关键指标是使用 CLIP 模型计算的描述之间的相似度得分。
如果没有找到有效的边界框或差异，则返回空描述和零化的边界框。
如果任何融合操作支持，则使用 ‘cuda’ 作为加速器。
在处理过程中缓存临时图像并在之后清除它们。
Type 算子类型: mapper
Tags 标签: cpu
🔧 Parameter Configuration 参数配置¶
name 参数名 |
type 类型 |
default 默认值 |
desc 说明 |
|---|---|---|---|
|
typing.Optional[typing.Dict] |
|
Arguments for multimodal language model mapper. |
|
typing.Optional[typing.Dict] |
|
Arguments for image-text matching filter. |
|
typing.Optional[typing.Dict] |
|
Arguments for text pair similarity filter. |
|
|
||
|
|
📊 Effect demonstration 效果演示¶
not available 暂无