clean_html_mapper

Cleans HTML code from text samples, converting HTML to plain text.

This operator processes text samples by removing HTML tags and converting HTML elements to a more readable format. Specifically, it replaces <li> and <ol> tags with newline and bullet points. The Selectolax HTML parser is used to extract the text content from the HTML. This operation is performed in a batched manner, making it efficient for large datasets.

将HTML代码从文本样本中清理，将HTML转换为纯文本。

此算子通过删除HTML标签并将HTML元素转换为更易读的格式来处理文本样本。具体来说，它将<li>和<ol>标签替换为换行符和项目符号。使用Selectolax HTML解析器从HTML中提取文本内容。此操作以批量方式执行，使其适用于大型数据集。

Type 算子类型: mapper

Tags 标签: cpu, text

🔧 Parameter Configuration 参数配置

name 参数名

type 类型

default 默认值

desc 说明

args

''

extra args

kwargs

''

extra args

📊 Effect demonstration 效果演示

test_complete_html_text

CleanHtmlMapper()

📥 input data 输入数据

Sample 1: text
<header><nav><ul><tile>测试</title><li><a href="#">Home</a></li><li><a href="#">About</a></li><li><a href="#">Services</a></li><li><a href="#">Contact</a></li></ul></nav></header><main><h1>Welcome to My Website</h1><p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.<button>Learn More</button><...
Show more 展开更多 (74 more chars)
<header><nav><ul><tile>测试</title><li><a href="#">Home</a></li><li><a href="#">About</a></li><li><a href="#">Services</a></li><li><a href="#">Contact</a></li></ul></nav></header><main><h1>Welcome to My Website</h1><p>Lorem ipsum dolor sit amet, consectetur adipiscing elit.<button>Learn More</button></main><footer><p>&copy; 2021 My Website. All Rights Reserved.</p></footer>

📤 output data 输出数据

Sample 1: text
测试
*Home
*About
*Services
*ContactWelcome to My WebsiteLorem ipsum dolor sit amet, consectetur adipiscing elit.Learn More© 2021 My Website. All Rights Reserved.

✨ explanation 解释

This example demonstrates the operator’s ability to process a full HTML document, converting it into plain text. It removes all HTML tags and preserves the text content. The <li> tags are replaced with bullet points, and other elements like headers and paragraphs are flattened into a continuous string. This is useful for extracting readable text from web pages. 此示例展示了算子处理完整HTML文档的能力，将其转换为纯文本。它移除所有HTML标签并保留文本内容。<li>标签被替换为项目符号，而其他如标题和段落的元素则被展平成连续的字符串。这对于从网页中提取可读文本非常有用。

test_no_html_text

CleanHtmlMapper()

📥 input data 输入数据

Sample 1: text
This is a test
Sample 2: text
这是个测试
Sample 3: text
12345678

📤 output data 输出数据

Sample 1: text
This is a test
Sample 2: text
这是个测试
Sample 3: text
12345678

✨ explanation 解释

In this example, the input data does not contain any HTML tags. As a result, the operator simply returns the original text without making any changes. This case illustrates that the operator can handle plain text inputs effectively, ensuring that non-HTML content remains unchanged. 在此示例中，输入数据不包含任何HTML标签。因此，算子直接返回原始文本，不做任何更改。这个案例说明了算子可以有效处理纯文本输入，确保非HTML内容保持不变。