data_juicer.utils.fingerprint_utils module

class data_juicer.utils.fingerprint_utils.Hasher

基类：object

Hasher that accepts python objects as inputs.

dispatch: Dict = {}

__init__()

classmethod hash_bytes(value: bytes | List[bytes]) → str

classmethod hash_default(value: Any) → str

Use dill to serialize objects to avoid serialization failures.

classmethod hash(value: Any) → str

update(value: Any) → None

hexdigest() → str

data_juicer.utils.fingerprint_utils.update_fingerprint(fingerprint, transform, transform_args)

Combining various objects to update the fingerprint.

data_juicer.utils.fingerprint_utils.generate_fingerprint(ds, *args, **kwargs)

Generate new fingerprints by using various kwargs of the dataset.