trinity.buffer package

Subpackages

• trinity.buffer.reader package
  • Submodules
    • trinity.buffer.reader.file_reader module
      • DummyProgressBar
      • BaseFileReader
      • ExperienceFileReader
      • TaskFileReader
    • trinity.buffer.reader.queue_reader module
      • QueueReader
    • trinity.buffer.reader.sql_reader module
      • SQLReader
  • Module contents
• trinity.buffer.schema package
  • Submodules
    • trinity.buffer.schema.sql_schema module
      • TaskModel
      • ExperienceModel
      • SFTDataModel
      • DPODataModel
      • init_engine()
  • Module contents
    • init_engine()
• trinity.buffer.writer package
  • Submodules
    • trinity.buffer.writer.file_writer module
      • JSONWriter
    • trinity.buffer.writer.queue_writer module
      • QueueWriter
    • trinity.buffer.writer.sql_writer module
      • SQLWriter
  • Module contents

Submodules

• trinity.buffer.buffer module
  • get_buffer_reader()
  • get_buffer_writer()
• trinity.buffer.buffer_reader module
  • BufferReader
    • BufferReader.read()
    • BufferReader.read_async()
    • BufferReader.state_dict()
    • BufferReader.load_state_dict()
• trinity.buffer.buffer_writer module
  • BufferWriter
    • BufferWriter.write()
    • BufferWriter.write_async()
    • BufferWriter.acquire()
    • BufferWriter.release()
• trinity.buffer.queue module
• trinity.buffer.ray_wrapper module
• trinity.buffer.utils module
  • retry_session()

Module contents

class trinity.buffer.BufferReader

Bases: ABC

Interface of the buffer reader.

abstract read(batch_size: int | None = None) → List

Read from buffer.

abstract async read_async(batch_size: int | None = None) → List

Read from buffer asynchronously.

state_dict() → Dict

load_state_dict(state_dict: Dict) → None

class trinity.buffer.BufferWriter

Bases: ABC

Interface of the buffer writer.

abstract write(data: List) → None

Write to buffer.

abstract async write_async(data: List) → None

Write to buffer asynchronously.

abstract async acquire() → int

Acquire the buffer writer.

Returns:

The reference count of the buffer after acquiring.

Return type:

int

abstract async release() → int

Release the buffer writer. After release, the buffer writer can not be used again.

Returns:

The reference count of the buffer after releasing.

Return type:

int

trinity.buffer.get_buffer_reader(config: TasksetConfig | ExperienceBufferConfig | StorageConfig) → BufferReader

Get a buffer reader for the given dataset name.

trinity.buffer.get_buffer_writer(config: TasksetConfig | ExperienceBufferConfig | StorageConfig) → BufferWriter

Get a buffer writer for the given dataset name.