离线 DPO 和 SFT

离线 DPO 和 SFT#

本示例描述了基于 Qwen2.5-1.5B-Instruct 模型的 DPO 和 SFT 流程。

第一步：模型和数据准备#

模型准备#

将 Qwen2.5-1.5B-Instruct 模型下载到本地目录 $MODEL_PATH/Qwen2.5-1.5B-Instruct：

# 使用 Modelscope
modelscope download Qwen/Qwen2.5-1.5B-Instruct --local_dir $MODEL_PATH/Qwen2.5-1.5B-Instruct

# 使用 Huggingface
huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct --local-dir $MODEL_PATH/Qwen2.5-1.5B-Instruct

更多关于模型下载的细节，请参考 ModelScope 或 Huggingface。

数据准备#

对于 DPO，我们将 Human-like-DPO-dataset 下载到本地目录 $DATASET_PATH/human_like_dpo_dataset：

# 使用 Modelscope
modelscope download --dataset HumanLLMs/Human-Like-DPO-Dataset --local_dir $DATASET_PATH/human_like_dpo_dataset

# 使用 Huggingface
huggingface-cli download HumanLLMs/Human-Like-DPO-Dataset --repo-type dataset --local-dir $DATASET_PATH/human_like_dpo_dataset

以下是部分以 JSONL 格式存储的数据样本：

{"prompt":"Oh, I just saw the best meme - have you seen it?","chosen":"\ud83d\ude02 Ah, no I haven't! I'm dying to know, what's the meme about? Is it a funny cat or a ridiculous situation? Spill the beans! \ud83e\udd23","rejected":"I'm an artificial intelligence language model, I don't have personal experiences or opinions. However, I can provide you with information on highly-rated and critically acclaimed films, as well as recommendations based on specific genres or themes. Would you like me to suggest some notable movies or discuss a particular genre of interest?"}
{"prompt":"Have you tried any new hobbies or activities recently?","chosen":"You know, I've been meaning to try my hand at gardening, but I haven't gotten around to it yet. I've heard it's super relaxing and a great way to get some fresh air. Maybe I'll finally get around to buying some seeds and pots this weekend. What about you? Have you taken up anything new and exciting lately? \ud83c\udf31\ud83d\udc40","rejected":"I'm an artificial intelligence language model, and as such, I don't have personal experiences or engage in physical activities such as dining or cooking. My purpose is to provide information, answer questions, and assist with tasks to the best of my abilities, while maintaining a professional and impartial demeanor. If you have any specific questions or topics related to restaurants or recipes, I'd be happy to provide information or guidance."}

更多关于数据集下载的细节，请参考 ModelScope 或 Huggingface。

注意该数据集包含 prompt、chosen 和 rejected 三个键。如果你使用其他数据集，请在配置中传入正确的键名。

对于 SFT，我们下载 open-r1/Mixture-of-Thoughts 数据集到本地目录 $DATASET_PATH/Mixture-of-Thoughts，其中包含基于 message 的数据，这里列出一个简化的样本：

{"messages": [{"content": "You will be given a competitive programming problem...","role": "user"},{"content": "<think>\n...
</think>
\n...This approach efficiently combines hashing and dynamic programming to solve the problem within the given constraints.","role": "assistant"}], "num_tokens": 22185, "source": "open-r1/codeforces-cots"}

第二步：配置设置#

DPO 配置#

我们在实验中使用 dpo.yaml 中的配置。以下列出一些重要设置：

我们在 train 模式下运行实验，因为没有使用 explorer。要启用此模式，需将 mode 设置为 train，并将数据路径传递给 trainer。

project: <project_name>
name: <experiment_name>
mode: train
algorithm:
  algorithm_type: dpo
  kl_loss_fn: k1
  kl_loss_fn_args:
    kl_coef: 0.1  # DPO 中 beta 的值
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-1.5B-Instruct}
cluster:
  node_num: 1
  gpu_per_node: 8
buffer:
  total_epochs: 2
  train_batch_size: 64
  trainer_input:
    experience_buffer:
      name: human_like_dpo
      storage_type: file
      path: $DATASET_PATH/human_like_dpo_dataset
      format:
        prompt_type: plaintext
        prompt_key: prompt
        chosen_key: chosen
        rejected_key: rejected
trainer:
  save_interval: 30
  trainer_config:
    ... # 省略其他配置

buffer.trainer_input.experience_buffer 指定了用于训练的数据集，包括其名称、存储类型、路径和格式。

  • 名称 human_like_dpo 是此数据集配置的唯一标识符，只要在项目内保持唯一性，你可以使用其他名称。

  • 存储类型 file 表示数据集存储在本地文件系统中的文件里，path 指向数据集所在的本地目录。注意 file 类型也支持使用 HuggingFace 数据集路径，例如 HumanLLMs/Human-Like-DPO-Dataset。

  • format 定义了数据集内部的数据结构，具体如下：

    • prompt_type: plaintext 表示提示是纯文本格式。

    • prompt_key: prompt 指定数据集中包含用户提示的字段名。

    • chosen_key: chosen 指定数据集中包含被选中回复的字段名。

    • rejected_key: rejected 指定数据集中包含被拒绝回复的字段名。

更多配置选项，请参考 参数配置指南。

SFT 配置#

我们将 algorithm_type 设为 sft 来运行 SFT 流程，并对配置文件 examples/sft_mot/sft.yaml 进行如下修改：

project: <project_name>
name: <experiment_name>
mode: train
algorithm:
  algorithm_type: sft
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-1.5B-Instruct}
cluster:
  node_num: 1
  gpu_per_node: 2
buffer:
  total_epochs: 5
  train_batch_size: 64
  trainer_input:
    experience_buffer:
      name: <sft_dataset_name>
      storage_type: file
      path: $DATASET_PATH/Mixture-of-Thoughts
      split: train
      format:
        prompt_type: messages
        messages_key: messages
trainer:
  save_interval: 50
  trainer_config:
    ... # 省略其他配置

此处我们将 buffer.trainer_input.experience_buffer.format.prompt_type 设为 messages，因为源数据是 message 格式。同时设置 buffer.trainer_input.experience_buffer.format.messages_key 为 messages，以指定数据集中包含消息的字段名。

第三步：运行实验#

使用以下命令运行 DPO 流程：

trinity run --config examples/dpo_humanlike/dpo.yaml

或运行 SFT：

trinity run --config examples/sft_mot/sft.yaml
目录