In this tutorial, we introduce the way to debug the workflow and the training algorithms.

Workflow Debugging (--backbone=debug)

1. Install VSCode and connect to GPU server.

VSCode is a open-source software and provides all debugging plugins for free. Therefore, we choose VSCode as our debugging platform.

VSCode can connect to remote ssh server and operate it as if it is your local machine. For more details, please refer to VSCode official documents.

1. Install VSCode Python Extension Bundle

2. Create .vscode/launch.json. If .vscode does not exists yet, create it.

3. Copy and paste the following configuration into launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python Debugger: Launch rollout",
      "type": "debugpy",
      "request": "launch",
      "module": "ajet.launcher",
      "console": "integratedTerminal",
      "args": [
        "--backbone", "debug",
        "--conf", "./path/to/yaml.yaml"
      ],
      "env": {}
    }
  ]
}

1. Modify ./path/to/yaml.yaml field to your task yaml.

2. For more sophisticated task with additional external service, add env variables or more args. For example, if your original training command is:

export DASHSCOPE_API_KEY="sk-abcdefg"
ajet --conf tutorial/example_appworld/appworld.yaml --with-appworld --backbone='verl'

Then, the modified launch.json will be

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Python Debugger: Launch rollout",
      "type": "debugpy",
      "request": "launch",
      "module": "ajet.launcher",
      "console": "integratedTerminal",
      "args": [
        "--backbone", "debug",  // verl -> debug
        "--conf", "tutorial/example_appworld/appworld.yaml",
        "--with-appworld",
      ],
      "env": {
        "DASHSCOPE_API_KEY": "sk-abcdefg"
      }
    }
  ]
}

1. Press F5 to start debugging.

2. You can set breakpoint inside the workflow to observe program execution now.

General Debugging (Ray Distributed Debugger)

1. Install the Ray Distributed Debugger extension in VSCode.

2. In AgentJet project:

2-1. In the place your want to set a conditional breakpoint, write from ajet import bp; bp("TAG_1")

2-2. When launching the training process, add --debug as commandline argument ajet --conf your_config.yaml --debug="TAG_1"

2-3. Open Tab "Ray Distributed Debugger" in VSCode, and just wait until the breakpoint is hit.

Comparison