Debugging Techniques

Advanced debugging with breakpoints, log analysis, and ASL replay

Overview

Level 3 debugging in AlpaSim enables breakpoint debugging of individual components while running a full simulation. This allows you to debug complex interactions between microservices with your favorite debugger attached.

This section covers debugging simulation components (controller, runtime, driver), not vehicle behavior within simulation. For basic configuration debugging, see the Quickstart Guide.

Breakpoint Debugging Workflow

The basic approach to enable breakpoint debugging:

1. Generate config files without running

   Use the wizard to create configuration files but skip execution:

   alpasim_wizard +deploy=local \
     wizard.log_dir=$PWD/tutorial_dbg \
     wizard.run_method=NONE \
     wizard.debug_flags.use_localhost=True
   
2. Start the component manually

   Launch the component you want to debug from your IDE or terminal with breakpoints set

3. Run the rest of the simulation

   Start remaining services via docker-compose, excluding the manually-started component

Debugging the Controller

The controller manages ego vehicle trajectory tracking and pose estimation.

Terminal-Based Debugging

1. Generate configs

   alpasim_wizard +deploy=local \
     wizard.log_dir=$PWD/tutorial_dbg \
     wizard.run_method=NONE \
     wizard.debug_flags.use_localhost=True
   
2. Note the allocated port

   Check tutorial_dbg/docker-compose.yaml for the controller port. Look for components in the sim profile: controller-0, driver-0, physics-0, runtime-0, sensorsim-0.

   Example: controller-0 might be allocated port 6003.

3. Start controller with debugger

   From src/controller/, start the controller manually:

   cd src/controller/
   mkdir my_controller_log_dir
   # Port must match docker-compose.yaml
   uv run python -m alpasim_controller.server \
     --port=6003 \
     --log_dir=my_controller_log_dir \
     --log-level=INFO
   
4. Launch remaining services

   From the generated directory:

   cd tutorial_dbg
   docker compose -f docker-compose.yaml --profile sim up \
     runtime-0 driver-0 physics-0 sensorsim-0
   

VSCode Debugger Configuration

For a better debugging experience, use VSCode’s built-in debugger:

.vscode/launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Controller (Level 3 Tutorial)",
      "type": "debugpy",
      "request": "launch",
      "module": "alpasim_controller.server",
      "justMyCode": false,
      "cwd": "${workspaceFolder}/src/controller",
      "args": [
        "--port=6003",
        "--log_dir=my_controller_logdir",
        "--log-level=INFO"
      ],
      "console": "integratedTerminal"
    }
  ]
}

1. Set breakpoints

   Open controller source files and click in the gutter to set breakpoints

2. Launch debugger

   Press F5 or go to Run and Debug → “Debug Controller”

3. Start simulation

   Breakpoints will hit as the simulation runs!

Make sure the --port argument matches the port allocated in docker-compose.yaml.

Debugging the Runtime

Debugging the runtime requires a different approach since other services must be running before the runtime starts.

Key Differences

• Start order: Other services must be up before launching the runtime
• Speed iteration: Set runtime.endpoints.do_shutdown=False to keep containers running between runs

Terminal-Based Debugging

1. Generate configs with no shutdown

   alpasim_wizard +deploy=local \
     wizard.log_dir=$PWD/tutorial_dbg_runtime \
     wizard.run_method=NONE \
     wizard.debug_flags.use_localhost=True \
     runtime.endpoints.do_shutdown=False
   
2. Start non-runtime services

   cd tutorial_dbg_runtime
   docker compose -f docker-compose.yaml --profile sim up \
     driver-0 controller-0 physics-0 sensorsim-0
   
3. Start runtime manually

   From src/runtime/:

   cd src/runtime/
   uv run python -m alpasim_runtime.simulate \
     --usdz-glob=../../data/nre-artifacts/all-usdzs/**/*.usdz \
     --user-config=../../tutorial_dbg_runtime/generated-user-config-0.yaml \
     --network-config=../../tutorial_dbg_runtime/generated-network-config.yaml \
     --log-dir=../../tutorial_dbg_runtime \
     --log-level=INFO
   

VSCode Debugger Configuration

.vscode/launch.json

{
  "name": "Debug Runtime (Level 3 Tutorial)",
  "type": "debugpy",
  "request": "launch",
  "module": "alpasim_runtime.simulate",
  "justMyCode": false,
  "cwd": "${workspaceFolder}/src/runtime",
  "args": [
    "--usdz-glob=../../data/nre-artifacts/all-usdzs/**/*.usdz",
    "--user-config=../../tutorial_dbg_runtime/generated-user-config-0.yaml",
    "--network-config=../../tutorial_dbg_runtime/generated-network-config.yaml",
    "--log-dir=../../tutorial_dbg_runtime",
    "--log-level=INFO"
  ],
  "console": "integratedTerminal"
}

ASL Log Analysis

What are ASL Logs?

ASL (AlpaSim Log) files contain most messages exchanged during simulation as size-delimited protobuf messages. They’re stored in rollouts/{scene_id}/{batch_uuid}/rollout.asl.

Use cases:

• Debugging model behavior
• Replaying stimuli on a driver instance
• Reproducing bugs with a debugger attached
• Creating evaluation videos

Reading ASL Logs

Use the async_read_pb_log utility to stream messages:

Read First 20 Messages

from alpasim_grpc.utils.logs import async_read_pb_log

i = 0
async for log_entry in async_read_pb_log("path/to/rollout.asl"):
    print(log_entry)
    i += 1
    if i == 20:
        break

Message Types

ASL files contain three types of messages:

RolloutMetadata

Metadata header for reproducibility and bookkeeping:

• Session UUID and scene ID
• Batch size and simulation steps
• Timestamp and control frequency
• Version information (runtime, driver, API)
• Actor definitions

ActorPoses

Location of all actors (including 'EGO') in global coordinate space:

• Position (x, y, z)
• Orientation (quaternion)
• Velocity
• Timestamp

Service Requests/Responses

Microservice RPC calls enable replay mode:

• RolloutCameraImage - Camera frame requests (used to create MP4 videos)
• Driver planning requests/responses
• Physics update calls
• Sensor simulation requests

Replaying Stimuli

The replay_logs_alpamodel.ipynb notebook shows how to:

1. Read an ASL log
2. Extract driver stimuli (camera frames, ego poses)
3. Replay them on a driver instance
4. Reproduce behavior with a debugger attached

ASL logs don’t specify the USDZ file UUID - you’ll need to match by scene_id from metadata.

Console Log Analysis

The docker-compose console contains logs from all microservices and is the first place to look when something goes wrong.

Basic Troubleshooting

1. Check for missing output

   If the rollouts directory doesn’t appear, check console logs for errors

2. Find first error

   Scroll to find where the first errors occurred - subsequent errors are often cascading failures

3. Service-specific logs

   Additional logs may be in:

   • txt-logs/ - Per-service text logs
   • driver/ - Driver service logs
   • telemetry/ - Performance metrics

Microservices may produce additional debug logs not shown in the console. Check txt-logs/ for detailed service output.