OWA Data Pipeline: From Raw MCAP to VLA Training¶

Quick Demo: 3 Commands to VLA Training¶

Step 1: Process raw MCAP files

Step 2: Create time bins (optional)

Step 3: Train your model

That's it! Your MCAP recordings are now ready for VLA training.

The OWA Data Pipeline is a streamlined 2-stage processing system that transforms raw MCAP recordings into training-ready datasets for Vision-Language-Action (VLA) models. This pipeline bridges the gap between desktop interaction capture and foundation model training.

Pipeline Architecture¶

graph LR
    A[Raw MCAP Files] --> B[Stage 1: Event Dataset]
    B --> C[Stage 2: Binned Dataset]
    B --> D[Dataset Transforms]
    C --> D
    D --> E[VLA Training Ready]

    style A fill:#e1f5fe
    style B fill:#f3e5f5
    style C fill:#e8f5e8
    style D fill:#fff3e0
    style E fill:#ffebee

Key Features:

• 🔄 Flexible: Skip binning and use Event Dataset directly, or use traditional Binned Dataset approach
• 💾 Storage Optimized: Since event/binned dataset saves only reference to media, the entire pipeline is designed to be space-efficient.

  /data/
  ├── mcaps/           # Raw recordings (400MB)
  ├── event-dataset/   # References only (20MB)
  └── binned-dataset/  # Aggregated refs (2MB)
  

• 🤗 Native HuggingFace: Event/binned dataset is a true HuggingFace datasets.Dataset with set_transform(), not wrappers.

  # Since event/binned datasets are true HuggingFace datasets,
  # they can be loaded directly into training pipelines
  from datasets import load_from_disk
  dataset = load_from_disk("/data/event-dataset")
  dataset = load_from_disk("/data/binned-dataset")
  
  # Transform to VLA training format is applied on-the-fly during training
  from owa.data import create_binned_dataset_transform
  transform = create_binned_dataset_transform(
      encoder_type="hierarchical",
      instruction="Complete the computer task",
  )
  dataset.set_transform(transform)
  
  # Use in training
  for sample in dataset["train"].take(1):
      print(f"Images: {len(sample['images'])} frames")
      print(f"Actions: {sample['encoded_events'][:3]}...")
      print(f"Instruction: {sample['instruction']}")
  

• ⚡ Compute-optimized, On-the-Fly Processing: During preprocess stage, media is not loaded. During training, only the required media is loaded on-demand.

Stage 1: Raw MCAP → Event Dataset¶

Script Usage¶

python scripts/01_raw_events_to_event_dataset.py \
  --train-dir /path/to/mcap/files \
  --output-dir /path/to/event/dataset \
  --rate mouse=60 --rate screen=20 \
  --keep_topic screen --keep_topic keyboard

Key Parameters¶

Output Schema¶

The Event Dataset uses a flat structure optimized for temporal queries:

{
    "file_path": Value("string"),      # Source MCAP file path
    "topic": Value("string"),          # Event topic (keyboard, mouse, screen)
    "timestamp_ns": Value("int64"),    # Timestamp in nanoseconds
    "message_type": Value("string"),   # Full message type identifier
    "mcap_message": Value("binary"),   # Serialized McapMessage bytes
}

Stage 2: Event Dataset → Binned Dataset¶

Script Usage¶

python scripts/02_event_dataset_to_binned_dataset.py \
  --input-dir /path/to/event/dataset \
  --output-dir /path/to/binned/dataset \
  --fps 10 \
  --filter-empty-actions

Key Parameters¶

Output Schema¶

The Binned Dataset organizes events into temporal bins with state-action separation:

{
    "file_path": Value("string"),      # Source MCAP file path
    "bin_idx": Value("int32"),         # Time bin index
    "timestamp_ns": Value("int64"),    # Bin start timestamp
    "state": Sequence(feature=Value("binary"), length=-1),    # Screen events
    "actions": Sequence(feature=Value("binary"), length=-1),  # Action events
}

Dataset Transforms: The Magic Layer¶

Dataset transforms provide the crucial bridge between stored data and training-ready format. They apply on-demand during data loading, not during preprocessing.

Unified Transform Interface¶

Both Event Dataset and Binned Dataset support the same transform interface:

from datasets import load_from_disk
from owa.data import create_event_dataset_transform

# Load dataset
dataset = load_from_disk("/path/to/event-dataset")

# Create transform
transform = create_event_dataset_transform(
    encoder_type="hierarchical",
    load_images=True,
    encode_actions=True,
)

# Apply transform
dataset.set_transform(transform)

# Use in training
for sample in dataset["train"]:
    images = sample["images"]          # List[PIL.Image]
    events = sample["encoded_events"]  # List[str]

from datasets import load_from_disk
from owa.data import create_binned_dataset_transform

# Load dataset
dataset = load_from_disk("/path/to/binned-dataset")

# Create transform
transform = create_binned_dataset_transform(
    encoder_type="hierarchical",
    instruction="Complete the computer task",
    load_images=True,
    encode_actions=True,
)

# Apply transform
dataset.set_transform(transform)

# Use in training
for sample in dataset["train"]:
    images = sample["images"]          # List[PIL.Image]
    actions = sample["encoded_events"] # List[str]
    instruction = sample["instruction"] # str

Transform Parameters¶

References¶

• Format Guide - OWAMcap details
• Recording Data - Create with ocap
• CLI Reference - owl commands
• HuggingFace Datasets - datasets library