Delphi Digital AI summary of the transcript:
- Open source language models are being enhanced with reinforcement learning (RL) infrastructure to overcome limitations of supervised learning
- RL allows for nearly arbitrary reward functions including discrete, non-differentiable objectives across multi-step trajectories
- Language models map perfectly to RL: text prefixes as states, next tokens as actions, with LMs serving as policies
- The industry is entering an "era of environments" similar to data scaling, requiring standardization for open source collaboration
- News Research's distributed training infrastructure uses three microservices: trainer (backprop), inference (model serving), and environment manager
- Their minimal environment interface consists of two functions: get_item (data loading) and collect_trajectories (fused inference and scoring)
- The approach supports multi-turn, multi-agent setups by handling multiple inference-scoring cycles within collect_trajectories
- Environments return token-level outputs rather than text, enabling custom chat templates, mixing of instruction/base model fine-tuning, and token-level reward weighting
- The extensible framework allows for environment-specific customizations such as different attention masking schemes for specialized use cases