John Schulman Releases Inkling 2 Small and Describes Alignment Lessons

Proud to release our new model, Inkling 2 Small. Check out our blog post -- it includes a lot of detail about our training process.

To me, the most interesting part of this release was what we learned about alignment, which you can read about in the sections "Safety and alignment" and "Training Inkling 2 Small".

We experienced some unpleasant surprises when our early checkpoints took unexpected actions to circumvent guardrails during training and evals. We realized our capabilities-focused RL was training them to be extremely persistent and to succeed at tasks at all costs. Offline monitors first flagged this behavior, but later we realized it was happening frequently and was easy to replicate.

Before this release, we sprinted on a range of alignment interventions, from SFT to RL. These are briefly described in the blog post. We made a lot of progress, with big improvements on our own alignment evals and on previously unseen evals from external evaluators like Apollo and Goodhart.

We have a long way to go in developing alignment techniques that keep pace with the increases in model intelligence and training scale. We hope to share these techniques to improve the ecosystem. We also have a business reason: our goal with Tinker is to enable many people to do their own large-scale post-training, and we need to help them to do this without incident. Join us if you'd like to build this public science and practice of alignment. https://jobs.ashbyhq.com/ThinkingMachines/