Training a coding model to paint watercolours with TRL and OpenEnv

A viral machine learning experiment published on Hugging Face demonstrates how open-source developers can train small language models to paint realistic digital watercolours using reinforcement learning and interactive execution feedback.

Authored by researchers using Hugging Face's TRL (Transformer Reinforcement Learning) library and the OpenEnv framework, the project explores an intriguing alternative to standard diffusion models: teaching coding LLMs to write procedural Python scripts that physically simulate watercolour physics, brush pressure, and pigment diffusion.

The project paper and codebase, titled *"Training a coding model to paint watercolours with TRL and OpenEnv"*, has quickly climbed developer ranking boards on GitHub and Hacker News.

Most modern image generators (like Midjourney or Stable Diffusion) operate as dense diffusion networks, predicting pixel values directly from noise vectors. While visually impressive, they are notorious black boxes: users cannot edit individual brushstrokes, adjust pigment layers, or inspect how the artwork was constructed.

The Hugging Face researchers took a radically different path:

1. 1.The LLM Writes Executable Canvas Commands: The model does not output images. Instead, it outputs Python scripts using graphics libraries like PyCairo and NumPy to render procedural splatters, brush washes, and translucent pigment blending.
2. 2.The OpenEnv Feedback Sandbox: The generated code is compiled and executed in a headless sandbox environment.
3. 3.Reward Modeling: An automated visual critic compares the rendered image against a target reference, assigning a reward score based on color harmony, texture accuracy, and stroke efficiency.
4. 4.Policy Updates with TRL: The model updates its code generation policy using PPO (Proximal Policy Optimization), learning which mathematical formulas produce realistic wet-on-wet watercolour bleeding.

According to the author's notes:

"In every single training run, the first thing the model learned was to stop producing invalid Python syntax and runtime crashes. Once it achieved deterministic execution, it rapidly developed complex layering techniques—laying down light background washes first before applying detailed dry-brush outlines, mimicking the exact technique human watercolorists use."

Beyond digital art, this research offers a compelling blueprint for software engineering agents operating in the real world:

• Self-Correcting Execution Loops: Models trained with environment feedback learn how their code behaves at runtime, drastically reducing syntax and logic bugs.
• Compact Footprint: Because the model only outputs lightweight text commands, the entire system runs on modest compute clusters without needing multi-GPU diffusion backends.
• Deterministic Reproducibility: Every painting is a clean, version-controlled Python script that can be scaled infinitely to any resolution without pixelation.

The training scripts, reward functions, and OpenEnv configurations are available openly on Hugging Face.