🔍 Read the full analysis: Training A Coding Model To Paint Watercolours With TRL And OpenEnv on ThorstenMeyerAI.com
TL;DR
A developer has created an open, end-to-end reproduction of Surya Narreddi’s watercolour AI model, utilizing TRL and OpenEnv. All artifacts, including datasets, scripts, and trained models, are now publicly available, enabling further research on aesthetic reinforcement learning.
An independent engineer has published a full, open-source reproduction of Surya Narreddi’s viral watercolour-painting language model, which gained widespread attention in August. This reproduction, built using the frameworks TRL and OpenEnv, includes all datasets, training scripts, and trained models, making it accessible for further experimentation and development. The project aims to explore whether reinforcement learning can optimize a model based on aesthetic taste rather than verifiable correctness, marking a significant step in AI art research.
The reproduction leverages a pipeline that trains a Qwen model with reinforcement learning against a reward mix designed to favor style and aesthetic qualities. The reward components include a compilation check, a length penalty, a style judge, and a preference model called HPSv3, which is trained on human choices between image pairs. The style judge, Qwen3-VL-30B-A3B-Instruct, assesses the model’s output by comparing it to four reference images, guiding the training process toward more artistic and stylistically consistent outputs.
All artifacts—datasets, training scripts, reward models, and the trained language model—are hosted on Hugging Face, ensuring open access. The training process involves a single command that runs 110 steps, with 240 episodes per step and eight generations per episode, using the Hugging Face inference infrastructure. The project’s core innovation lies in its open implementation and the release of a curated reference pool, enabling others to replicate or extend the work.
This development addresses a core question in AI art: whether reinforcement learning can be used to optimize models based on aesthetic preferences, rather than explicit correctness. The original project by Narreddi involved training a language model to generate JavaScript code that produces watercolour-style images through the p5.js library, with style enforced via restrictions on library methods. The open reproduction aims to validate and expand on these findings, providing the community with tools to explore aesthetic reinforcement learning further.
Implications for AI-Generated Art and Reinforcement Learning
This open reproduction represents a significant step in democratizing AI art research, particularly in the domain of aesthetic optimization. By releasing all components publicly, it enables researchers and artists to experiment with reinforcement learning techniques that prioritize subjective qualities like style and taste, rather than objective correctness. This could lead to new forms of AI-generated art that are more expressive and aligned with human aesthetic preferences.
The project also challenges the prevailing focus on verifiable rewards in AI training, demonstrating that models can be guided by human-like taste signals. As a result, it opens pathways for developing AI systems capable of producing more nuanced, handcrafted-style artworks, blurring the lines between algorithmic and human creativity. The approach may influence future work on AI-assisted art creation, training methodologies, and evaluation metrics rooted in subjective judgment.
digital watercolor painting tablet
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Background of AI Art and Reinforcement Learning for Style
The project builds on a long history of AI art, starting with early experiments like DeepDream (2015), which transformed neural networks into artistic tools, and progressing through projects like Edmond de Belamy (2018) and neural portraits by Mario Klingemann. These works explored the medium’s potential, often focusing on generative adversarial networks (GANs) and other models trained on curated datasets.
Surya Narreddi’s initial work involved prompting text-to-image models and training language models to generate JavaScript code that produces watercolour-like images via the p5.js library. His viral video in August, which showcased these watercolour paintings, sparked widespread interest, partly because of the model’s imperfect, handmade aesthetic contrasting with the polished outputs typical of mainstream image generators. The open reproduction aims to extend this lineage by making the entire training pipeline accessible and replicable.
Prior efforts in this space have often relied on closed datasets or proprietary models, limiting community engagement. Narreddi’s approach, as described in his blog, emphasizes hand-curated datasets and reinforcement learning based on human preferences, positioning his work within a broader movement exploring AI as a tool for artistic expression rather than purely functional outputs.
“This open-source reproduction provides a valuable resource for exploring how reinforcement learning can be applied to aesthetic judgment, a frontier in AI art.”
— Thorsten Meyer, AI researcher
As an affiliate, we earn on qualifying purchases.
Unconfirmed Aspects and Future Validation Needs
While the reproduction is comprehensive, it is not yet clear how closely the generated outputs match the quality and style of Narreddi’s original viral paintings. The project’s authors have not provided quantitative comparisons or final evaluation metrics for the different reward mixes tested. Additionally, the full technical report promised by Narreddi remains unpublished, leaving some technical details and validation results unverified. It is also uncertain whether the open implementation will produce consistent results across different hardware and settings, or how it will perform on new datasets or styles.
Further validation and peer review are needed to assess the reproducibility and artistic quality of the outputs, as well as the broader implications of using reinforcement learning for aesthetic judgment in AI art.
As an affiliate, we earn on qualifying purchases.
Next Steps for Community Engagement and Technical Development
The immediate next step is for the community to experiment with the released artifacts, testing the reproduction’s outputs and potentially extending the training environment with new datasets or reward configurations. Researchers are expected to analyze the impact of different reward mixes on style and quality, providing more quantitative assessments in future reports.
Meanwhile, the original author, Narreddi, has announced that a comprehensive technical report will be published soon, which should clarify the methodology and results further. The open-source project also invites collaboration, with potential improvements in the training pipeline, reward design, and model architecture to enhance aesthetic fidelity and diversity.
As the field progresses, future developments may include integrating more sophisticated human preference models, expanding the dataset pool, or applying similar reinforcement learning techniques to other artistic styles and mediums.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main goal of this open reproduction?
The primary aim is to demonstrate that reinforcement learning can be used to optimize AI models based on aesthetic taste, specifically in generating watercolour-style artworks through code, and to provide open tools for further research.
What artifacts are available for the community to use?
All datasets, training scripts, reward models, and trained models are publicly hosted on Hugging Face, enabling others to replicate or extend the work.
How does this differ from traditional AI art models?
Unlike models trained solely on pixel data or static datasets, this approach uses reinforcement learning guided by human-like aesthetic preferences, with the output being editable code that produces watercolour effects.
What remains to be validated about the results?
It is still unclear how closely the open reproduction matches the original in artistic quality, and quantitative evaluations comparing reward mixes and output styles are not yet available.
Will the original creator publish a detailed technical report?
Yes, Narreddi has announced that a full technical report is forthcoming, which should provide more detailed insights into the methodology and results.
Primary source: Hugging Face · via ThorstenMeyerAI.com
