🔍 Read the full analysis: How AutoSynthData Generates Training Data For Enterprise Agents on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
ServiceNow CoreAI describes AutoSynthData, a pipeline that uses a target agent’s failures and a stronger teacher model’s successful runs to generate and verify new enterprise training tasks. The company points to the released EnterpriseOps Gym dataset as an example, but the supplied account reports no task counts, measured performance gains or comparisons with other training methods.
ServiceNow CoreAI says it has built AutoSynthData, a system that turns an enterprise agent’s observed failures into new training tasks and checks those tasks inside the target environment, as described in the original analysis. The company presents the released EnterpriseOps Gym dataset as an example of the approach, but the account provides no measured results showing whether training with the generated tasks improves agent performance.
In the described pipeline, a target model attempts diagnostic tasks in an enterprise environment, while a stronger teacher model attempts the same tasks. ServiceNow CoreAI says the paired runs help identify the capability being tested, the tools and workflow involved, where the target agent fails, how the teacher succeeds, and what conditions a valid result must meet.
AutoSynthData distills those observations into sanitized capability specification cards. The company says task generators use the cards rather than the original evaluation prompts, entities, action trajectories or verifier details. They then produce varied tasks, changing wording, starting states, entities, tool combinations and difficulty. Each task includes an environment specification, a user prompt and a verifier that checks whether the request was completed within the environment’s constraints.
Generated tasks are checked in the environment; accepted samples are used for post-training. The updated model can then be evaluated again, with remaining weaknesses used to guide another round. The supplied description does not say how many tasks were generated or accepted, which target and teacher models were used, or how much performance changed after training.
Why Workflow-Specific Training Matters
Enterprise agents must often do more than produce a plausible answer. They may need to update a record, follow access rules, choose among available tools and leave the system in a specified state. A model that performs well on broad evaluations can still mishandle an organization’s particular workflows. AutoSynthData is designed to focus training on those local capability gaps.
If the process works as intended, generating varied practice tasks from observed failures could give teams another way to build training data beyond relying only on manually written examples. That possibility matters because agent actions can change operational data. But the method description does not establish that the approach improves reliability, lowers costs or transfers to other enterprise settings. Those outcomes remain to be tested and reported.
The verifier is central to the proposal. A verifier that accepts an incorrect outcome could reward unsafe or ineffective behavior; one that rejects valid solutions could penalize an agent for taking a different but acceptable path. ServiceNow’s account says verifiers should reflect the request and environment, reject failures and policy violations, and allow valid solutions without requiring one exact sequence of actions.
Products Worth Considering
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
DJI Osmo Mobile 8 3-Axis Gimbal Stabilizer for Phones
As an affiliate, we earn on qualifying purchases.
From Agent Failures to New Tasks
The approach addresses a distinction between what an environment permits and what constitutes realistic work. An agentic environment defines what an agent can observe and change, which tools or APIs it can use, and how its actions affect the system. A task combines that setup with a user-facing prompt and a method for verifying completion. The setup may include instructions, policies or task-specific information such as a seeded database or knowledge articles.
A task can be technically executable but still resemble no plausible workplace request. The reverse is also possible: a realistic-sounding request may be impossible because a tool is unavailable, required information cannot be accessed, the requested change cannot be made, or policy forbids it. The framework says generated tasks are intended to avoid those cases while remaining feasible, realistic and challenging for the target model.
ServiceNow CoreAI cites EnterpriseOps Gym as its example and refers to the released dataset, citing Malay et al. (2026). The supplied material does not provide the dataset’s size, describe the specific workflows tested or give model scores. It also does not specify when the AutoSynthData account was published.
““A model may be broadly capable and still struggle with a particular environment.””
— ServiceNow CoreAI
Performance Evidence Is Missing
The supplied account describes a method, not a results report. It gives no before-and-after performance measurements, evaluation criteria or comparisons with other ways of creating training data. It also omits task-generation volume, verifier acceptance rates, training volume, and the time and cost required to run the pipeline.
It remains unclear how well tasks generated for EnterpriseOps Gym would transfer to other enterprise systems or workflows. The account says the generators receive capability cards instead of original evaluation-task details, but provides no analysis of possible overlap between generated tasks and evaluation material. Without the models, scores, baselines and validation data, readers cannot assess the strength or breadth of the example.
Results Needed to Test the Approach
The next useful evidence would be a reported evaluation of a post-trained model, including the number of tasks generated and accepted, verifier performance, before-and-after results, and a clearly identified baseline. Testing across multiple workflows and environments would help show whether the pipeline addresses recurring operational weaknesses or mainly fits one dataset.
ServiceNow CoreAI’s supplied account does not announce a date for such results or describe a next evaluation milestone. Until the company or independent researchers report those details, AutoSynthData’s practical effectiveness, cost and generalizability remain unverified.
Key Questions
What is AutoSynthData?
AutoSynthData is a system ServiceNow CoreAI says it built to create and check enterprise-agent training tasks based on observed weaknesses in a target model.
How does the task-generation process work?
A target model and a stronger teacher attempt diagnostic tasks. The system uses information from their runs to make capability cards, then generates varied tasks that are checked in the target environment before accepted samples are used for post-training.
What role does EnterpriseOps Gym play?
ServiceNow CoreAI cites the released EnterpriseOps Gym dataset as an example of the approach. The supplied account does not give its size, workflow details or model scores.
Has ServiceNow shown that AutoSynthData improves agent performance?
Not in the supplied material. It reports no measured improvement, baseline comparison or other performance results, so the effectiveness of the approach cannot be assessed from this account alone.
What evidence would help evaluate the system?
Useful evidence would include task-generation and acceptance counts, verifier validation, before-and-after performance against a stated baseline, and results across different workflows and enterprise environments.
Primary source: Hugging Face · via ThorstenMeyerAI.com
