method
Executable task packages for reinforcement learning and agent evaluation.
What You Receive
A package defines the work an agent must complete, the tools it can use, and how its actions are scored.
How It Runs
The agent takes actions, receives updated information, and gets a score for its work.
Domains and Workflows
We scope custom task packages around workflows such as these.
| Domain | Example Workflows |
|---|---|
| Finance Operations | Invoice disputes, payment reconciliation, approvals |
| Credit and Collections | Contact strategy, repayment plans, escalation |
| Customer Operations | Complaint resolution, refunds, account changes |
| Procurement | Supplier onboarding, purchase approvals, contract exceptions |
| IT Service Operations | Incident triage, access requests, change approvals |
| Enterprise Implementation | Scope changes, kickoff planning, go-live readiness |
Agent Evaluation
Graders check completed work, required actions, and policy constraints. Reports include task scores and action histories.
Run different models on the same tasks to compare performance. Repeat the evaluation after a model or agent update to check for regressions.
Environment and Grader Tests
| Test | Scope | Result |
|---|---|---|
| Task validation | 110 packaged tasks | 110 passed |
| Mutation testing | 997 candidate tasks | 9,799 attempts |
| Incorrect changes rejected | Mutation attempts | 6,953 |
| Equivalent changes | Mutation attempts | 1,339 |
| Changes blocked | Mutation attempts | 1,507 |
| Wrong results that passed | Mutation attempts | 0 |
Results from one environment package. These test the environment and grader, not a model's performance.