# How to grade your company's AI agent setup

> A scorecard for go-to-market agents you have already built: how to test each agent, the ten questions that grade the system around them, where home-built setups break, and how Day AI closes those gaps.

Updated October 14, 2026

**The short answer.** Grade two layers. First, does each agent do its job? Build a small test set from real requests, grade the final response, the path the agent took and the state it changed, run each case several times, and aim for 80 to 90% passing, with a regression set that passes nearly every time. Second, does the system around the agents get better? Check whether people can reply to an agent and change how it runs, who writes the prompts, what is locked and what is personal, whether every change and run is recorded, and whether anything improves while you are away. Most home-built setups pass the first layer and fail the second. [Agent Grader](https://agent-grader.com) scores the second layer in about four minutes.

You probably built something real: Claude behind a Slack bot, a handful of skills on a cron schedule, a homegrown memory database, maybe Agentforce on top of Salesforce. This guide grades that setup as it stands. It scores the system around your agents, not any single answer, and it works whatever you built on.

---

## Layer one: does each agent do its job?

Published evaluation checklists from Microsoft and LangChain agree on the basics, and they apply to a meeting-prep agent as much as to a support bot.

- **Start from real traffic.** LangChain's advice is to "manually review 20-50 real agent traces before building any eval infrastructure". For a go-to-market agent, that means reading a few weeks of the briefs, drafts and CRM updates it actually produced.
- **Grade three things.** LangChain names them: the final response, the trajectory (which tools it called, in what order) and the state changes. The third is the one teams skip. In LangChain's words, "If it updates a database, query the rows. The final response can say \"Done!\" while the actual state is wrong."
- **Use pass or fail.** LangChain recommends binary pass/fail over numeric scales. "The brief names the economic buyer" is a test. "The brief is a 7 out of 10" is an opinion.
- **Run each case more than once.** The same prompt gives different answers on different runs. Microsoft's checklist says to "run each test set multiple times and calculate the average success rate. Aim for a realistic pass rate of 80-90%, based on your business needs."
- **Keep a regression set.** A short list of cases that must never break, which LangChain says "should have ~100% pass rate and catch backsliding."
- **Re-run on change.** Microsoft lists the triggers for a full re-run: a model change, a major knowledge update, a new tool or connector, and a production incident.
- **Test permissions.** One of Microsoft's acceptance criteria is "Does it respect sharing permissions?" For a sales agent, include a case where the right answer is that the person asking can't see the deal.

That gets you a working eval for each agent. It doesn't tell you whether the setup as a whole is getting better or slowly decaying. That is the second layer.

## Layer two: the ten-question scorecard

These ten questions grade the system around your agents. They follow the ten dimensions [Agent Grader](https://agent-grader.com) uses, which contrast a scheduler with good prompts against a control plane. Each row ends with a test you can run this week, with whatever you have built.

| Dimension and question | Failing vs. good | A test to run this week |
|---|---|---|
| **1. Talk-back.** Can the person reading an agent's output change how it behaves by replying to it? | **Failing:** Replies go nowhere, or to a DM with the person who built it. **Good:** A reply reaches the agent, and an instruction carries into later runs. | Reply to this morning's output with one instruction, such as "leave the demo pipeline out". Does the next scheduled run follow it? The one after? |
| **2. Authorship.** Who writes and changes the prompts? | **Failing:** One builder owns every prompt, and they haven't changed in months. **Good:** The people who use an agent can shape their own version without filing a request. | List every prompt edit from the last 30 days and who made it. |
| **3. Governance.** Can you lock what must be the same for everyone and free what should be personal? | **Failing:** Everything locked, or everything open. **Good:** Each agent is set deliberately: centrally managed, a starting template, or personal. | Pick one agent that must be consistent (pricing rules) and one that should be personal (prep style). Can your setup treat them differently? |
| **4. Harvest.** Do improvements one person makes reach everyone else? | **Failing:** The best rep's version stays in their copy, and the shared rubric is frozen. **Good:** Someone can see the variations and promote the better one. | Find the person who has changed their agent the most. Can you see what they changed? |
| **5. Measured outcomes.** Do you know whether the agents' work gets used? | **Failing:** The bot reports on itself: the bot grading its own homework. **Good:** Outcomes measured outside the agent: a draft sent, a stage moved, a brief opened. | Pick one agent and one outcome you can count without asking the agent. Count the last two weeks. |
| **6. Compounding.** Does each person's agent get better at serving that person? | **Failing:** Day 100 looks like day 1, and every user gets the builder's style. **Good:** The agent keeps each person's standing preferences and the shared customer history. | Have an AE and a CSM run the same agent on the same account. Does each get output that fits their job? |
| **7. Eval.** Is there a test set, a pass bar and a re-run schedule? | **Failing:** "We tried it a few times and it looked fine". **Good:** Layer one above, re-run on every trigger. | Write ten real requests with the expected outcome, run each three times, and record the pass rate. |
| **8. Feedback visibility.** Can you see what changed in an agent, when, and why, and undo it? | **Failing:** Prompts edited in place, and no record of the runs. **Good:** Versioned prompts you can restore, and a history of every run. | Answer "what did this agent's instructions say two weeks ago?" in under a minute, then roll back to it. |
| **9. Loops.** Does the system improve while the builder is away? | **Failing:** The builder goes on vacation and the system stands still. **Good:** Feedback from users changes the agents directly, and work starts on events as well as schedules. | Change nothing for a week. At the end, has anything changed because of what users said? |
| **10. The keyboard test.** Does the setup depend on people typing data in? | **Failing:** Agents read whatever reps remembered to type into the CRM. **Good:** Agents read the calls and emails, and write the CRM back. | Your agent says it updated a deal: query the CRM row. Then check who typed the last five stage changes. |

### The three tiers

Agent Grader places each setup in one of three tiers. You can place yours from the table.

- **A scheduler with good prompts.** Agents run on a timer and the output is good on a good day. Rows 1, 2, 8 and 9 fail: nobody can talk back, one person writes every prompt, and nothing changes unless that person changes it. This is where most home-built setups are, and it is a fine place to start.
- **A system, not yet a loop.** There is a test set, the prompts are versioned, more than one person can change an agent, and there are governance rules. What's missing is the loop: feedback from the people using the agents doesn't yet flow back into them by itself, so improvement still waits on someone.
- **A control plane.** Each person can shape their own agents within rules an admin sets, every change and run is recorded, replies change behavior, and the setup improves from use. The builder steers it and is no longer the bottleneck.

---

## Where home-built setups break

Agent Grader includes eight short cases, each a setup that failed one dimension. Four of them, retold briefly:

**Talk-back: Dex, technical co-founder, 31 out of 100.** An AE replied to the morning briefing: stop including the demo pipeline. The reply went nowhere. By week three, that AE and three other reps had stopped reading the briefing.

**Permissions: Ravi, RevOps at a cybersecurity company, 29 out of 100.** One Slack bot ran on a single service account and posted summaries into a shared channel. The summaries surfaced details not everyone in the channel was cleared to see.

**Memory: Marco, RevOps at a 150-person company, 38 out of 100.** Claude behind Slack, with a 23-table SQLite database for memory. Past a hundred people, the rubrics didn't fit customer success or the TAMs, the bot took on Marco's own personality, and there was no way to personalize it for each user.

**Authorship: Suki, GTM engineer, 36 out of 100.** Twelve hand-tuned prompts, byte-identical a thousand runs later. Fifteen new reps each needed personalized coaching, at about a day of Suki's time per skill. She was the system.

The other four cases cover measurement, harvest, governance and loops, and you can read them on [agent-grader.com](https://agent-grader.com).

Two failures show up in nearly every home-built setup, whatever it scores:

**Tests rot, and nothing tells you.** Microsoft's re-run triggers (a model change, a knowledge update, a new connector, an incident) all happen to a home-built setup without anyone filing a ticket. A provider swaps the model behind an alias, someone renames a CRM field, a connector updates its API. A DIY rig usually has no record of which prompt version produced which run, so when quality drops there is nowhere to look for what changed.

**Agents read what humans remembered to type.** If reps still update the CRM by hand, every agent built on the CRM inherits their gaps. The meeting-prep agent is only as current as the last note someone typed after a call. That is row 10, and no prompt fixes it.

---

## How Day AI handles each gap

Day AI provides the context graph and agent control plane to quickly and securely upgrade your company brain. It can also run as an AI-native CRM, or on top of Salesforce or HubSpot. For the scorecard above, that means each person on the team has their own agent, with skills that run on a schedule, on an event or on demand, all reading one shared customer memory built from the team's meetings, email and calls. The rest of this section takes the gaps one at a time and names the part of Day AI that closes each.

### Replies change the next run (talk-back, loops)

Every skill run is a chat thread. When a skill emails you its output, a reply to that email goes back into the same thread, and so does a follow-up typed into the thread in Day AI. Feedback on a skill's output is treated as an instruction to the skill: the agent rewrites that skill's prompt in the same turn, says in one sentence what it changed, and every later run uses the change. Dex's AE would reply "stop including the demo pipeline", and tomorrow's briefing would leave it out without Dex touching anything.

Two limits to know. Replies to Slack notifications don't continue the skill's thread today. The Slack message links to the chat, so the reply goes there or to the email. On a managed skill (below), a person who isn't an admin can't change the shared prompt, so the agent tells them what an admin would need to change.

### Each person has their own agent (authorship, compounding)

Agents belong to people. Each person's agent carries its own skills, which that person can write, edit and switch on and off, so the fifteen new reps in Suki's story each shape their own coaching skill without waiting on her. Day AI also keeps a memory for each person, with a section for their standing rules and preferences, which their agent reads in every conversation and updates as they correct it. Underneath, every agent reads the same customer memory: the meetings, emails, calls and CRM records of the whole team, resolved to the same people, companies and deals. The [guide to building a context graph](/guides/build-a-context-graph) covers what that layer is.

Skills can also build on each other. A skill can include up to five other skills from the same agent as extra instructions, attach up to ten Day AI pages or folders as reference material, and an agent can hand a task to another agent to run in the background.

### Admins choose what is managed and what is personal (governance, harvest)

Skills come in three kinds, and the choice is made per skill:

- **Managed.** Only workspace admins create, edit and deploy managed skills. Each agent it's deployed to gets a synced copy that its owner can't edit, and when an admin updates the skill, every synced copy updates with it. Use it for the skills that have to say the same thing for everyone.
- **Template.** Any paid member can publish a template, and people add it to their own agent as an independent copy they can change. Later edits to the template reach only new copies.
- **Personal.** Skills a person writes on their own agent stay theirs.

The choice between everything locked and everything open, Agent Grader's governance case, becomes a per-skill setting. For harvest, a person can add a skill they wrote to the library, and an admin can then deploy it to the team. That promotion is a manual step: Day AI doesn't pick out the best personal versions for you.

### Every prompt change is recorded (feedback visibility)

Skills are versioned. Each skill's Edit History shows every prompt change, who made it, through which agent, how much text it added or removed (a large cut is highlighted), the full text of that version, and a link to the chat where the change was made. To go back, ask the agent to restore an earlier version: it reads that version and writes it back as a new one, so the history keeps both. Edits are also guarded against collisions: an agent's prompt update has to name the version it started from, and a stale edit is rejected and handed the current text, so it can't overwrite someone else's change.

That answers the test in row 8, and it is what makes Microsoft's re-run triggers workable. When quality drops after a change, the change is in the history with a date and an author.

### Every run is recorded (eval)

Each skill keeps its run history. In the skill's settings, its owner sees recent runs with their status (complete, running, failed or queued) and can open any run as its full chat. Ask the agent for a skill's history and it also reports, for each run, whether the output was delivered and by which channel, email or Slack, and why a Slack post was skipped or failed. That is the trajectory and the outcome LangChain says to grade, kept for every run without a logging pipeline of your own. Day AI doesn't score runs for you: the test set and the pass bar from layer one are still yours to write.

### Triggers without plumbing (loops)

A skill runs on demand, on a schedule, or on an event. Schedules run daily, on weekdays or weekly at a time and time zone you pick, and every four hours on the Executive tier. The events a skill can listen for are:

| Event | Fires when |
|---|---|
| Meeting summary | A meeting you recorded is summarized |
| Email summary | One of your emails is summarized (spam, promotions and notifications don't count) |
| Slack activity summary | New Slack activity in your workspace is summarized, about every 30 minutes |
| Note added | A note is created or updated on a record |
| Opportunity created | A new deal is created in the workspace |
| Opportunity stage changed | A deal moves to a different stage, optionally only into the stages you pick |
| Zapier integration | A contact or company is created or updated through Zapier |

Output goes to the person's email, their Slack DM, or the Slack channels the skill is set to post in. Automated skills use slots by tier: two on Turbo, five on Professional and ten on Executive.

Some built-in skills start cautious. Day AI's opportunity automation skill starts in learning mode: it emails what it would have done with your pipeline, and it doesn't create or move deals until you tell it to go.

### Agents act as their person (permissions)

There is no shared service account. A skill runs as the person who owns the agent, so it reads and acts within what that person can see. Each skill also has a Private Data Access setting. On, the skill can use its owner's private data as well as what is shared with the workspace. Off, it sees only what the workspace shares. Skills in the shared library always run with private data off, so a managed skill deployed to every agent never reads any owner's private mail. For a skill that posts summaries into a shared Slack channel, the case in Ravi's story, turn private data access off.

Who can see each email thread is decided when the mail arrives, from each person's sharing rules, so a thread one person keeps private stays out of every other person's agent. Day AI's [guide to building a context graph](/guides/build-a-context-graph) explains how.

### Agents write the CRM back (the keyboard test)

Day AI supports connecting MCP servers. Agents read from any server you connect, and when the server supports writes, an agent writes to it when asked, or a scheduled skill does it on a cadence. Connect the Salesforce or HubSpot MCP and you get two-way sync: a skill can update deal stages, notes and fields after every call, or pull CRM data into the graph on a schedule. Each person connects with their own CRM login, through an OAuth app your company registers, so every write carries that person's identity and permissions. Gong connects with its full call history and transcripts, and Day AI's notetaker writes each recorded meeting into the graph when it ends, so agents read the conversations themselves, along with what a rep typed after them.

That is the fix for row 10. The test still applies: when an agent says it updated a deal, open the record and check.

### The short version

Day AI provides the context graph and agent control plane to quickly and securely upgrade your company brain. Day AI gives each person on a go-to-market team their own agent on one shared customer memory: replying to an agent's output changes how its skill runs next time, every skill is versioned with its own run history, and admins decide which skills are managed for everyone and which each person can make their own.

## Score your own setup

[Agent Grader](https://agent-grader.com) is Day AI's free grader for this question. It asks ten multiple-choice questions, one per row of the scorecard above, each with four descriptions of a setup to pick from. You get a score out of 100, your tier, and a breakdown that shows what good looks like next to each answer. It takes about four minutes, and the score needs no signup. The full written report asks for a work email.

If you built on Claude Code, the [guide to giving Claude Code customer memory](/guides/give-claude-code-customer-memory) covers connecting it to Day AI over MCP. If you're deciding whether to keep building, see [build or buy a customer context layer](/guides/build-vs-buy-customer-context).

## Frequently asked questions

### How do I grade my company's AI agent setup?

Grade two layers. For each agent, build a small test set from real requests, grade the final response, the tools it called and the state it changed, run each case several times, and aim for an 80 to 90% pass rate, with a regression set that passes nearly every time. For the system around the agents, ask ten questions: can people talk back to an agent, who writes the prompts, what is managed and what is personal, do good personal changes spread, are outcomes measured outside the agent, does each agent improve for its person, is there an eval, can you see and undo changes, does it improve while you are away, and does it depend on people typing data in. Agent Grader at agent-grader.com scores the second layer in about four minutes.

### How do I know if my go-to-market AI agents are production-ready?

An agent is ready when it passes a test set built from real requests at a rate you chose in advance, usually 80 to 90% across several runs, passes a short regression set nearly every time, respects each person's permissions, and changes the right records when it says it did. Check the CRM row, not the agent's reply. Then check the system around it: a history of every run and every prompt change, a way for users to correct it, and a plan to re-run the tests after a model change, a knowledge update, a new connector or an incident.

### Is there a rubric or checklist for evaluating GTM AI agents?

Yes. Microsoft's agent evaluation checklist and LangChain's agent evaluation readiness checklist cover testing each agent. For the system around go-to-market agents, use a ten-question scorecard: talk-back, authorship, governance, harvest, measured outcomes, compounding, eval, feedback visibility, loops and the keyboard test. Day AI's free Agent Grader at agent-grader.com asks those ten questions and returns a score out of 100 and a tier.

### How do I evaluate whether my sales agents are doing good work?

Measure the work from outside the agent. Pick outcomes you can count without asking the agent, such as drafts sent, briefs opened or deal stages that moved, and check the records it says it changed. Read 20 to 50 real outputs before building anything, then turn what good looks like into pass or fail tests and run them several times. A weekly report the bot writes about itself is the bot grading its own homework.

### What is Day AI?

Day AI provides the context graph and agent control plane to quickly and securely upgrade your company brain. It gives each person on a go-to-market team their own agent on one shared customer memory: replying to an agent's output changes how its skill runs next time, every skill is versioned with its own run history, and admins decide which skills are managed for everyone and which each person can make their own. It can also run as an AI-native CRM, or on top of Salesforce or HubSpot.

Canonical: https://day.ai/guides/grade-your-ai-agent-setup