Introducing Pheebs

The problem

I’ve spoken to several engineering leaders who want to better understand their team’s AI usage, but don’t know where to start. I’ve also spoken to several engineers who are stuck and don’t know how to level up. There’s a lot of FOMO building up and being vulnerable about your AI skills is not easy.

Adoption metrics like DAU/WAU, and token counts tell you who has access and who uses AI tools. While an ok proxy for AI proficiency and productivity, they say very little about what happens inside a session and how engineers and teams actually use AI.

Which models get used and were they the right fit for the work? Are we spending consciously? Do tests run before a commit? Do outputs get questioned or accepted as is? Do outputs need a lot of repair or do they just get refined? Are repos missing out on scaffolding opportunities (skills, context files)? How’s my team doing in AI competencies like context management, evals, and orchestration?

None of these show up in a usage dashboard. They are the questions that tell you whether a team is working well with AI, and we didn’t have a way to answer them either.

So we built something for ourselves. Every Eversynced engineer works with AI coding agents daily, and we wanted to see how that work actually happened, not just how much of it there was. It started as a small internal observability tool, but the more we used it, the more we were able to improve our own practice. Then we noticed every engineering leader we talked to was asking the same questions we were.

That became Pheebs: an open-source AI telemetry tool for engineering teams. It shows how engineers actually work with AI coding agents, not just whether they use them.

The AI ROI question is the wrong question

Every engineering leader is being asked some version of this question. Are we getting a return on AI? Are engineers more productive? It sounds like one question, but it isn’t.

Engineering productivity has always been very hard to measure, way before AI. Lines of code, PR counts, and cycle time have all been tried and all have known failure modes. AI makes it harder, not easier. Output goes up, but some of that output is rework. Some teams actually say they’re not moving faster.

The bigger issue is that ROI means something different in every organization:

• A startup racing to product-market fit cares about speed to ship.
• A fintech or healthtech cares about meeting compliance requirements at a third of the cost.
• An enterprise sitting on legacy systems cares about migrating them in a fraction of the time.
• A scaling org cares about ramping up new engineers in weeks instead of months.

And answering the AI ROI question depends on how teams actually work with it, not just how much they use it. How proficient they are as individuals and as a group. Proficiency is upstream of every outcome above. A team that has a consistent AI practice, verifies output, challenges it when needed, picks the right model, builds reusable scaffolding, and acts on the right structural gaps will get better results on whichever outcome matters to them.

That’s what Pheebs measures.

How it works

Pheebs runs inside Claude Code, Cursor, and Codex as hooks and captures the shape of each session. It registers lifecycle hooks in the agent’s own config, plus OpenTelemetry export where the tool supports it, and from then on it fires in the background.

When a hook fires, Pheebs extracts lightweight fields: event type, durations, counts, models, trigger types. A session start, a prompt, a tool call, a model switch, a test run, a context compaction, a sub-agent spawn. A prompt becomes a character count and, when the prompt intent classifier is enabled, an intent label like task, debug, or pushback. npm test is recorded as tool_intent: test_run.

It records the shape of the session, never its contents. Pheebs does not store source code, file contents, file paths, prompt text, command strings, or your name and email.

Every event is written to a local JSONL log on the engineer’s machine. A fresh install ships with no backend endpoint, so it stays local until you give it a base URL and a token. From there you choose where it goes: a backend you run yourself, or one we run for you.

Pheebs gives engineers a private view of their own work, separate from the team reports. Engineers can see where they stand and where to act with intention. Team views answer a different question. Is a gap a coaching problem, where one engineer is missing a practice, or a structural problem, where nobody on the team runs tests inside the agent loop?

The AI Proficiency Model

Telemetry on its own is a stream of events. To make sense of the data, Pheebs proposes an AI Proficiency Model, which looks at two things: repertoire and judgement. Proficiency is the combination of both.

1. Repertoire

Repertoire is which harness capabilities show up in an engineer’s work. The model defines 24 practices across six competencies. Each practice is detectable from telemetry.

Competency	What it covers
Models	Models used, effort settings, plan mode, autonomy modes
Artifacts	Reusable config that shapes the agent: skills, sub-agents, slash commands, context files
MCP	Live connections to external systems: tickets, databases, browsers, documentation
Evals	Verification wired into the agent loop: tests, typecheck, lint, build, review passes
Context management	Management of the context window: compaction and the save, resume, clear lifecycle
Orchestration	More than one agent at a time: sub-agents, parallel work, worktrees, hooks, plugins

Each practice sits in one of three states for a given engineer: Unobserved (never showed up), Adopted (showed up at least once), or Recurring (showed up in at least 3 of the last 4 active weeks). A coverage index sums it up per engineer: the share of applicable practices at Recurring.

This makes gaps legible e.g. if nobody has Evals at Recurring that’s an opportunity to implement a verification harness at the team level.

2. Judgement

Judgement is what happens to AI output. Does it get verified, challenged, refined? And was the model used appropriate for the task? This layer is inspired by the Discernment competency from Anthropic’s AI Fluency Index. It is five continuous rates computed from session telemetry.

Signal	What it measures
Verification coverage	Share of AI edits followed by a test run, typecheck, lint, build, or spec check
Pushback rate	How often the engineer challenges AI output instead of accepting it
Refinement-to-repair ratio	Whether follow-up prompts refine intent (iteration) or repair breakage (rework)
Wholesale-accept rate	Sessions with no pushback, no repair, and no verification, weighted by lines changed
Model-fit rate	Share of sessions where the model class matched the size of the work

In Anthropic’s AI Fluency framework, Discernment is the ability to evaluate what AI produces, how it got there, and whether the collaboration is working. It is the skill that separates using AI from using AI well, and it’s the one that more easily weakens the better outputs look.

The Judgement signals make Discernment measurable. Tracked over weeks, per engineer and per team, they tell whether discernment is a habit, a one-off, or missing.

Also, most teams run the largest model for everything. Pheebs sizes the work in every session and compares it with the model used. It then shows how much teams could save by running smaller models where they fit.

Where this leaves us

The AI ROI question will keep coming, and it will keep having a different answer for every organization. The reality is AI isn’t going anywhere.

What organizations can act on is understanding better how their team works with AI, and where the gaps are, so they can be intentional about improving them.

Try it

If you’re an engineering leader who wants to know how your team actually works with AI, or an engineer who wants a private read on your own habits, give it a try and send us some feedback.

• Website: pheebs.ai
• GitHub: github.com/Eversynced/pheebs
• Docs: The AI Proficiency Model