Checks, rules and problems

This page explains the words Geometer uses when it checks an agent, and how they fit together. Guard, what Guard does for each tool and certificates have their own pages.

How the pieces fit

A project holds one or more agents. Each agent has tasks and calls tools. Some operations of those tools change something in the world; those are actions. Rules say what must always be true about the actions. A check runs the tasks while injecting faults into the tool calls and tests the rules. When a rule breaks, you get a problem. You replay the problem, change your code, check the fix and keep the check so your CI re-runs it.

Project

A project is a folder on your computer, such as the root of a repository, that holds your agent and the things around it. Geometer opens it read-only, works out what it contains and keeps everything it learns in its own records folder, not in your project. Opening a project does not run, install or change anything. See Where your records live.

geo onboard shows what Geometer found: agents and their tasks, outside services, the project’s own tests and the rules that apply. Geometer reads files such as package.json, pyproject.toml, go.mod, compose.yaml and Dockerfiles. If it cannot work something out, it asks at most one grouped question with a suggested answer. Your answers are remembered for the project.

If you want Geometer to use hand-written declarations instead of inferring them, you can declare tools, tasks, rules, workloads and conversations in a .geometer/ folder, or keep them in a folder outside the project and pass it with --declarations. Declared files win over inferred ones.

Agent and task

An agent is a program that uses an AI model to decide what to do and then calls tools: for example, a support agent that issues refunds. A task is one thing Geometer asks the agent to do, such as answering a customer who wants a refund. Geometer takes tasks from your declarations or, when you have none, from what it can read in the project, such as examples in your README and your evaluation cases. It also runs the same request twice to see what a repeated request does.

During a check, Geometer answers your agent’s model calls with a stand-in, so a check does not need a model and repeats the same answers. The stand-in shows how your application and the check behave, not how a real model behaves. To try a real model, see Choose a model.

Tool and action

A tool is something your agent calls to do work: a payment service, an email service, a database or a tool server. Each tool has operations, such as payments.refund. An operation that changes something is an action, and actions are what authority is described for.

During a check, outside services are simulated. Nothing real is charged, sent or changed. Geometer records what the agent asked for and what each simulated tool did, and it keeps the tool’s own record of effects apart from the agent’s account of what happened.

Rules

A rule is something that must always be true, for example “a refund is paid at most once per request”. Geometer tests rules during every run.

Rules come from two places. Rules you declare are yours. Suggested rules are read from your code: your database schema, your API description, the way your agent calls outside services and your tests. A model can suggest more, if you choose one; see Choose a model.

geo rules propose lists the suggestions. With geo rules review <ID>, you decide what to do with each one:

• --accept. The rule is checked from now on, and a run that breaks it is shown as a confirmed problem. It fails your CI only if you also add --enable-ci. Until then its problems stay suggestions in CI.
• --reject. The rule is set aside and not suggested again.

Keep a number inside a range

A rule can keep a number that your agent sends to a tool inside a range, for example “a shop price is between 1.00 and 1000.00” or “a spawn is at most 500 units”. Declare it in .geometer/rules.toml:

[[rule]]
id = "price-range"
family = "argument_bound"
tool = "shop"
operation = "set_price"
field = "price"
min = "1.00"
max = "1000.00"

field is the name of the call’s argument; for a number inside another object, join the names with a dot, such as amount.value. Give min, max or both. Both are allowed values themselves, so a price of exactly 1000.00 passes. Write money as text such as "80.00" or as a whole number: Geometer compares them exactly, with no rounding. In a check, the agent may send the number as text or as a number. Guard, in front of the real service, is stricter: it lets the number through only as a JSON number (80.00, not "80.00"), and where both ends of the range are whole numbers, only a whole number, because the service may read text or a fraction another way.

Geometer checks every call the agent makes to that operation. A call outside the range is a problem, and Guard refuses it before it is sent, saying which side of the range it broke. A call with no such argument, or with a value that is not a number, is never counted as fine: the result is “cannot tell”, and Guard refuses the call unless you chose otherwise. If your tool’s own description already gives a minimum or maximum, Geometer suggests this rule from it, and it is checked only after you choose Use this rule.

Checks

A check is one run of the whole routine against a project. geo check starts one. There are three ways Geometer can check a project, and the output says for each whether it will run here and why not when it will not:

• Test the AI agent with things going wrong. Geometer runs your agent’s tasks and makes things fail on purpose. This finds mistakes like a customer charged twice or an email sent five times.
• Send requests to the app’s API. Geometer starts your app and sends it sequences of requests, such as create, read back, change and delete, each time from a clean database. This finds requests that crash the app or leave data in a wrong state.
• Run the project’s own tests. Geometer runs the tests you already have inside a container. This shows whether they pass in a clean environment, and it only finds what your tests already cover.

Two further checks use a model: geo model-upgrade (a model change) and geo stability (constraint stability). See Choose a model and Check your agent.

Every check you start keeps its record, including ones you stopped or that were cut short.

Situations and fault kinds

A fault is a kind of failure at one tool action. geo status lists, for each action, which kinds of fault Geometer can create and which were tried:

The kinds of fault Geometer can create

Kind	What happens
Normal run	Nothing goes wrong. This is the comparison for every other run.
Reply lost	The service did what was asked, but its answer never arrives. Agents often try again, which can repeat a charge or an email.
Request lost	The request never reaches the service, so nothing happens there. The agent sees a closed connection.
Partial effect	A call that creates several parts has only some of them applied before the reply is lost.
Acknowledgement lost	For a queue, the message that says “I handled this” never arrives, so the queue delivers the message again.
Crash and restart	The agent’s process is killed while it waits at a call and started again. The simulated tools and the container’s disk survive; the agent’s memory does not.

Each combination of tool and fault is tried and the rule held, tried and a rule broke, or not tried yet. It counts as tried when it was tried within the check’s budget. What was not tried stays listed as not tried, so what was not checked reads as territory to explore. It is a coarse summary of what was stored and never a percentage. The report also names what Geometer does not model, and the summary says nothing about those. Duplicate delivery of a request, for example, is not a kind of fault.

To try one combination, run geo check --fault <kind> --at <tool>: Geometer explores that kind of fault at that tool and says that is what it did. A crash is offered at most once per run and only where the agent is waiting at a call. A partial effect applies only to a call that declares its parts.

Problems

A problem is a run in which a rule broke, written as what happened to a customer or record. Problems come in two kinds:

• Confirmed: a rule you rely on broke. Fix these first.
• Possible: a suggested rule broke. Decide with geo rules review whether the rule is right.

A problem is described by its consequence first. The full record of a problem (geo ci --json, or explain_problem in your coding assistant) holds the rule with what was expected and observed, each step your agent or app took, the effects recorded by the simulated tools and which of them the tool itself confirmed. It also lists what the run did not cover, so you know the limits of what it shows. Geometer compares the runs where the rule broke with the runs where it held, for example that every broken run had a lost reply on the refund. With fewer than three runs on either side it says it cannot tell which differences matter. It lists what it did not compare, such as the values inside calls and the timing.

Replay

Replay repeats a problem from a clean start, with the same answers as the first time. Because the model’s answers come from a recorded tape, a replay checks your application, not the model. A problem you can repeat exactly is a problem you can fix and check again.

Replay comes in forms, and Geometer says which one it used:

• Exact. The recorded run, including the faults, is repeated from a clean reset. This is what geo replay <finding> does. With --clean-worker, the replay runs on a second clean worker, with a fresh records folder and fresh containers from a copy of the project at the recorded revision. It reports exact, property reproduced, diverged or incompatible.
• Bounded. A search limited to one kind of fault at one tool, under the usual budgets. It is an ordinary exploration, so it can find new problems.
• Checking a fix. The recorded run on your code as it is now, plus a bounded set of nearby cases. See the next section.

A problem whose run cannot be repeated is detected before anything starts, and the command says why in words.

Checking a fix

geo verify <finding> (checking your fix) replays the problem on your changed code, confirms that the run really reached the point where the rule broke, and tries a few nearby cases. It answers reproduced, no_longer_reproduced, inconclusive or incompatible. The counts says what was checked: the recorded run and a number of nearby cases, not every situation. Geometer never calls a problem fixed without that qualification.

Kept checks

geo regression keep <finding> records a problem as a check that can be re-run on every future version of your code. Geometer stores it in its records and writes a reviewable file under .geometer/regressions/ in your project. Commit the file with your fix. geo ci and geo regression run re-execute your kept checks on the current checkout. A kept check that reproduces the problem is a confirmed failure, one that cannot be decided is missing evidence, and one that cannot run is an infrastructure failure.

When a kept check expects a different outcome than the original run, for example a fix that refuses a duplicate request with a 409, you allow that outcome explicitly with geo regression keep --allow node=outcome. That is a decision you make, and Geometer records it.

Next

• Guard (How Geometer watches and stops calls as they happen.)
• What Guard does for each tool (How to read what Guard does with each action.)
• Fix a problem and keep the check (The step-by-step guide.)