Troubleshooting a shop

When something is wrong, Geometer sends you a short alert with what it saw and what to look at first. This page explains each one.

How alerts reach you

Your server sends a heartbeat to Geometer. If something looks wrong, you get an alert by the contact you gave, and a second message when it clears. An alert says only what Geometer saw. A missing heartbeat does not show that anything is fine. Alerts need the deployment key from Deploy on your server.

One example

Backup is late: glow-salon on my-server

The last good nightly snapshot of glow-salon's record was about 40 hours ago. It should be made every night, so last night's has probably failed.
Check first: run "backup now" on the server and read what it says. The usual causes are a full disk, storage keys that have expired, or the storage place being unreachable.

Run it as node bundle/geometer.ts backup now.

The alerts

Alerts and what to check first

Alert	What it means	Check first
Server went quiet	No heartbeat for a while. The server is down, or it cannot reach Geometer	Is the server on and online? Run docker compose ps and start anything that stopped
Client agent missing	A shop was in every heartbeat and is not in the latest	docker compose ps, and whether the shop’s containers run. If you removed the shop on purpose, tell Geometer
Agent stopped / Agent not ready	The runtime is not running, or runs but cannot read its record or has a stopped queue	The runtime’s container and log. For “not ready”, free disk space
Guard stopped	A Guard reported not ready. Calls through it fail or wait	Is its container running? Read its log
Guard kill switch on	Guard blocks calls. Held calls wait	Did you or the owner switch it on? If so, nothing to do. If not, read Guard’s log. start releases it
Failures rising	More tasks failed. A failed task is a customer event the agent gave up on after retries	The runtime’s log. Also check whether WhatsApp, the calendar or Razorpay has an outage
Errors spiking	Many step errors in a short time. Steps retry, so customers may not have noticed	The same
Work piling up	Many tasks waiting for a while. Customers may be waiting for a reply	Is the model or the WhatsApp provider slow? Then the runtime’s log
Agent looks stuck	Tasks are waiting but no step has finished	The runtime’s log. If nothing moves, restart the runtime container. Waiting tasks are kept and run again
Backup is late	The last good nightly snapshot is older than 36 hours	backup now. Usual causes: a full disk, expired storage keys, storage unreachable
Restore test failed	The weekly restore did not match the live record. A restore might not work when needed	status shows the reason. Fix it, then backup drill
Restore test overdue	The weekly restore has not run for more than 8 days	Is the deployer running (apply --watch)? Then run backup drill once

A deploy is refused

A refused deploy starts nothing and names the rule. Read the line that starts with the rule in square brackets. Typical causes:

• A secret is missing from secrets-inbox/<shop>/. Run check <shop>: it lists everything wrong at once.
• The Guard image is not loaded on the server, or it is for the wrong CPU.
• A limit could not be shown to bite on the running Guard. Fix the shop’s settings and deploy again.
• A rollback or an older --to is refused when the newer version changed how data is saved. See Versions per client.

In the panel

• “Not confirmed yet” instead of “live”: the deployer did not hear from the shop’s runtime and all three Guards in the last minute. Check that apply --watch runs.
• A Razorpay key refused by name: it is not the key ID the connection check confirmed, or it is for the other mode (live or practice).
• A list that has not changed after you pressed a button: the list is read from a record exported every quarter of an hour.
• Hindi, Malayalam or Kannada looks odd: those words are machine-written and wait for a native reader.

Owner approval problems

The approvals relay tells you in its log when it could not ask the owner. Typical causes: the runtime shows no booking for that refund, the runtime could not be asked, or Guard did not accept the owner’s signed answer. For the last one, check the approver id and secret. If many wrong codes arrive in one hour, the relay ignores codes for the rest of the hour. If nobody at the shop did that, someone may be guessing.

Still stuck

Write to support. A person answers. The address is hello@geometer.dev.