Troubleshooting

This guide covers the most common issues you will encounter when operating a motebit relay and CLI, how to read structured logs, and how to resolve each error class.

Reading relay logs

The relay emits structured JSON to stdout (info/warn/debug) and stderr (error). Every line is a single JSON object:

{"ts":"2026-03-20T14:32:01.123Z","level":"info","msg":"receipt.verified","service":"relay","correlationId":"a1b2c3d4","status":"completed","motebitId":"01234..."}

Key fields

Filtering logs

On Fly.io, pipe through jq for readable output:

# Follow all logs, pretty-printed
fly logs | jq '.'

# Trace a single request by correlation ID
fly logs | jq 'select(.correlationId == "a1b2c3d4")'

# Show only warnings and errors
fly logs | jq 'select(.level == "warn" or .level == "error")'

# Show only settlement events
fly logs | jq 'select(.msg | startswith("settlement."))'

# Show federation activity
fly logs | jq 'select(.msg | startswith("federation."))'

Set the minimum log level with the LOG_LEVEL environment variable (default: info). Set to debug for maximum verbosity during investigation.

fly secrets set LOG_LEVEL=debug

Common errors

HTTP 400 — Bad Request

The relay returns 400 for malformed or missing fields. Common causes:

• Missing motebit_id on registration or task submission
• Missing prompt or empty prompt on task submit
• Invalid endpoint_url — must be a valid URL with protocol
• Invalid key succession — signatures don't verify, or the predecessor key doesn't match the registered key
• relay_task_id mismatch — the receipt's relay_task_id doesn't match the task route, meaning the receipt is bound to a different economic contract

What to check: Read the message field in the JSON error response. It tells you exactly which field is missing or invalid.

HTTP 401 — Unauthorized

Authentication failed before the request reached business logic.

Fix: Regenerate the signed token. Tokens expire after 5 minutes. Ensure the device's Ed25519 private key matches the public key registered with the relay. Check that the system clock is accurate — clock skew beyond 5 minutes will cause token expiry.

HTTP 402 — Payment Required

The target agent requires payment and the delegator has insufficient funds.

{"error":"Insufficient funds — deposit to virtual account or pay via x402"}

Causes:

• The target agent has a pay_to_address in its service listing
• The delegator's virtual account balance is too low to cover the estimated cost
• No x402 payment header was provided

Fix:

1. Check the delegator's balance: GET /api/v1/agents/:motebitId/listing to see the price, then check the account balance
2. Fund the account via the deposit endpoint or Stripe checkout
3. Alternatively, delegate to an agent that doesn't require payment

Related logs:

• account.debit — shows the debit attempt and balance
• settlement.created — confirms successful settlement after task completion

HTTP 403 — Forbidden

Authorization succeeded but the caller lacks permission for the specific action.

Related logs:

• federation.receipt_signature_invalid — Ed25519 signature on a federated receipt doesn't verify
• federation.receipt_key_missing — no public key found for the executing agent on a federated receipt

HTTP 404 — Not Found

The requested resource doesn't exist.

For expired tasks: The relay keeps tasks in memory with a TTL. If the executing agent takes too long, the task entry expires and the result endpoint returns 404. Increase the task timeout or investigate why the agent is slow.

HTTP 409 — Conflict

A resource already exists or is in an unexpected state.

Federation duplicate task: When a peer relay receives a forwarded task with a task_id it has already seen, it returns 409. This is intentional idempotency — the origin relay should not retry.

HTTP 410 — Gone

Pairing code expired

Pairing codes have a short TTL. Generate a new one.

HTTP 429 — Rate Limited

The relay enforces 5-tier sliding-window rate limits per IP address:

Federation endpoints have a separate per-peer limit of 30 requests/min per relay ID.

WebSocket connections have a per-connection limit of 100 messages per 10 seconds.

Fix: Back off exponentially. If you're hitting rate limits in normal operation, check for:

• Polling loops that are too aggressive (use WebSocket instead of HTTP polling)
• Multiple processes sharing the same IP
• A misconfigured sync interval (default is 30 seconds — shorter intervals may trigger limits)

HTTP 501 — Not Implemented

Stripe is not configured on this relay

The deposit or webhook endpoint was called but STRIPE_SECRET_KEY is not set. Configure Stripe or use a different funding method.

HTTP 503 — Service Unavailable

Maximum peer limit reached

The relay has reached its federation peer limit (default: 50). Increase MOTEBIT_FEDERATION_MAX_PEERS or remove inactive peers.

Receipt verification failures

Receipt verification happens at multiple points: task result ingestion, federation forwarding, and multi-hop settlement. Each has distinct log signatures.

receipt.verification_failed

The Ed25519 signature on the receipt doesn't match any known public key for the executing agent.

{"level":"error","msg":"receipt.verification_failed","reason":"invalid Ed25519 signature","correlationId":"task-123"}

Causes:

• The agent rotated its key but didn't submit a succession record
• The agent is signing with a different keypair than the one registered
• The receipt was tampered with in transit

Fix: Verify the agent's registered public key matches the signing key. If the key was rotated, ensure the succession record was submitted and accepted by the relay.

receipt.prompt_hash_mismatch

The receipt includes a prompt_hash that doesn't match the task's prompt, and no relay_task_id binding is present.

{"level":"warn","msg":"receipt.prompt_hash_mismatch","reason":"receipt prompt_hash does not match task prompt — relay_task_id binding not present"}

This is a hard rejection (HTTP 400). The relay_task_id field is required for cryptographic task binding — there is no legacy fallback. The fix is to upgrade the agent to include relay_task_id in signed receipts (motebit runtime 0.4.0+).

delegation_receipt.signature_invalid

A nested delegation receipt (in a multi-hop chain) failed Ed25519 verification.

{"level":"warn","msg":"delegation_receipt.signature_invalid","parentAgent":"alice","delegatedAgent":"bob"}

The invalid sub-receipt is skipped — it won't be settled or produce trust edges. Check the sub-agent's key registration.

Settlement issues

settlement.created

Normal operation — a settlement was successfully recorded.

{"level":"info","msg":"settlement.created","correlationId":"task-123","net":0.0045,"fee":0.0005}

settlement.retry.refunded

Settlement forwarding to a federation peer failed after 5 retry attempts (exponential backoff: 30s, 2min, 8min, 32min, 2h). The locked funds were automatically refunded to the delegator.

{"level":"info","msg":"settlement.retry.refunded","taskId":"task-123","amount":0.005,"delegator":"alice-id"}

Fix: Check whether the peer relay is reachable. The circuit breaker may have already suspended the peer.

multihop.settlement.task_not_found

A sub-task referenced in a delegation receipt wasn't found in the relay's task queue. The sub-task may have expired or the relay_task_id doesn't match.

{"level":"warn","msg":"multihop.settlement.task_not_found","subTaskId":"sub-task-456","subAgent":"bob-id"}

multihop.settlement.sig_invalid

A sub-receipt in the multi-hop chain failed signature verification. Settlement for this hop is skipped.

multihop.settlement.depth_limit

The delegation chain exceeds 10 hops. This may indicate circular delegation. Settlement stops at the depth limit.

{"level":"warn","msg":"multihop.settlement.depth_limit","depth":11,"subAgent":"agent-id"}

Federation issues

x402.facilitator.init_failed

The x402 payment facilitator is unreachable at startup. Virtual account deposits and withdrawals still work. Agents with pay_to_address listings will return 402 to callers who don't have sufficient virtual balance.

{"level":"warn","msg":"x402.facilitator.init_failed","facilitator":"https://x402.org/facilitator"}

Fix: Check network connectivity to the facilitator URL. This is expected in test environments.

federation.heartbeat.missed

A peer relay didn't respond to the heartbeat ping. The relay tracks consecutive misses:

• 1-2 missed: Warning logged, peer stays active
• 3 missed: Peer auto-suspended — excluded from routing
• 5 missed: Peer removed

{"level":"warn","msg":"federation.heartbeat.missed","peerId":"peer-relay-id","missed":2}

federation.peer.suspended

A peer was suspended due to missed heartbeats or the forward-path circuit breaker (failure rate exceeds 50% over 6+ samples with 3+ consecutive failures). Agents on suspended peers are excluded from task routing.

The peer will be restored to active on the next successful heartbeat.

federation.receipt_signature_invalid

The relay received a federated receipt but the executing agent's Ed25519 signature didn't verify. The receipt is rejected — no settlement, no trust update, no credential issuance.

Execution ledger debugging

Use the CLI to inspect execution ledgers:

# View the execution timeline for a goal
motebit ledger <goalId>

The ledger shows each step in the goal's execution, including:

• routing_choice — explains why a specific agent was selected (semiring scores, provenance)
• content_hash — SHA-256 hash covering the full timeline including routing provenance
• Ed25519 signature — verifies the manifest hasn't been tampered with

If routing decisions look wrong, check the agent trust records (/agents info <motebitId> in the CLI or the Agent Trust tab in the admin dashboard) and the trust closure (GET /api/v1/agents/:motebitId/trust-closure).

Key rotation recovery

If an agent's private key is compromised or needs rotation:

# Rotate the key — creates a signed succession record
motebit rotate --reason "scheduled rotation"

This generates a new Ed25519 keypair and creates a succession record where:

1. The old key signs a tombstone declaring the new key as successor
2. The new key signs an acknowledgment
3. Both signatures are required — non-repudiation from the old key, acknowledgment from the new key

The relay requires a valid succession record when re-registering with a changed public key. Without it, registration returns HTTP 409:

Public key changed — succession record required

If you lost the old key: You cannot create a succession record without both keys. You will need to create a new identity (npm create motebit) and re-establish trust from scratch. This is by design — identity continuity requires cryptographic proof.

Verify the succession chain:

# Verify an identity file including succession records
motebit verify <path-to-motebit.md>

CLI error output

The CLI displays errors inline during streaming:

• [tool] name...done — tool executed successfully
• [warning] suspicious content in tool_name — memory injection defense triggered
• [delegating to relay] delegate_to_agent...done — delegation completed
• [approval] tool_name(args) Allow? (y/n) — tool requires operator approval

Fatal errors during bootstrap (missing identity, unreachable relay, invalid config) are printed to stderr and cause a non-zero exit code. Check ~/.motebit/config.json for configuration issues and ensure the identity keypair exists.

Environment variables reference