Beyond Vibe coding - Developing Production Software with AI Agents

0. The core mental model

Treat the agent as a strong but flawed mid-level engineer: unlimited patience, no fatigue, fearless about tooling, and with four persistent tendencies that will bite you:

1. Drifts to plausible-but-wrong. It will quote a price, a version, or an API shape from memory and sound confident while being wrong.
2. Overbuilds. Given freedom, it will reach for Kubernetes, multi-region, and GPU clusters before there's a trigger.
3. Leaks abstractions. Because it "sees" the whole stack, it will happily surface internal IDs and lower-level concepts into product surfaces.
4. Skips verification. It will trust a green test suite or a "success" return code and declare done.

Your job as the human is therefore not to write code; it's to set guardrails, define done, and hold the agent to evidence rather than vibes.

1. The operating rhythm

Checkpoint-first, resume-later

Keep a compact state vector, not a history log. Ours is one file: current state, locked invariants, risks, and a single authoritative NEXT ACTION. The hardest rule to honor: if git, an ADR, a test, or the code can tell you it does not belong in the checkpoint.

A checkpoint that becomes a narrative is already failing; you'll stop reading it.

Smallest end-to-end slice first, then expand

Ship a thin vertical slice and verify it live before adding breadth. We didn't build "the platform"; we built auth, then one working deploy loop (GitHub → webhook → worker → Coolify → HTTPS URL), then billing. Each slice was live-verified before the next began.

Progressive scaling, not speculative architecture

Stage 1 (1–50 users) is one control node + one app node. Kubernetes/GPU enter only when a real density or HA trigger fires. The agent will want to build the Stage 4 stack on day one; your job is to say "not yet" and mean it.

Decision records, not vibes

Every significant choice gets an ADR:

Decision → Context → Options → Recommendation & trade-offs → Risks → Migration path.

Locked decisions are re-litigated only with evidence, never "I think this is better now."

2. Guardrail: the abstraction firewall

The single most important architectural rule of the whole build: an implementation detail must never leak into a product surface. Coolify UUIDs, Docker container IDs, and Traefik concepts; none of it appears in public APIs, CLIs, or error messages. Coolify is an implementation engine behind a ComputeProvider seam; the customer never sees it.

Why this matters with AI agents specifically: the agent knows the whole stack, so it will cheerfully put a Coolify UUID in a JSON response or an error string, because to the agent that is the value.

Concrete countermeasures:

• Whitelist serializers (toPublicApplication, toPublicDeployment) so a field can't leak even by accident, and a test that greps for the forbidden field's absence.
• Translate, don't surface. Raw infra errors become actionable DX copy (exit 137 → "your app exceeded its 512 MB memory limit"), never raw Docker output.
• Provider IDs never cross the boundary. The internal identifier and the public identifier are different objects, always.

3. Guardrail: verify against reality, not memory

This is the one that produces the most "how did that happen?" bugs.

• We got burned quoting stale Hetzner pricing and reading the main branch of Nixpacks instead of the version Coolify actually shipped. Both times the agent answered confidently from memory/training data.
• Mock third-party APIs by copying field names from a live response, never by inventing the shape. A test will pass forever against a wrong invented schema and give you false confidence, we shipped a bug where the deploy poll never matched because the mock invented a UUID while the real API returned deployment_uuid.
• After any state-changing write, read back the exact target before claiming success. A "successful" tool call is not a successful task.
• Declared totals are hard assertions. If you say "16 records," you re-fetch and count. Never finalize on "go with what I have."

The takeaway: the agent's memory is a suggestion engine, not a source of truth. Everything load-bearing gets checked against the live system.

4. Guardrail: a real Definition of Done

"Done" was never "code compiles."

It was:

Code complete · tests passed · error handling mapped · security verified · telemetry wired · docs/ADR updated · rollback strategy confirmed.

Two refinements that matter:

• Never trust a green suite alone for an infra change. Also test the denial paths—unauthenticated, wrong-role, cross-tenant—and assert that internal fields never serialize.
• Reversibility is part of the spec. Every critical mutation supports retry/rollback/redeploy. The old commit is the rollback point.

4.1 Guardrail: architecture validation is a separate gate.

Implementation is not approval.

For any significant architectural change, maintain two separate artifacts:

• Architectural Decisions—what we agreed to build and why.
• Acceptance Criteria—how we will prove that the decisions were implemented correctly.

In our DX platform, the approved architecture contained 9 architectural decisions and 86 acceptance criteria. Those criteria became the contract between architecture and implementation.

The implementation was not considered complete when the code was written, tests passed, or the agent reported success. It was complete only when an independent validation pass demonstrated that all 86 criteria were satisfied.

The validation loop

Enlarge

The validator must not assume that implementation claims are correct.

For every acceptance criterion, record:

• Criterion ID
• Architectural decision
• Expected behavior
• Evidence required
• Evidence found
• Status: PASS / FAIL / PARTIAL / NOT TESTABLE
• Relevant code, test, configuration, or live-system evidence
• Remaining gap, if any

Validation is independent from implementation.

The agent that implements a feature should not be the sole authority that declares it complete.

A separate validation pass should attempt to falsify the implementation:

"Show me evidence that this criterion is satisfied."

not:

"Does the implementation look correct?"

The validator should inspect the actual repository, run the relevant tests, inspect configuration, and perform live verification where required.

Failed validation is expected.

Our first two final validation attempts failed to satisfy all acceptance criteria.

That was not evidence that the acceptance criteria were too strict.

It demonstrated why the validation gate exists.

A validation failure means:

1. The criterion remains open.
2. The implementation agent receives the specific gap.
3. The implementation is corrected.
4. The same criterion is re-validated.
5. Previously passing criteria must not silently regress.
6. Validation continues until every criterion is either PASS or explicitly waived by the human owner.

Never convert a FAIL into PASS because the implementation is "close enough."

No partial completion claims

Do not report:

"86 acceptance criteria reviewed; most are complete."

Instead report:

84/86 PASS — 2 FAIL

The system is not done until the remaining criteria are resolved or explicitly accepted as exceptions by the human decision-maker.

Preserve validation history

Keep the validation reports.

For example:

validation/

validation-01.md

validation-02.md

validation-03.md

This creates an auditable record of:

• What failed?
• Why did it fail?
• What was changed?
• What subsequently passed?
• Whether regressions occurred?

The validation history is part of the project's engineering knowledge.

Validation is a release gate.

For significant architectural work:

No complete validation → no architectural sign-off.

Tests are evidence.

Acceptance criteria are the contract.

Independent validation is the gate.

Human approval remains the final authority for security-sensitive, irreversible, strategic, legal, or financial decisions.

5. Guardrail: irreversible actions need a consent boundary.

• Security-sensitive ops are never silent. Placing a private key on a server, changing auth, rotating credentials — the agent must get explicit operator sign-off and must not bury the action in a wall of other work.
• Destructive commands go through a consent gate (docker rm -f, rm in a root path). Two rules that keep this sane: silence is not consent (a timeout isn't approval), and a timeout is not a denial (re-running on explicit "go" is legitimate).
• The half-deployed state must be safe by construction. A new image was built, but the old container is still running. Additive migration was applied, but the old code is live — that in-between state should never corrupt anything. Design the deploy so the "worst" intermediate state is a no-op for the old version.

6. Guardrail: secrets are the sharpest edge.

The most surprising failure class of the whole project, by volume:

• Secrets are write-only. The UI submits deltas, never full sets — a value-blind client must never be able to wipe a secret it can't read back.
• Never type a secret inline. Tool-level display redaction can turn the literal *** into the actual file. Write to a gitignored file, reference it, and verify by length or grep-count, never by reading it back (the read is redacted too, so you can't trust your eyes).
• Specific corruption triggers we hit: $(...) command substitution, ${VAR} interpolation, and field names like apiKey / Bearer / <prefix>_key. Workarounds that worked: terminal heredocs (display-only redaction) or constructing the token at runtime so the trigger string never appears in the source.

7. Guardrail: reproducibility & the "stale tree" problem

• The deploy node's source tree is a file-by-file sync, not a clean checkout. A change that was "committed but not deployed" leaves other files stale — and a later partial deploy will fail the build on a file you never touched (we shipped exactly this: a freshly-synced barrel re-exported a symbol its stale sibling didn't have).
• Rollback = re-sync the old commit's files + rebuild. No git on the node, so the old commit is the rollback point, always know which one it is.

8. Where AI agents genuinely shine (lean in)

• Mechanical multi-step work — running the same build → test → deploy → verify loop dozens of times without fatigue.
• Exhaustive "what could go wrong" enumeration. The agent's paranoia is an asset: it was the one that surfaced the double-charge idempotency guard, the webhook-redelivery status-regression case, and the OOM exit-code mapping.
• Translating between layers — raw third-party API into a typed internal interface, with tests. This is the single highest-leverage delegation.
• Tests and documentation — writing them after the agent itself has done the work, so they capture the real shape, not the imagined one.

9. Where you must NOT delegate

• Strategic, legal, and financial decisions. Pricing, provider choice, compliance claims — the human retains final authority, always.
• Anything requiring user interaction or external accounts — OAuth grants, KYC/merchant onboarding, GitHub installation authorization. An agent cannot and should not drive these.
• Final sign-off on irreversible or security-sensitive changes.

10. The daily loop (concrete steps)

1. Resume: read checkpoint → load the skill (procedures + gotchas) → read the relevant ADRs → inspect the repo → execute the single NEXT ACTION.
2. Work tight: build → typecheck/test → deploy → verify live → iterate.
3. Finish: update the checkpoint (state + invariants + next action), record reusable lessons in the skill (not memory), add an ADR if a decision was made.
4. Never claim done on a subset. "Done" means every acceptance criterion has been independently validated, not merely implemented or tested.

11. Failure-mode checklist (tape this to the monitor)

When reviewing the agent's work, scan for:

• It invented an API/price/version from memory instead of a live source.
• It over-built (K8s/multi-region/GPU) before a real trigger.
• It leaked an internal ID or raw error into a public surface.
• It trusted a green suite and skipped live verification.
• It did a security-sensitive thing silently.
• Its secret-handling corrupted a value to ***.
• It synced its own changed files but left the tree stale on a sibling.
• It claimed "done" on a subset of the acceptance criteria.
• It claimed a criterion was satisfied without concrete evidence.
• It treated implementation evidence as validation evidence.
• It ignored a FAIL because the implementation was "close enough."
• A later change caused a previously passing criterion to regress.
• Validation was performed without an independent challenge.
• Validation results were not preserved as an auditable artifact.

12. The architecture-to-validation lifecycle

The complete process we learned from the project is:

Enlarge

The key principle is:

The agent's ability to implement software is not the same as its ability to determine that the software is correct.

The architecture defines what should exist.

The acceptance criteria define what "correct" means.

Implementation produces the system.

Validation produces evidence.

Human sign-off produces approval.

Every one of these rules exists because it was violated at least once and cost real debugging time. The agent got better each session — but the guardrails are what made that compounding improvement safe, not luck.