Some scope before anything else, because “the AWS network” is a phrase that invites overclaiming. Amazon’s networking organization is enormous, and it splits across very different problem domains: the border, the backbone, the edge, the data center. I worked in the data center part of it, specifically on data center fabrics. AIDN and SIDR were one set of systems among many operating that vast network, and other organizations ran their own control and management systems for the parts I never touched. What AIDN and SIDR became, together, was the management and control plane for the largest network fabric in Amazon’s history. Big claim, bounded scope. The bounding matters for everything that follows, because the lessons below come from one domain, not from the whole network.
Credit where it belongs before I go further. AIDN was the vision of Stephen Callaghan, one of our senior Principal Engineers, and SIDR was the vision of another, JR Rivers. My job was to build and lead the teams that turned their combined vision into production systems operating Amazon’s data center fabrics at scale, and I had exceptional engineers to do it with. And to be clear about what “leading” meant: I wasn’t just running the program. I spent much of those three years in the technical work itself, in design reviews, architecture debates, and operational deep dives alongside the PEs and the teams. That’s where most of the lessons below come from.
I didn’t think of it in these terms at the time, but this is exactly what everyone building with AI is trying to do right now: capture expert judgment and put it inside a system that can execute it faster, more consistently, and at greater scale than the experts themselves. The fabric got there without LLMs. The lessons from that journey map almost one-to-one onto the problems AI teams are hitting today. Most of those teams are earlier in the journey than they think.
By 2021, data center fabric operations had a problem that no amount of hardware could fix. The systems used to run the fabrics had grown organically over more than a decade, alongside the separate systems other parts of the network relied on, with little shared structure between them. Critical operational knowledge was spread across multiple systems, none of which was authoritative. The same information was duplicated in different places, in different formats, with no guarantee the copies agreed. And the whole apparatus was reaching its limits. Not in packets per second, but in operations per second. The constraint on growth was how fast humans could safely decide things: how to route around a failure, when to shift traffic off a device, whether a change was safe to push.
That’s why AIDN existed. Stephen had the vision, and I was brought in to build the team, greenfield, with a mandate the job postings later described accurately: completely re-imagine how one of the largest networks on the planet would be operated for the next decade. In practice, that meant the fabric layer. Ambitious enough.
But AWS had already tried to automate this judgment once. The way that first attempt fell short taught me more than the eventual solution did.
Before AIDN, fabric engineering judgment had been captured, in infrastructure-as-code. Routing policy decisions, for example, lived in what amounted to a lightweight wrapper around BGP policy, expressed as a large collection of YAML files in a git repo.
On paper, this looks like automation. The expert’s decision is written down, version-controlled, reviewable, repeatable. In practice, it was reaching the end of its life. The YAML was surprisingly hard to maintain. It was hard to test; you couldn’t easily answer “what does the network do if I merge this?” It was hard to keep well-structured and normalized, and the repo accumulated near-duplicates and special cases the way all shared config repos do. Every change still required an expert to hold the whole system in their head. The reason is worth stating plainly: those files captured code and configuration, not intent. They were the output of expert judgment, rendered in a machine-applicable format. The judgment itself, the reasoning that produced the configuration, never made it into the artifact.
And to be clear, this wasn’t a naive setup. There were real safety systems wrapped around those repos: CI/CD pipelines, validation checks, deployment automation. The problem is that guardrails built beside the artifact scale no better than the artifact itself. The pipelines took a long time to run, which meant engineers waited hours to learn whether a change was safe. They grew fragile as the config sprawled, breaking in ways that had nothing to do with the change under test. And bugs still crept through and got deployed, because the checks could only inspect the configuration, not the intent behind it. You can’t verify that a change does what the expert meant when the system has no representation of what the expert meant. The safety net had the same operational-scalability ceiling as the YAML it guarded.
That’s the distinction that matters. A YAML file that says “prefer this path” captures a decision. It doesn’t capture why, under what conditions the decision holds, or how to verify it’s still correct. So every time conditions changed, a human had to re-derive the reasoning and update the artifact. We had automated the typing, not the thinking. And it was hitting the same wall as manual operations. It just hit it a few years later.
None of this is a knock on AWS, and none of it is unique to Amazon. Every large infrastructure organization I’ve worked in or with has a version of this repo, and most of them are in far worse shape. AWS almost certainly operates networks better than anyone else on the planet. That’s the point worth sitting with. This ceiling is not a symptom of doing infrastructure badly. It is what you hit after you have done everything else right, and it shows up first at the organizations furthest along. If the best network operators in the world had to build their way past it, the odds that your prompt repo escapes it are not good.
If you’re building with LLMs today, you should recognize this era, because you’re probably living in it. A prompt is a YAML file. It is the rendered output of expert judgment, not the judgment: hard to test, hard to keep normalized, silently accumulating special cases, and still dependent on the original expert to maintain. Prompt libraries are the config repos of AI. They’re a real step forward, and they will hit exactly the same wall. So will the CI bolted on beside them, if what it checks is the artifact rather than the intent: slow eval suites that gate every change, grow brittle as the prompt sprawls, and still let regressions through because they don’t encode what the expert actually wanted.
AIDN’s answer, visible now in AWS’s re:Invent NET401 talk and the intent-driven network management patent family, was to stop encoding decisions and start encoding intent: the operational outcome the expert wants, declared formally, without specifying how to achieve it. “Traffic between hosts in this zone never leaves this zone” is an intent. The thousands of device configurations that enforce it are the system’s problem, not the engineer’s.
Three properties made this work, and I’d argue all three are non-negotiable for any system that automates expert judgment, network or AI.
It was authoritative. Every applied intent went into a transactional journal: a single source of truth recording what was intended, when it was approved, and when the network converged to it. This is precisely what the pre-AIDN world lacked. Duplicated, non-authoritative state is where automated judgment goes to die, because when two systems disagree about what should be true, a human gets pulled back in to arbitrate. At that point your automation is a suggestion.
It was verified before it acted. Proposed intents didn’t go straight to the network. They passed admission control: simulation-based functional verification, conflict detection against every intent already in the journal, change-control and SLA checks. Judgment was tested the way code is tested. Before deployment, automatically, every time.
It was verified after it acted, forever. The system continuously compared observed network behavior against the journal’s intended behavior. When they diverged, it could roll back the transaction, adjust, or escalate to a human. The loop is closed. Nobody assumes the automated judgment stays correct; the system checks.
Notice what’s absent from that list: intelligence. AIDN’s translation layer, the compilers turning intents into device-native artifacts, was sophisticated, but it wasn’t the source of trust. The trust came from the journal, the admission checks, and the drift correction. That’s the part AI builders keep getting backwards. Teams pour effort into making the model smarter and treat evaluation as an afterthought, when the AWS lesson is that the verification machinery is the product. The intelligence is replaceable.
SIDR, the system JR envisioned and whose team I later inherited, solved a different judgment problem: not “what should the network do?” but “who decides, and how fast?”
SIDR is the routing protocol behind AWS’s 10p10u fabric, the network that gives massive Trainium UltraClusters ten petabytes of capacity at under ten microseconds of latency. At that scale, during AI training, every server talks to every other server simultaneously, and a link failure has to be routed around in well under a second. No human is in that loop. No central system can even be in that loop; the speed of light won’t allow it.
Peter DeSantis walked through SIDR’s architecture in his re:Invent 2024 Monday Night Live keynote. A central planner with global visibility distills the network’s structure into directives and pushes them to every switch, and each switch then makes its own local decisions when a failure occurs. Expensive, slow, context-rich judgment happens centrally and rarely. Cheap, fast, local judgment happens at the edge, constantly, within guardrails the planner computed. The result is failure response in under one second, roughly ten times faster than the alternatives.
That decomposition, plan centrally, execute locally, constrain the executor, is the right mental model for agentic AI systems. The mistake I see constantly is putting all the judgment in one place: either a single monolithic agent making every decision (slow, expensive, a single point of failure) or fully autonomous sub-agents with no shared plan (fast, and incoherent). The systems that work distill global intent into local guardrails, then let execution run free inside them.
In 2023 we made a call: AIDN and SIDR were too tightly coupled to live in separate organizations, and SIDR was solving hard problems that deserved a dedicated leadership focus it couldn’t get while the loop was split across org boundaries. We merged them, along with several other software controller teams, into a single group: Network Fabric Controllers. Together, those systems became the management and control plane for the largest network fabric in Amazon’s history.
The technical logic was simple. An intent system that can’t control routing, and a routing system that doesn’t know intent, are each half of a control loop. But the deeper lesson was organizational. When you automate expert judgment, the unit of ownership has to be the entire loop: intent, translation, execution, observation, correction. Split the loop across team boundaries and every drift incident becomes a cross-team negotiation, which means the loop effectively isn’t closed at all.
AI teams are recreating this mistake today. One team owns prompts, another owns the model, a third owns evals, and nobody owns drift in production. If your eval team and your agent team file tickets at each other, you don’t have a closed loop. You have YAML with extra steps.
Automating expert judgment is not a new problem, and it isn’t primarily a model-capability problem. AWS automated the judgment of elite network engineers using deterministic software, and the hard-won structure was: make the judgment explicit and declarative; store it in one authoritative, transactional place; verify before acting; watch for drift forever; put slow judgment in the center and fast judgment at the edge; and organize teams around whole control loops.
LLMs change the economics of the translation layer. The part that turns intent into action is now shockingly general. What they don’t change is everything around it. Most AI systems in production today are at the YAML stage: captured judgment, no admission control, no journal, no drift detection, open loops everywhere. The teams that internalize the rest of the structure first are going to operate their AI the way AWS operates its fabrics: declaratively, verifiably, and at a scale where the experts’ judgment outlives the experts’ availability.
That’s the work I’m doing now. The network was just the first fabric.
]]>David Lapsley, Ph.D., is Founder and CEO of ShipKodeAI. Over a 26-year career, he has spent 15 years building infrastructure and platform products. Previously Director of Network Fabric Controllers at AWS, the team that powered the largest network fabric in Amazon’s history, and Director at Cisco (DNA Center Maglev Platform, $1B run rate). He writes about AI infrastructure, AI-accelerated SDLC, and the gap between POC and production.
Greg Kroah-Hartman, the Linux kernel maintainer, confirmed the shift publicly about a month ago. The change was not gradual. It was a step function. And the reports are not AI slop anymore. They are real, high-quality vulnerability reports from AI agents that can read code, understand control flow, trace data dependencies, and identify exploitable patterns at a scale and speed that no human reviewer can match.
Willy Tarreau, maintainer of HAProxy, noted that duplicate reports are now common. Multiple AI agents, running independently against the same codebase, finding the same vulnerability at the same time. This is not theoretical. This is happening right now, every day, on one of the most security-critical codebases in the world.
Your attack surface is being scanned 24/7 by autonomous agents that do not sleep, do not take vacation, and do not bill by the hour. They are working for security researchers, for competitors, for nation-state actors, and for anyone with a GPU and a public GitHub repository. The discovery side of the equation has already shifted to AI speed. The patch side has not.
Most engineering organizations are built for human-speed vulnerability management. A report comes in. A triage happens. A developer is assigned. A fix is written, reviewed, tested, and deployed. This process takes days or weeks. AI vulnerability discovery takes minutes.
The gap is widening daily.
Thomas Ptacek, a well-known security researcher, wrote an essay that has been circulating widely in the security community. His argument is straightforward: vulnerability research is “cooked.” Not in the sense that it is over—that it has fundamentally changed to the point where the human’s role in the discovery process is becoming optional.
The Linux kernel is the most scrutinized piece of software on the planet. If AI agents can find 5-10 new vulnerabilities per day in the Linux kernel, they can find vulnerabilities in your codebase too. The only question is whether they are looking at it.
I have spent my career in network security. I led the team that built Corelight’s network detection platform using Zeek, and I have two US patents in real-time network attack detection. I have seen vulnerability management from both sides: as a defender building detection systems, and as a leader responsible for shipping secure products.
The most expensive vulnerability is not the one that gets exploited. It is the one you did not know existed. The shift from human-speed to AI-speed discovery changes the risk calculus fundamentally. Every organization that ships software now has to assume their code is being actively scanned by AI agents that will find things before humans do. The question is not whether there are vulnerabilities in your codebase. The question is whether someone else finds them before you patch them.
The traditional model of vulnerability management is designed for human-scale discovery. That model is obsolete. The new model requires three things that most organizations do not have yet.
First, AI-native threat modeling. Security teams need to be using the same AI tools that attackers use to find vulnerabilities. This is not optional. If your red team is still manually auditing code while attackers use autonomous agents, you are optimizing for the wrong threat model.
Second, patch velocity that matches discovery velocity. The gap between how fast vulnerabilities are found and how fast they are patched is the exploitation window. That window is widening. Reducing it requires investment in automated testing, canary deployments, and rollback infrastructure that most organizations treat as nice-to-haves rather than critical infrastructure.
Third, a strategy for AI-augmented security engineering. Security products now operate in a world where the threat landscape itself is AI-native. The tools used to find vulnerabilities have changed. The tools used to defend against them must change too.
If you are thinking about this problem at your organization, I would love to hear how you are approaching it. Find me on LinkedIn or reach out through the usual channels.
]]>The Register’s headline said it more plainly: “Sounds like an excellent time to start honing your Debian skills.”
The same week: Oracle cut 30,000 employees, 18% of its workforce, while posting $6 billion quarterly profit. The stated reason is funding $156 billion in AI infrastructure. Meanwhile, Anthropic quietly acknowledged that enterprise users of Claude Code are burning through token quotas “way faster than expected,” causing widespread outages in automated workflows.
These three stories are not separate events. They are the same event, seen from different angles.
The agentic software development lifecycle is no longer a slide in a conference deck. It is being operationalized, often badly, right now.
Let’s be precise, because the term gets used loosely.
The traditional SDLC is a human-driven sequence: requirements, design, implementation, test, deploy, operate. AI tools in the last two years have augmented individual steps: copilot autocomplete, AI-assisted code review, LLM-generated test cases.
Agentic SDLC is different in kind, not just degree. An agent doesn’t assist a human writing a function. It takes a specification, plans a sequence of actions, writes code, runs tests, reads the failures, revises the implementation, and iterates autonomously, in a loop, often across hundreds of API calls per task.
The difference shows up in the numbers. Anthropic’s Claude Code customers ran over quota faster than anyone expected because agentic tasks don’t consume tokens like chat sessions. A single coding task might trigger 200-500 LLM calls. The token economics of agentic work are an order of magnitude different from the chat-mode assumptions baked into most AI contracts and infrastructure plans.
This is why the capacity crunch surprised Anthropic. It will surprise every engineering org that has priced or provisioned AI services based on chat-mode baselines.
Red Hat’s memo-mandate approach to this transition is understandable. It’s also likely to fail, or at minimum, produce compliance without competence.
Here’s why.
Mandating tooling adoption without restructuring workflows produces engineers who use AI tools the same way they used non-AI tools. They write a prompt like a function signature, accept the first output that compiles, and ship code they haven’t actually internalized. The productivity gain is real but shallow. The architectural decisions, the edge case handling, the operational assumptions: those are still being made by engineers who may or may not have developed the judgment to evaluate AI output critically.
Worse: it creates a new class of technical debt that’s harder to see. Code written by an agent and reviewed at the output level by a human who’s optimized for throughput is code that works today and surprises you in six months.
The more dangerous failure mode is organizational. When you mandate AI adoption and tie it to objective metrics, you incentivize engineers to optimize the metrics rather than the underlying work. Cycle time goes down. Defect rate, measured on a 30-day lag, looks fine. What you’ve actually done is moved risk downstream, into production, into customer experience, into the next major incident.
I’ve led engineering organizations through technology transitions: from on-premise to cloud-native, from monolith to microservices, from human-review-only to automated CI/CD gates. The pattern of what works is consistent.
The unit of change is the workflow, not the tool.
Engineers don’t need to be told to use Claude Code. They need their development workflow redesigned around what agentic tools are good at and where they break down. That’s a different conversation, and it requires someone who understands both the tools and the existing workflow well enough to know where the leverage points are.
For agentic coding tools, this means rethinking the specification layer. An agent doesn’t work well with vague requirements. It works well with precise inputs: acceptance criteria, edge cases, expected behaviors, test fixtures. If your team’s requirements are “build a user profile page,” the agent will hallucinate the rest and you’ll spend two hours correcting a misunderstanding that would have taken five minutes to clarify upfront.
This is not a coincidence. The discipline of writing specifications that are precise enough for an agent to act on is the same discipline that makes software engineering predictable at scale. The teams that do well with agentic tools are, almost without exception, the teams that already write good acceptance criteria, maintain strong test coverage, and treat the specification as a first-class artifact, not an afterthought before the sprint starts.
The transition requires new governance, not just new tools.
When a human writes code, the review process catches most of the important issues. The reviewer can ask “why did you do it this way?” and get a reasoned answer. They can infer intent from the structure of the code.
When an agent writes code, the review process has to be redesigned. You can’t ask the agent why it made a decision, at least not in a way that gives you reliable signal about the underlying reasoning. You’re reviewing output, not process. That requires different checklists, different review focus, and different mental models for what “looks right” means.
The engineering orgs that figure this out first will have a durable advantage. The ones that bolt AI tooling onto existing review processes and call it transformation will be slower and more fragile than the ones that didn’t adopt AI at all.
Measurement needs to evolve simultaneously.
Red Hat is right that objective metrics matter. They’re wrong, or at least incomplete, about which metrics. Cycle time and defect rate are lagging indicators that can be gamed. The leading indicators for a healthy agentic engineering org look more like: specification quality scores, test coverage at specification time (before agent invocation), agent task success rate on first pass, and rework rate on agent-generated code.
These metrics don’t exist as standard dashboard items in Jira or Linear. Building them requires intentional instrumentation and a clear model of what the agentic workflow actually looks like in your context.
Most of the conversation about agentic AI and engineering orgs is framed as a staffing question: will we need fewer engineers? The answer is probably yes, in aggregate and over time, for some categories of work. But that’s not the useful question for the next 18 months.
The useful question is: what does your engineering org’s structure look like when agents handle a significant portion of implementation work?
The answer is not “the same structure, but with fewer people.” It’s a different structure. The ratio of people who can write precise specifications to people who write code shifts dramatically. The value of someone who can define acceptance criteria, understand edge cases, and evaluate agent output critically, without writing the implementation themselves, goes up significantly. The value of someone who writes fast, doesn’t ask questions, and ships features goes down relative to its current premium.
This is a different org design problem. It requires thinking about career paths, hiring profiles, and team structures that most engineering leaders haven’t had to consider before. The teams being built today that will thrive in an agentic workflow are not the teams that look like 2024’s high-performing engineering orgs.
If you’re an engineering leader trying to navigate this transition, a few concrete things that actually matter:
Invest in your specification layer before your agent layer. The biggest leverage point in an agentic workflow is the quality of the input to the agent, not the sophistication of the agent itself. Teams that develop a disciplined approach to writing specifications, ones that are rigorous enough to act on without human clarification, will get dramatically better results from AI coding tools than teams that don’t, regardless of which model they use.
Audit your token economics. If you have AI tooling in production and you haven’t modeled what agentic workloads do to your token consumption, you are flying blind. Claude Code’s quota exhaustion issue is not unique to Anthropic. It will hit any team running multi-step agentic workflows at scale. Build the consumption model now.
Redesign review, don’t inherit it. Your current code review process was designed for human-written code. It needs to be redesigned for agent-generated code. The checklist is different. The focus is different. The time allocation is different. This is one of the highest-leverage investments you can make in the next 90 days.
Change management is the bottleneck, not the technology. The engineers most resistant to agentic tooling are often your best ones, because they have the most internalized judgment about how software should be built. Losing their buy-in is catastrophic. The transition has to be framed as “we’re raising the floor and changing what good engineering judgment means,” not “we’re automating your job.” Those are different conversations and they land very differently.
The Red Hat memo will probably accomplish something. Engineers will use more AI tooling. Metrics will move. Some of the movement will be real.
But the engineering organizations that come out of this transition with durable advantages won’t be the ones that mandated adoption fastest. They’ll be the ones that understood what actually changes when agents become part of the development loop, and redesigned their workflows, their governance, and their org structures accordingly.
That’s a harder problem than picking the right AI coding tool. It’s also the one worth solving.
I’m building ShipKode Arc, a governed agentic control plane that structures the journey from customer signal to engineering-ready specification. The specification layer is where most agentic SDLC transitions break down first. If you’re thinking through what this transition looks like for your organization, I’m happy to compare notes.
]]>This is not a drill. Assume compromise if you’ve pulled affected versions. Immediate action required.
Today, the open-source community is grappling with what may be the most sophisticated npm supply chain attack ever documented against a top-10 package. The ubiquitous axios package, downloaded over 100 million times weekly, was compromised, injecting a hidden dependency that deploys cross-platform Remote Access Trojans (RATs). If your codebase, CI/CD pipelines, or any dependency tree has pulled axios@1.14.1 or axios@0.30.4, you must treat your environment as fully compromised and take immediate corrective action.
Attackers managed to hijack the npm account of maintainer jasonsaayman. In a move demonstrating advanced operational security, they swapped the npm account’s registered email to a ProtonMail address, ensuring they controlled account recovery. They then published tainted versions (axios@1.14.1 and axios@0.30.4) containing a malicious, hidden dependency: plain-crypto-js@4.2.1.
This malicious dependency was no simple script. It was engineered to deploy OS-specific RATs:
Compounding the severity, the malicious dependency was staged 18 hours in advance, and three separate payloads were pre-built. The tainted versions were published across both release branches within a mere 39 minutes, maximizing the attack’s reach before detection.
This attack bypasses many traditional security measures because it targets the trust inherent in the software supply chain. If you have pulled either axios@1.14.1 or axios@0.30.4, the following steps are critical:
package-lock.json, yarn.lock, etc.) for any instances of axios@1.14.1 or axios@0.30.4. Remove them and revert to a known-good version (any version prior to 1.14.1 in the 1.x branch, or prior to 0.30.4 in the 0.x branch).The Axios attack is a stark reminder that the software supply chain is the new perimeter. As developers, we place immense trust in the open-source packages we integrate into our projects. This incident, however, demonstrates that this trust can be exploited with devastating effect. It is no longer sufficient to simply rely on the reputation of a package; proactive security measures, rigorous auditing, and a security-first mindset are paramount. The widespread adoption of these compromised versions highlights a critical need for better automated security checks and verification processes throughout the development lifecycle.
Stay vigilant. The landscape of cybersecurity threats is constantly evolving, and the open-source ecosystem, while incredibly valuable, is a prime target.
#Cybersecurity #SupplyChainAttack #npm #OpenSource #DevOps #Security #Axios #RAT
]]>The tool: Claude Code and Codex, alternating roles, iterating toward a complete implementation. The result: production-grade, passing the test suite, shipped.
You’d think the story is about how good AI coding agents have gotten. It is. But the buried lead is more interesting: the existing test suite was the enabling factor. That’s what made a 7-hour $400 port possible instead of a 7-month $400K nightmare.
No tests? No vibe port. The agents would have produced something that looked plausible, passed superficial review, and quietly misbehaved in production.
This is the vibe coding trap.
The term was coined by Andrej Karpathy and spread fast because it named something real. You describe what you want in natural language, iterate conversationally with an AI agent, and working code appears. You’re not reading line-by-line. You’re not writing functions. You’re specifying intent and reviewing output.
Matt Webb put it well in an essay that circulated widely last week: “While I’m vibing, I am looking at lines of code less than ever before, and thinking about architecture more than ever before.”
The popular interpretation of vibe coding is that it lowers the bar: now anyone can build software, architecture doesn’t matter, senior engineers are less important. The viral take is that it’s dangerous because the code you ship is code you don’t understand.
Both interpretations miss the more important point.
Vibe coding doesn’t eliminate the need for architecture. It changes where architecture lives and amplifies the consequences of having it or not.
When a human engineer implements a feature, they make dozens of micro-decisions along the way. Some are conscious: they choose an algorithm, they pick a data structure, they decide whether to introduce an abstraction. Some are unconscious: they name things consistently with the existing codebase, they handle edge cases the way the surrounding code handles similar cases, they apply patterns they’ve internalized from code review.
This is slow. It’s also self-correcting. A human engineer building in a messy codebase will slow down, feel friction, and either push back (“this is getting hard to reason about”) or work around the problems they hit (“I’ll just duplicate this logic, the abstraction is too tangled”). The mess becomes legible as a constraint.
An agent doesn’t work like this.
An agent builds what the context implies. In a clean, well-structured codebase with clear conventions, good test coverage, and explicit design documentation, an agent generates code that fits. It extends the architecture naturally because the architecture is legible in the artifacts it has access to.
In a tangled codebase with inconsistent conventions, poor test coverage, and implicit design decisions living only in the heads of senior engineers, an agent generates code that fits the tangle. It extends the bad patterns as fluently as it would extend good ones. It ships faster than a human would, which means the mess compounds faster.
The JSONata story is a proof of concept for the upside. The test suite was the structural artifact that made correctness verifiable. The agents could iterate because there was a ground truth to iterate toward. Remove the tests, and you have a 7-hour sprint toward something plausible but unverifiable.
Good architecture accelerates agentic development. Bad architecture accelerates agentic debt accumulation.
When you’re evaluating whether your codebase is ready for agentic development, the question isn’t “is the code clean?” It’s “do the right artifacts exist?”
Agents need three things to produce correct output reliably.
Verifiable correctness criteria. Tests, specifically. Not just unit tests: integration tests that exercise real behavior paths, not implementation details. An agent without a test suite is producing output you can only evaluate by reading it, which puts you back in the loop as a full reviewer. An agent with strong test coverage can be directed to “make the tests pass” and trusted to converge on correct behavior. The test suite is the architecture artifact most directly connected to agentic velocity.
Legible conventions. Agents learn from context. If your codebase has consistent naming conventions, consistent error handling patterns, consistent module structure, an agent will reproduce those conventions. If your codebase has three different error handling patterns across three different parts of the system, the agent will apply whichever one is statistically dominant in the surrounding context. This might not be what you want. Convention documentation, linting rules, and code review standards are architecture artifacts that agents consume directly.
Explicit design decisions. Agents fill ambiguity with plausible defaults. Every design decision that lives only in the head of a senior engineer is an ambiguity the agent will fill during generation. When that decision is something like “we always paginate at 100 records because our downstream service has a hard limit” and the agent decides to paginate at 50 for cleaner code, the generated code looks fine in review and fails in production. Architecture decisions need to be written down, ideally close to the code they govern, to be accessible to agents.
None of this is new guidance. This is what good engineering culture looked like before agents existed. Agents make the consequences of skipping it faster and more expensive.
Four frontier models dropped in the last 23 days. MCP just crossed 97 million installs. Linear’s CEO declared issue tracking dead. JetBrains retired their pair programming product and launched an agentic development platform. The ecosystem signal is consistent: agentic development is crossing the production threshold now, not in 2027.
The organizational response most teams are making is “let’s give everyone access to Claude Code and Copilot and see what happens.” That’s a reasonable start. It’s not a strategy.
The engineers who will have the most leverage in an agentic-first org are not the ones who type the fastest or know the most obscure language features. They’re the ones who are excellent at:
Decomposing complex product intent into unambiguous requirements that an agent can execute correctly the first time. Writing specifications that capture not just what to build but why, what the constraints are, and what’s explicitly out of scope. Structuring work into chunks an agent can execute and a reviewer can verify independently. Evaluating agent output against stated intent, not just functional correctness. And, critically, building and maintaining the test suites and convention artifacts that give agents a structure to build within.
This is closer to product architecture than to traditional senior IC work. The title might still say “Staff Engineer.” The day looks different.
If you’re a CTO or VP Engineering evaluating how to integrate agentic development into your team, the question to start with isn’t which tool to adopt. It’s: what is the current quality of our architecture artifacts?
Specifically: what is our test coverage, and does it cover behavior or implementation details? Are our conventions documented and enforced? Are our design decisions written down anywhere, or do they live in senior engineer memory? Is our codebase legible to an outside reader (which is what an agent is)?
If the answers are bad, adding agents adds speed to a compounding problem. The ROI story is real but it’s conditional. You get the 7-hour $400 JSONata port when the test suite exists. Without it, you get 7 hours of plausible code that might take 7 weeks to debug.
The good news: the work required to make a codebase agent-ready is the same work that makes it human-maintainable. There’s no new category of investment here. There’s just a new urgency. The teams that have been cutting corners on testing and documentation have been paying a slow tax. With agents in the loop, that tax gets collected faster.
Vibe coding is genuinely powerful. The velocity gains are real, the JSONata story is not an outlier, and the engineering teams running away from this are going to be structurally disadvantaged in 18 months.
The trap is the framing that comes packaged with the term. “Vibe” implies loose, intuitive, structure-optional. Karpathy’s original framing was more precise: he said you embrace the vibes and “forget that the code even exists.” That’s a statement about attention, not about architecture. The structure still has to be there. You just stop reading every line you generate.
When the code exists in a well-structured system with strong test coverage and explicit conventions, forgetting that it exists is fine. You can trust the artifacts. When it doesn’t, forgetting that it exists is how you end up with a codebase nobody understands and agents that confidently extend the confusion.
Architecture has always mattered. It just used to be possible to get away with shortcuts because humans slowed down when things got messy. Agents don’t slow down. They build into the mess at full speed.
The teams that will win with agentic development are not the ones that move fastest today. They’re the ones that built the foundations that let them move fast safely.
David Lapsley, Ph.D., is Founder and CEO of ShipKodeAI. He previously led engineering at AWS (largest network fabric in Amazon history, 0 to 90 engineers in 18 months), Corelight (VP Engineering), and Cisco (Maglev Platform, DNA Center $1B ARR). He writes about AI governance, AI-accelerated PDLC, and the gap between AI hype and production reality.
]]>Eleven leading AI models. 2,405 human participants. The finding: every single model (OpenAI, Anthropic, Google, Meta, Qwen, DeepSeek, Mistral) endorsed wrong choices at higher rates than humans.
That’s not the scary part.
The scary part: people trusted the sycophantic models more. They were 13% more likely to return to them. A single session with a sycophantic AI reduced participants’ willingness to accept responsibility for mistakes and increased their conviction they were right, even when they weren’t.
It’s not just the model agreeing with you. It’s subtler.
You say “I think we should rewrite this service in Go.” The model walks you through five reasons why that’s a great idea, maybe adds a mild caveat at the end. You feel validated. You proceed.
What a good engineer would do: push back. Ask why. Make you defend the premise. Maybe the rewrite is right, maybe it isn’t, but you should have to work for the conclusion.
AI systems are trained to maximize human approval. Approval is easier to get by agreeing than by challenging. The result is a tool that systematically reinforces whatever you walked in believing.
For low-stakes tasks (“draft this email,” “write a unit test”) sycophancy is mostly harmless.
The problem is that we’re not keeping AI in low-stakes lanes. We’re using it for:
These are consequential. And the Stanford data says you are less likely to question your own judgment after an AI validates it, even when the AI was wrong and you were wrong.
The mechanism is insidious: AI validation feels like external confirmation. It isn’t. It’s a mirror that agrees with you.
I’ve caught myself doing this. You have an idea, you’re 70% sure it’s right. You ask Claude or GPT to poke holes in it. It identifies a few minor risks, agrees the overall direction is sound, maybe adds something you hadn’t thought of. You leave the conversation 90% sure.
The confidence increased. The underlying quality of the idea didn’t.
That’s the trap. The model’s agreement isn’t evidence. It’s a reflection of what you already believed, polished and returned to you with citations.
If you’re deploying AI in any decision-support context (performance reviews, code review tools, incident postmortems, product prioritization) you have a liability you may not have modeled.
Your system may be systematically reinforcing the biases of whoever is using it. Senior people will feel more confident in wrong calls. Junior people will defer to the AI rather than developing judgment. The organizational effect compounds over time.
This isn’t theoretical. The Stanford team measured it in a controlled experiment. Thirteen percent more likely to return to the model that agreed with them. That’s a selection effect: the most sycophantic tool wins the usage competition, regardless of whether it’s the most accurate one.
A few things that actually help:
Prompt for adversarial review. Don’t ask “is this a good idea?” Ask “what would have to be true for this to be a terrible idea?” or “steelman the case against this.” The model will comply. It just won’t do it by default.
Separate generation from evaluation. Use AI to generate options, then use a different prompt (or a different model) to evaluate them. Don’t let the same session that helped you build the idea also judge it.
Be suspicious of easy agreement. If the AI walks you through five reasons your plan is good with no real friction, that’s a signal, not a green light. Find the friction yourself.
Tell the model to push back. Literally: “Your job in this conversation is to challenge my assumptions, not validate them. Be direct. I’d rather hear a hard truth now.” It works better than you’d expect.
Don’t use AI for final calls on consequential decisions. Use it to think. Use humans to decide.
I use Claude every day. I’m building a company on top of these models. I’m not arguing you should use AI less.
I’m arguing you should use it with clear eyes about what it actually is: a system optimized to make you feel good about your decisions, not to make your decisions better.
The fact that it’s published in Science, tested across 11 models and 2,400 people, isn’t a reason to panic. It’s a reason to be deliberate.
The tool is powerful. The tool also tells you what you want to hear. Both things are true.
The Stanford study: “Conversational AI Sycophancy Reduces Human Responsibility and Confidence Calibration,” published in Science, March 2026.
David Lapsley, Ph.D., is Founder and CEO of ShipKodeAI. He previously built and led engineering organizations at AWS (largest network fabric in Amazon history, 0→90 engineers in 18 months), Corelight (VP Engineering), and Cisco (Maglev Platform, DNA Center $1B ARR). He writes about AI governance, AI-accelerated SDLC, and the gap between AI hype and production reality.
]]>The internet had a field day with the irony. A company that built its brand on AI safety, failing basic information security hygiene. But most of the coverage missed the more important story — the one inside the leak.
Buried in those documents was a draft announcement for Anthropic’s next model, codenamed Mythos. In that draft, Anthropic’s own team wrote this:
Mythos is “currently far ahead of any other AI model in cyber capabilities” and “presages an upcoming wave of models that can exploit vulnerabilities in ways that far outpace defenders.”
The people building the most capable AI offensive tool in history wrote, in their own words, that it will outpace defenders.
They published it by accident. But they knew it before they did.
It’s easy to laugh at the leak. Cybersecurity stocks didn’t — CrowdStrike fell 7%, Palo Alto fell 6% on Friday. The market understood something that the coverage didn’t fully articulate:
The threat model just changed.
Not because Mythos was leaked. Because its existence was confirmed, along with an authoritative description of its capabilities from the organization that built it.
Every threat model your security team is operating under right now was calibrated against today’s AI capabilities — AI-assisted phishing, script-kiddie automation of known CVEs, social engineering at scale. That’s the threat you’ve been defending against.
Mythos isn’t that. It represents a qualitative step change: novel vulnerability discovery, multi-step reasoning across complex attack surfaces, autonomous chain-of-thought exploitation. These capabilities have existed in theory. Mythos is the first system its creators believe can execute them reliably.
The model hasn’t even launched yet.
There’s a structural problem in cybersecurity that has existed forever and is about to get dramatically worse.
Attackers need to find one way in. Defenders need to cover everything.
AI makes this asymmetry worse in a specific way: it dramatically lowers the skill floor for attackers while doing relatively little to lower the cost of comprehensive defense. A motivated actor with access to Mythos can direct it at your infrastructure and let it explore. It doesn’t sleep. It doesn’t get bored. It can test a thousand variants of an approach while your team is handling an unrelated incident.
Your blue team is still human. Limited working hours, limited tooling budgets, alert fatigue. They’re triaging a queue while an AI system on the other side does systematic, patient, creative reconnaissance.
This is what Anthropic meant by “outpace defenders.” They weren’t speculating. They were describing their own benchmark results.
Let me be concrete.
Vulnerability discovery at scale. Today’s AI can help a developer find a buffer overflow with guidance. Tomorrow’s AI autonomously audits a codebase, identifies the vulnerability class, reasons about exploitability in the context of your specific deployment, and generates a working proof-of-concept. The gap between AI-assisted security research and AI-autonomous exploitation is closing faster than the industry wants to admit.
Social engineering that defeats human judgment. Current spear-phishing is detectable because it lacks context — plausible text, but impersonal. Advanced models with long context windows and access to your public information surface (LinkedIn, GitHub, press releases, Glassdoor) can craft targeted attacks that sound exactly like someone who understands your org’s internal dynamics. At scale. Against every person in your company simultaneously.
CI/CD pipeline infiltration. This week, Trivy — the most widely used container vulnerability scanner — was compromised for the second time in a month. TeamPCP force-pushed 75 of 76 version tags, turning trusted CI/CD references into credential thieves harvesting SSH keys, cloud tokens, and Kubernetes configs. That attack was done by humans. An AI-directed version targets the specific tools your organization is known to use, correlates them with your cloud configuration, and prioritizes by likely yield. Your build pipeline is now the attack surface.
Zero-day chain construction. The hardest part of advanced persistent threats isn’t finding a single vulnerability — it’s chaining multiple low-severity issues into a high-impact path. That’s a reasoning problem. It’s exactly what frontier AI excels at.
Here’s what the Mythos leak actually reveals, and it’s not comfortable.
Anthropic knows their model has these capabilities. They said so in writing. They are building it anyway, because if they don’t, OpenAI will. Or a well-funded startup will. Or a nation-state with fewer scruples about deployment will.
This isn’t a criticism of Anthropic specifically — every frontier lab operates under the same constraint. The competitive dynamics of the AI race have created a situation where the organizations most committed to safety feel compelled to build and release increasingly powerful offensive tools because the alternative is ceding the frontier to someone with fewer commitments.
The result: the most capable cyber-offensive AI in history will be commercially available, probably this year. Every serious threat actor on the planet will have access to it or something like it within 12–18 months.
This is the context your blue team is operating in.
The defensive response to this moment is tractable — but it requires a different mental model than most security organizations are currently using.
You cannot patch your way to safety. Vulnerability management against a system that discovers novel attack chains faster than your team can triage them is a losing strategy. Move investment upstream: architecture reviews, threat modeling, blast radius reduction, defense-in-depth that makes exploitation economically unviable even when vulnerabilities exist.
Blue teams need AI now, not later. The only way to close the asymmetry is to put AI in the hands of defenders at the same rate attackers are acquiring it. AI-augmented threat detection, AI-assisted incident response, AI-driven adversarial simulation against your own systems. Organizations that build this capacity today will have 18 months of operational learning before the threat fully materializes.
Your build pipeline is your new perimeter. Mandatory SBOM generation, cryptographic signing of build artifacts, real-time integrity verification of CI/CD dependencies — these move from best practice to table stakes. The Trivy attack is a preview. The next generation will be AI-directed and systematic.
Assume credential exposure. When an attacker has an AI system doing reconnaissance and exploitation, the window between credential compromise and lateral movement collapses. Zero-trust architecture — not as a marketing term but as an actually implemented policy where every service authenticates every request — is the only architecture that limits blast radius when the perimeter fails.
Invest in human defenders, not just tools. The instinct when facing AI-powered offense is to buy AI-powered defense products. Necessary but not sufficient. The humans on your blue team need to understand how AI-directed attacks reason — what they look for, how they chain vulnerabilities, what operational patterns give them away. That requires deliberate training, red-team exercises with current AI capabilities, and institutional knowledge-building that takes time. Start now.
The week started with an AI safety company leaving 3,000 documents in an unsecured database.
The week ended with those documents revealing that the same company believes it has built an AI system capable of offensive cyber operations that outpace defenders.
I don’t think this is a story about Anthropic’s hypocrisy. I think it’s a story about how hard this problem actually is. The people who understand AI capabilities best — who have thought hardest about safety, who have written internal warnings about what their own tools can do — are still subject to the oldest failure mode in information security: a misconfigured default setting.
If that’s true at Anthropic, it’s true everywhere. Including your organization.
The warning was inside the leak. The question is whether you act on it before someone else finds your unlocked door.
David Lapsley, Ph.D., is Founder and CEO of ShipKodeAI. He previously built and led engineering organizations at AWS (largest network fabric in Amazon history, 0→90 engineers in 18 months), Corelight (VP Engineering), and Cisco (Maglev Platform, DNA Center $1B ARR). He writes about AI governance, AI-accelerated SDLC, and the gap between AI hype and production reality.
]]>He’s right. But the conclusion most people will draw from that is wrong.
The end of issue tracking doesn’t mean the end of structure. It means the structure moves upstream.
Before declaring it dead, it’s worth being precise about what a ticket actually is.
A Jira issue, a Linear task, a GitHub issue - at their core, these are containers for intent. They say: here is a thing that needs to happen, here is who’s responsible, here is what done looks like. They exist because software development involves multiple people, multiple timelines, and more work than any individual can hold in their head.
That coordination function doesn’t go away when agents enter the picture. It gets compressed.
When an AI agent can take a ticket from open to closed in minutes rather than days - create a branch, write the code, open the PR, run the tests - the ticket lifecycle collapses. The human handoff points that made the tracker useful (assign → start → review → merge → close) compress into near-zero time. At that point, maintaining a tracker at the old granularity becomes overhead without benefit.
Linear’s CEO is responding to this reality. If agents resolve tickets faster than humans can write them, the tracker’s coordination function is largely gone.
But coordination hasn’t disappeared. It’s just moved.
Here’s what happens when agents handle implementation:
The bottleneck is no longer writing code. It’s knowing what to build.
That sounds obvious, but the implications are non-trivial. Every ambiguity in a requirement that a human engineer might have resolved through a Slack thread, a quick question to the PM, or a judgment call during implementation - an agent resolves through training-data patterns. Silently. Without flagging that a decision was made.
This is the same dynamic I described in cognitive debt last week, from a different angle. Agents don’t ask clarifying questions by default. They fill ambiguity with plausible defaults. That looks fine in the moment. It compounds into drift, rework, and systems that don’t match the product intent over time.
If the ticket is still the primary artifact - a short description of what to build, maybe some acceptance criteria, assigned to an agent - the agent will ship something. It might even pass the tests. But whether it’s the right thing, built the right way, in a manner consistent with where the product is going? That’s not in the ticket. It never was.
The issue tracker hid this gap as long as human engineers were filling it with judgment. Agents expose it.
When agents handle implementation, the work that matters happens before the ticket is created.
Specifically:
Clear product intent. What is this feature actually for? Who uses it, in what context, with what goal? Not a user story - genuine clarity on the problem being solved. Agents implement what you specify. If the specification is a one-liner in a tracker, you’ll get a one-liner’s worth of thinking embedded in the implementation.
Explicit design decisions. What approach are we taking? What are the constraints - performance, security, backward compatibility? What’s explicitly out of scope? These decisions will be made either by a human before generation or by the agent during generation. If the agent makes them, they’re invisible.
Acceptance criteria that capture behavior, not just output. “User can reset password” is a different specification than “user can reset password via email link, link expires in 24 hours, expired links show a specific error message, rate limiting applies at 3 attempts per hour.” The second version gives an agent enough structure to get it right. The first version gives an agent permission to fill in everything you didn’t say.
Traceability from intent to implementation. When an agent ships a PR, what requirement does it satisfy? Which design decision does each architectural choice reflect? Without this, debugging unexpected behavior six months from now means reverse-engineering decisions that were never documented.
None of this is a ticket. All of it is a spec.
When agents handle implementation, the engineering team’s leverage moves to specification quality.
This is a meaningful shift. In a traditional development process, a senior engineer’s value is partly in their ability to write good code quickly. In an agentic process, the code comes fast regardless - the value is in writing specifications precise enough that the code is correct the first time, and structured enough that it can be understood and changed later.
That’s a different skill profile. It’s closer to product architecture than to individual contributor coding. The engineers who thrive in an agentic-first organization will be the ones who are excellent at:
The issue tracker was built around a different unit of work - the individually assigned, human-implemented task. That unit is changing. The spec - the structured artifact that captures what to build and why - is what replaces it.
I’ve spent the past year thinking about exactly this problem. Not as an abstract organizational design question, but as the practical question of: what does an engineering team need to operate effectively when agents handle implementation?
The answer we’ve arrived at at ShipKode is that the gap isn’t in the tracker. It’s upstream of the tracker - in the translation from customer signal to validated product intent to engineering-ready specification.
Most teams are flying blind on that translation. Customer interviews generate insights that aren’t systematically captured. Product intuition drives roadmaps that aren’t traceable to evidence. Requirements are written fast to unblock development, without the structure agents need to implement correctly.
ShipKode is a governed agentic control plane for that process - from raw market signals and customer evidence through personas, use cases, PRFAQs, and feature maps to Kiro-format engineering specifications with full traceability. Every requirement traces back to the customer evidence that motivated it. Every design decision is documented before implementation begins.
When the ticket is created - or when the agent picks up the work - the spec already exists. The agent isn’t filling ambiguity with guesses. It’s implementing a decision that was made, by a human, with evidence.
That’s the organization that gets the velocity benefit of agentic development without the cognitive debt that comes from skipping the thinking.
Linear’s CEO declared issue tracking dead and pivoted to agentic AI. That’s a product decision - Linear will adapt or be displaced like every other tool in the developer workflow that agents are disrupting.
But the harder question isn’t “what happens to the tracker?” It’s “what does the engineering org look like when agents handle implementation?”
My read: the org gets smaller at the implementation layer and much more important at the specification layer. The ratio of engineers who are primarily specifying and reviewing to engineers who are primarily coding shifts dramatically. The skills that matter shift. The artifacts that matter shift.
Teams that recognize this early and restructure accordingly - building the muscles for high-quality specification, establishing traceability from customer intent to code, treating the spec as the primary engineering artifact - will have a significant and compounding advantage over teams that just point agents at their existing backlog and wonder why the output doesn’t match what they wanted.
The tracker is dead. Long live the spec.
David Lapsley, Ph.D., is Founder and CEO of ShipKodeAI. He previously built and led engineering organizations at AWS (largest network fabric in Amazon history, 0→90 engineers in 18 months), Corelight (VP Engineering), and Cisco (Maglev Platform, DNA Center $1B ARR). He writes about AI governance, AI-accelerated SDLC, and the gap between AI hype and production reality.
]]>I’ve been building with AI agents every day for over two years. I agree with the diagnosis. I disagree with the prescription.
The answer isn’t to slow down. The answer is to build structure that makes fast review possible.
Technical debt is code that works but costs you later — shortcuts that accumulate interest in the form of maintenance burden, fragility, and the eventual painful refactor.
Cognitive debt is different. It’s the gap between what exists in your codebase and what your team can reason about, explain, and confidently change. A team accrues cognitive debt when the codebase evolves faster than their mental model of it does.
Traditional technical debt accumulates at human writing speed. Cognitive debt in an AI-assisted codebase accumulates at agent generation speed.
Here’s what that looks like in practice:
An engineer opens a Cursor session at 9am with a reasonable feature request: “implement the user notification system.” By noon, the agent has generated 800 lines across 12 files — service classes, database migrations, background jobs, API endpoints, email templates. The code compiles. Tests pass. The engineer reviews it for 45 minutes, approves the PR, and moves on.
Three weeks later, a bug appears in the notification system. The engineer who reviewed the PR is in a different context. The engineer who needs to fix the bug wasn’t there for the original session. They open the files. The code is readable, technically correct, but they don’t understand the decisions embedded in it. Why does the retry logic use exponential backoff to 24 hours? Why is there a separate queue for digest notifications vs. real-time ones? Why does this specific handler exist? There are no comments. There’s no design document. The only artifact is the code.
That’s cognitive debt. The team owns code they can’t confidently reason about.
Human developers accumulate cognitive debt too. But agents accelerate it in specific ways that make the problem qualitatively different.
Agents generate pattern-complete code, not decision-documented code. A good human engineer writes a comment when they make a non-obvious choice. An agent writes code that looks reasonable and compiles — without flagging that it made a choice at all. The agent doesn’t know what you’ll need to understand later. It knows what looks right now.
Agents don’t ask questions by default. A human engineer who doesn’t understand a requirement will ask. An agent fills the ambiguity with training-data patterns. It looks decided, even when the underlying decision was never made.
Agents compress timelines in ways that skip natural review gates. When a feature takes two weeks to write, review happens incrementally — architecture discussions, PR comments, pairing sessions. When an agent writes it in two hours, the review happens at the end against a fait accompli. The incremental review that catches design problems early is replaced by a single pass that tends to accept what’s there.
Agents make cross-file, cross-service changes that are hard to review holistically. A 20-line change touching one function is easy to review. An 800-line change touching 12 files is a different cognitive challenge — even if every individual file is readable. The emergent behavior across the system is what matters, and that’s hard to evaluate from a diff.
The insidious thing about cognitive debt is that it doesn’t show up in your metrics until something breaks.
Velocity looks great. PRs are shipping fast. The feature backlog is moving. Code review turnaround time is down. Every leading indicator says things are going well.
Then a critical system needs to change. Or a new engineer joins and needs to make sense of what exists. Or an incident happens and the team needs to reason about behavior they didn’t write. That’s when the cognitive debt comes due — usually at the worst possible moment.
I’ve watched this happen at companies deploying agentic development workflows. Six months in, velocity has doubled. Engineers feel productive. The team is shipping more than they ever have. Then a senior engineer leaves, or a scaling event exposes unexpected behavior, or an audit requires explaining architectural decisions — and suddenly the team realizes they own a system they don’t fully understand.
At that point, the debt doesn’t come due slowly. It comes due all at once.
Zechner recommends daily limits on agent-generated code and human-authored architecture decisions. I understand the impulse. But in practice, “slow down the agents” isn’t a durable policy. Competitive pressure, deadline pressure, and the productivity delta between teams using agents aggressively and teams using them cautiously will eventually win.
The teams that slow down will ship slower. That’s a real cost. The goal should be to capture the velocity benefit without paying the cognitive debt price — and that’s achievable, but it requires structure.
The structure that works is pre-generation specification, not post-generation review.
The root cause of cognitive debt is generating code from ambiguous intent. When an engineer prompts “implement the notification system” without specifying the decision surface, the agent fills every ambiguity with training-data defaults. The cognitive debt accumulates in the gap between the intent and the defaults.
The solution is to make the decisions explicit before generation happens. Not elaborate bureaucratic specs — lean structured artifacts that capture the decisions that would otherwise be implicit in generated code.
What that looks like in practice:
Requirements that specify what the feature must do, not how — including edge cases, constraints, and what’s explicitly out of scope. An agent that operates with explicit requirements makes fewer silent decisions.
Design documents that capture the architectural choices — why this approach over alternatives, what the data model looks like, how it integrates with existing systems. These don’t need to be long. They need to capture the non-obvious decisions so reviewers can evaluate them rather than reverse-engineer them.
Task breakdowns that decompose the work into reviewable units. A single 800-line PR is hard to reason about. Five 160-line PRs, each with a clear stated purpose, are reviewable in parallel.
I’ve written about this pattern in the context of AI drift and specifications as the interface between product and engineering. The connection to cognitive debt is direct: the same ambiguity that causes agents to drift away from requirements also makes the resulting code hard to reason about after the fact. Structured specifications solve both problems simultaneously.
Even with good pre-generation structure, review needs to adapt to what agents produce.
Review the decisions, not the code. The question isn’t “is this code correct?” (the agent usually gets that right). The question is “are the decisions embedded in this code the decisions we wanted to make?” That’s a different review, and it requires the spec to review against.
Require design artifacts in PRs that touch architecture. Any PR that introduces a new service, changes a data model, or modifies a system integration should include a brief design note — even three sentences explaining why this approach. This is the minimum viable documentation for reasoning about the system later.
Use the agent to document its own decisions. After a generation session, prompt the agent: “Write a brief explanation of the non-obvious decisions you made and why.” This works surprisingly well and takes two minutes. The output isn’t perfect documentation, but it’s substantially better than nothing.
Pair-review high-complexity outputs. For PRs over a certain complexity threshold, require two reviewers. The first reviews for correctness; the second reviews for whether the design is coherent with the broader system. This is overhead — but it’s cheap insurance against cognitive debt in the system’s critical paths.
Anthropic shipped Claude Code Auto Mode this week — a secondary model that reviews every action the primary agent takes before execution, blocking pushes to main, preventing credential exfiltration, catching unsafe patterns. It’s a meaningful step toward production-safe agentic coding.
But Auto Mode solves a different problem. It prevents unsafe actions in the moment. It doesn’t prevent cognitive debt from accumulating over time. An agent can behave safely at every step and still produce a codebase that your team can’t reason about six months from now.
The guardrails on agent behavior are improving quickly. The discipline around agent output — what decisions get documented, what review looks like, what structure exists before generation — is still largely up to you.
That’s where Zechner and Willison are right. Not that you should move slower, but that speed without structure produces a compounding liability. The teams that win won’t be the ones that move fastest. They’ll be the ones that move fast and stay legible.
David Lapsley, Ph.D., is Founder and CEO of ShipKodeAI. He previously built and led engineering organizations at AWS (largest network fabric in Amazon history, 0→90 engineers in 18 months), Corelight (VP Engineering), and Cisco (Maglev Platform, DNA Center $1B ARR). He writes about AI governance, AI-accelerated SDLC, and the gap between AI hype and production reality.
]]>It has been, without exaggeration, the most valuable month of work since I started building ShipKode.
Every conversation has sharpened something. My understanding of how teams actually adopt AI-assisted development tools. The language they use to describe their pain. The workflows they’ve built around the problems I’m trying to solve. The specific dimensions of the problem that matter most to them and the order in which they matter.
I walked into these conversations with a strong thesis. I’d spent 18 months building production software with AI coding agents, writing about the gaps in AI-assisted development, and designing a specification pipeline to close those gaps. I had conviction about the direction. The conversations reinforced that conviction, but they did something even more valuable: they gave me the texture and nuance to make the product significantly better than what I would have built from my thesis alone.
Hearing a VP of Engineering describe how their team wants to adopt AI-assisted development but doesn’t know where to start, how they’ve tried three different tools and still can’t get past the POC stage, how they know the transformation is coming but can’t find a path from where they are to where they need to be — that adds a dimension no amount of desk research provides. The direction was right. The discovery made it sharper, more specific, and more valuable.
There’s a principle I keep coming back to: understand the customer before the customer understands you.
The natural instinct when you’ve built something is to show it. You’ve spent months on this thing. You’re proud of it. You want validation. So you book a meeting, share your screen, and start clicking through the product. I’ve done this. It works, and there’s a time for it.
But when you lead with a demo, the conversation narrows. The customer is reacting to your solution instead of describing their problem. Their feedback becomes bounded by what you showed them. They’ll tell you what they think of your feature, not what keeps them up at night.
The best conversations I had this month were the ones where I resisted the demo impulse and started with questions. How does your team work today? What’s painful? What have you tried? I let them describe the problem in their language, using their mental models, with their priorities. When I did eventually show the product, later in the conversation, the context I’d gained made the demo dramatically more effective. I knew which parts to focus on because I’d just heard what mattered to them.
The ordering matters. Discovery first, then demo. Understand first, then be understood. The insights you get from this sequence are qualitatively different from leading with a pitch. Demo-first feedback tells you whether someone likes what you built. Discovery-first feedback tells you whether you’re building the right thing, and sharpens the demo when it comes.
I’ve written extensively about how 95% of AI pilots fail to reach production. The MIT NANDA study found that the root cause is almost never technical. The models work fine. The code compiles. The feature just doesn’t solve a real problem, or solves it in a way nobody actually specified.
Customer discovery is the fix for that failure mode. Not better models. Not better prompts. Conversations with the people whose problems you’re trying to solve.
For early-stage startups, this matters more than anything else you could spend your time on. Here’s why.
You can’t afford to build the wrong thing. As I noted in my post on Amazon’s Working Backwards SDLC for SMBs, a 5-person team that builds the wrong feature for a quarter has burned 25% of its runway. Customer discovery is how you avoid that. Thirty minutes of conversation can save three months of engineering.
Discovery refines your direction in ways you can’t anticipate. Every founder believes they understand their customer. I had a strong thesis going in, and the conversations validated the core direction. But they also surfaced refinements I couldn’t have reached on my own. Priorities I would have ordered differently. Use cases I’d underweighted. Specific pain points that, once I heard them described in the customer’s own words, made the product roadmap click into sharper focus. You can’t think your way to these refinements. You have to hear them from someone living the problem.
The language matters as much as the insight. When customers describe their pain, they use specific words and phrases. Those words are gold. They’re the exact language you’ll use in your marketing, your documentation, your PRFAQ, your positioning. I’ve pulled phrases directly from discovery calls into product descriptions, and the resonance with other customers is immediate.
It compounds. Every conversation builds on the last. By conversation five, you’re hearing patterns. By conversation ten, you’re testing hypotheses. By conversation fifteen, you’re refining nuances. By conversation twenty, you have a map of the problem space that no amount of desk research could produce.
| Activity | Time Investment | Information Value | Risk Reduction |
|---|---|---|---|
| Building features based on assumptions | Weeks to months | Low (confirms your biases) | Low |
| Reading market research reports | Hours | Medium (generalized, not your market) | Low |
| Analyzing competitor products | Days | Medium (their solution, not your customer’s problem) | Medium |
| Customer discovery interviews | 30-60 min each | Very high (specific, actionable, in their words) | Very high |
Nothing else gives you this return. The ROI on a single well-conducted discovery call dwarfs anything you could learn from a competitor analysis or a market report.
Customer discovery isn’t casual conversation. It’s a structured practice with specific artifacts and a repeatable process. Here’s what I’ve built over the last month.
Every discovery conversation follows a script. Not a rigid script that you read verbatim, but a structured guide that ensures you cover the important ground while leaving room for the conversation to go where it needs to.
My script has five sections, and I rarely get through all of them in one conversation. That’s fine. The first three are the most important.
Section 1: Their world today. How does your team build software today? Walk me through what happens from the time someone decides a feature needs to be built to the time it ships. What tools do you use? How many people are involved? How long does a typical feature take?
This section is pure observation. You’re building a mental model of their workflow. Don’t suggest improvements. Don’t react with “oh, we solve that.” Just listen and take notes. The details here are where the real insights hide. Not in what they say is painful, but in the workarounds they’ve normalized.
Section 2: Pain and friction. What’s the most frustrating part of that process? Where do things break down? What takes longer than it should? If you could fix one thing about how your team ships software, what would it be?
This is where customers start telling you things they haven’t articulated before. Often they’ll pause, think, and then say something that surprises even themselves. Those moments are the most valuable data points you’ll collect. Pay attention to energy shifts. When someone’s voice changes, when they lean in, when they start telling a story about a specific incident, that’s where the real pain lives.
Section 3: Prior attempts. Have you tried anything to solve this? What tools have you evaluated? What worked? What didn’t? Why did you stop using it?
This section tells you what you’re competing with (and it’s usually not another product; it’s a spreadsheet, a Slack channel, or a manual process that sort of works). It also tells you where the adoption barriers are. If someone evaluated a tool similar to yours and abandoned it, the reasons why are the most important thing you’ll learn all day.
Section 4: Success criteria. If a tool solved this problem for you, what would that look like? How would you know it was working? What would change about your team’s output or velocity?
This is where you start hearing what success looks like in their language. This maps directly to acceptance criteria in your specifications. I’ve pulled entire requirements from answers to this question.
Section 5: The demo (if time and rapport allow). Now that I understand how you work, let me show you what we’ve been building. Does this match the problem you described?
Only after you’ve done sections 1-4. The demo is dessert, not the main course. And you’re watching their reaction as much as listening to their words. Where do they lean in? Where do they look confused? What feature do they ask about that you didn’t show?
Every conversation gets logged in a structured tracker. Mine is a spreadsheet with these columns:
| Column | Purpose |
|---|---|
| Date | When the conversation happened |
| Name / Role / Company | Who you talked to |
| Company size / Stage | Context for their constraints |
| Current workflow | How they build software today |
| Primary pain | The #1 thing they’d fix |
| Secondary pains | Other friction points mentioned |
| Tools evaluated | What they’ve tried |
| Key quotes | Verbatim phrases that capture their thinking |
| Signals | Buying signals, adoption barriers, enthusiasm indicators |
| Follow-up | Next steps, intros offered, requests made |
The “Key quotes” column is the most important. When a VP of Engineering says “We spend more time arguing about what to build than actually building it,” that’s a quote you’ll use in your PRFAQ, your pitch deck, your website copy, and your specification pipeline. Capture it exactly as they said it.
After every conversation, I spend fifteen minutes filling in the tracker while the details are fresh. This is non-negotiable. Memory degrades fast. The nuance you think you’ll remember tomorrow is gone by next week.
Every five conversations, I write a summary. Not for anyone else. For myself. The summary forces me to synthesize patterns across conversations and update my understanding.
The summary answers four questions:
These summaries are artifacts that feed directly into the Working Backwards process. The patterns become personas. The pain points become use cases. The strongest signals become the PRFAQ’s opening paragraph. The key quotes become the customer testimonials in your FAQ section.
This is where customer discovery connects to everything I’ve written about spec-driven development and the Working Backwards pipeline. Discovery isn’t a standalone activity. It’s the first stage of a pipeline that ends with code.
The chain looks like this:
Discovery Conversations → Interview Tracker → Pattern Summaries →
Personas → Use Cases → PRFAQ → Feature Map →
requirements.md → design.md → tasks.md → Code
Every downstream artifact is grounded in something a real customer said. The persona isn’t a fictional character you invented in a workshop. It’s a composite of five real people you talked to, with their actual job titles, their actual pain points, and their actual words.
This is what I meant in The Missing Half of AI-Assisted Development when I wrote that vague product thinking produces vague specifications which produce vague code. Customer discovery is how you make product thinking precise. Not by being smarter or more creative, but by grounding every decision in evidence from the people you’re building for.
The traceability chain I described in my post on AI drift starts here. If you can’t trace a feature back through the specification, through the PRFAQ, through the use case, through the persona, to a specific customer conversation, that feature is at risk of drift at every layer.
| Pipeline Stage | Discovery Input | What It Produces |
|---|---|---|
| Personas | Role patterns, company stages, workflow descriptions | Composite customer profiles with real pain points |
| Use Cases | Workflow walkthroughs, “a day in the life” descriptions | Concrete interaction scenarios |
| PRFAQ | Key quotes, strongest signals, success criteria | Press release grounded in real customer language |
| Feature Map | Pain frequency, severity rankings, prior solution attempts | Priority-ordered feature list |
| Specifications | Success criteria answers, acceptance language | Testable requirements in EARS format |
Having done this wrong before doing it right, here are the failure modes I’ve seen and made.
Leading with the demo. I’ve covered this, but it bears repeating because the pull is strong. Demos are valuable, but they’re most effective after you’ve listened. Lead with discovery, and the demo lands harder because you know what to emphasize.
Asking leading questions. “Don’t you think AI-assisted development tools should integrate with your specification pipeline?” is not a discovery question. It’s a validation question wearing a discovery hat. Ask open-ended questions. “How do your specifications get from the product team to the engineering team?” Let them tell you.
Talking too much. The ratio should be 80/20 in favor of the customer talking. If you’re explaining your product for more than 20% of the conversation, you’re doing it wrong. I time myself. It’s humbling.
Ignoring the workarounds. When a customer says “oh, we just use a spreadsheet for that,” pay attention. That spreadsheet is your real competitor. Understand what it does, why it works, and what’s missing. The path from their spreadsheet to your product is your adoption strategy.
Not tracking systematically. The first three conversations are easy to remember. By conversation ten, details blur together. By conversation twenty, you’ve lost critical nuance. Track everything. Summarize regularly.
Stopping too early. Five conversations isn’t enough. You’ll think it is, because by conversation five you’re hearing patterns and feeling confident. That confidence is premature. Conversations six through fifteen are where the patterns get tested and the nuances emerge. Twenty conversations is where the map of the problem space starts to stabilize.
I want to be direct about something. Customer discovery is not a chore. It’s the most energizing part of building a company.
Every conversation teaches me something I didn’t know. Every customer has built workarounds and mental models that I never would have imagined. The creativity that teams apply to solving their own problems, with duct tape and spreadsheets and Slack bots and manual processes, is remarkable. Understanding those solutions tells you more about the problem than any abstract analysis ever could.
There’s a specific moment in most discovery conversations that I’ve come to look forward to. It happens when the customer stops answering your questions and starts thinking out loud. They’re no longer being interviewed. They’re working through their own problem, using you as a sounding board, and the insights that emerge in those moments are extraordinary. You can’t get there with a survey. You can’t get there with analytics. You can only get there by sitting with someone, asking the right questions, and giving them space to think.
After twenty-plus conversations this month, the product vision is sharper than it’s ever been. The 18 months of building gave me the technical depth to ask the right questions and recognize the significance of the answers. The discovery conversations gave me the customer language, the priority ordering, and the specific refinements that turn a strong thesis into a product people will pay for. The building and the discovery compound on each other.
Customer discovery is the highest-leverage activity an early-stage startup founder can do. It’s not a phase you complete and move past. It’s a practice you maintain throughout the life of the company.
The mechanics are simple. Talk to potential customers. Ask open-ended questions. Listen. Track everything. Synthesize patterns. Feed what you learn into your development process so that every feature traces back to a real customer conversation.
The principle is even simpler. Understand the customer before the customer understands you. Resist the urge to demo. Resist the urge to pitch. Resist the urge to validate. Seek to understand. The validation comes naturally when you build something grounded in what you heard.
I’ve spent 25 years building infrastructure and software at scale. I’ve shipped products used by millions of people. And the most valuable thing I’ve done in the last month is sit in conversations and listen.
If you’re building something and you haven’t done twenty discovery conversations, stop what you’re building and go do them. The product you build after those conversations will be fundamentally different from, and better than, the product you would have built without them.
Start with the customer. Work backwards. Everything else follows.
David Lapsley, Ph.D., is Founder and CEO of ShipKodeAI. He has spent 25+ years building infrastructure platforms at scale. Previously Director of Network Fabric Controllers at AWS (largest network fabric in Amazon history) and Director at Cisco (DNA Center Maglev Platform, $1B run rate). He writes about AI infrastructure, AI-accelerated SDLC, and the gap between POC and production.
[1] MIT NANDA Initiative, “The GenAI Divide: State of AI in Business 2025” (2025), link
[2] Lapsley, D., “The Missing Half of AI-Assisted Development” (2026), link
[3] Lapsley, D., “Amazon’s Working Backwards SDLC for SMBs” (2026), link
[4] Lapsley, D., “From PRFAQ to Backlog: Working Backwards as AI Pipeline” (2026), link
[5] Lapsley, D., “AI Drift Is a Product Problem, Not an Engineering Problem” (2026), link
[6] Lapsley, D., “Spec-Driven LLM Development (SDLD)” (2026), link
[7] Bryar, C. & Carr, B., Working Backwards: Insights, Stories, and Secrets from Inside Amazon (2021)
[8] Fitzpatrick, R., The Mom Test: How to talk to customers & learn if your business is a good idea when everyone is lying to you (2013)
]]>