GenXJamerican

This Tom's Hardware piece, and the timeline of Jer Crane (CEO of PocketOS) correctly place accountability for the recent destruction of his company's production data with Cursor, Claude, and Railway, but do not include Jer Crane himself. The phrase a computer can never be held accountable is even truer today than it was when first written in an internal IBM training manual back in 1979 because of how much more capable computer systems are today. Crane's timeline is valuable whether you're an engineer building systems that leverage AI, an executive selecting dependencies, or an engineering manager between them. This piece will explore those details to model a post-mortem with full accountability.

Crane lays out the stakes of his software working as-designed clearly:
"We build software that rental businesses — primarily car rental operators — use to run their entire operations: reservations, payments, customer management, vehicle tracking, the works. Some of our customers are five-year subscribers who literally cannot operate their businesses without us."
A single API call destroyed business-critical data for every PocketOS customer in 9 seconds. Railway's architecture storing volume-level backups in the same volume meant all the backups of that data were also deleted. They experienced what every business that uses AWS us-east-1 experiences when that region has an outage--but worse, because current backups no longer existed.

Crane includes the agent's "confession" in his post-mortem. Reading it in full reiterates numerous ways in which people granting an AI system both a broad scope of action and control over its own guardrails violates the principle of separation of duties with catastrophic results. Talking about Cursor's safeguards failing or the difference between Cursor's marketing and the reality of the tool in action doesn't change the fact that "a computer can never be held accountable". Buying access to Claude's most expensive model and configuring it "with explicit safety rules in our project configuration" integrated with Cursor does not transfer his share of accountability to Cursor, Anthropic, or Railway. Where specifically did the post-mortem fall short?

Railway's documentation on data & storage backups has a Caveats section which describes backups as "a newer feature that is still under development". Given this, a post-mortem that talked about migrating to managed database outside of Railway to enable multi-region database failover as a way to mitigate the risk of that feature would highlight the platform shortcoming and demonstrate a CEO taking proactive steps to make his service offering more resilient.

Per Crane's timeline, the agent encountered a credential mismatch while working on a routine task in a staging environment. The post-mortem doesn't explain what this routine task was, or why a credential mismatch led to volume deletion rather than a failed operation and a logged error. The lack of additional detail on this is an accountability gap for the PocketOS team, and ultimately the CEO. The same tool being used to make staging environment changes, generate code on its own, help engineers develop code, review it, and deploy it constitutes a separation of duties (SoD)failure. Before GenAI, my prior company used a GitHub bot to require different engineers to review and approve PRs for merge than those who implemented the change. If a team failed to install that bot on their repos, any SoD violations detected by the CI/CD pipelines would block production deployment until the violation was resolved or an accountable executive agreed to take responsibility for any production issues resulting from that deployment. Multi-layer SoD enforcement mechanisms are still needed even with GenAI.

From Railway's API documentation, the token created "to add and remove custom domains via the Railway CLI" was an Account token--the only type with "all resources and workspaces" scope. A post-mortem which pointed out the different available scopes and replacing that token with a Workspace or Project token scoped to staging would create a balanced picture of the CEO being accountable and reinforce his good points regarding the absence of feedback and human-in-the-loop opportunities in the token creation flow. Railway's token types are not as granular as those offered by classic personal access tokens or fine-grained tokens in GitHub, but they do provide the limited scope which limits the blast radius of misuse (accidental or intentional). The API tokens aren't time-limited, so a long-term step the CEO might recommend in the post-mortem is a Railway feature providing an expiration option as a security measure customers can choose to use. Crane's post-mortem does a lot of things right--but the missing piece is himself. Cursor, Anthropic, and Railway need to be accountable--but so does he.

Regardless of the field of endeavor, we need to take accountability for the outcomes of actions taken by generative AI systems. Here's what that looks like in practice:

• Build guardrails that people control, instead of delegating that responsibility to AI.
• Apply the principle of separation of duties anywhere AI agents act independently.
• Use environment-scoped access so that credentials which work in non-production environments don't work in production environments (and vice versa).
• Govern AI agents' scope of action by the principle of least privilege--just as we should for people.
• Require generative AI to ask permission for every change, even in your personal use--this shifts accountability back to people.