Cognitive Surrender and Cogent Engineering

"Co-intelligence" to "Co-existence"

Last week, Anthropic said AI is close to building itself. Before this occurs, they say, we should pause AI to prevent dire consequences to humanity. The trend continues from before. We have heard this before in the form of a letter, signed by Elon Musk, who subsequently proceeded to accelerate Grok’s training schedule, instead of pausing. Anthropic’s IPO is around the corner, and so are OpenAI and SpaceX IPOs. This is a 4 trillion plus market event based on valuations that focus on a future of mankind where, supposedly, entire industries could be replaced or wiped out. The optimism surrounding AI companies has led to the devaluation of SaaS companies and their earnings multiples. Naturally, in leading up to an IPO, convincing both their investors and the larger public around how impressive this technology is key to Anthropic’s success, and they are doing an excellent job of it. Whether you fully subscribe to “AGI is coming in 2 years” or to “AGI is already here” - my thesis is that we’ll all intuitively know when AGI really arrives, but I don’t know as much about this as Jack Clark does, and his thoughts in the link here are definitely worth a read.

This post isn’t really about the speed at which AI is showing recursive self improvement, but about how humans seem to be succumbing to “cognitive surrender”, a term coming out of research from Wharton. This paper is an excellent read for everyone who loved reading “Thinking Fast and Slow” by Daniel Kahneman. In my own life, I have found these 2-system mechanisms of explaining human cognition to be somewhat limiting, but very useful in a leadership context. Often when trying to explain a prediction that I might make intuitively (System 1), I reach for my System 2 analytical methods to reason through so I could explain it to myself and to others. Sometimes this is hard when there are decisions to be made quickly. Other times, you could take your time with these decisions, and take the System 2 path entirely.

Well - the paper and research introduces a new system, System 3 (Artificial), which is an AI system that sits “in silico” and helps you with decisions. When there’s a conflict between System 1 & System 2, usually people review their actions based on a detailed analysis from System 2 and come back and adjust their intuitions. The flow is illustrated by Shaw and Nave (authors of the paper).

The key new term is “Cognitive surrender”, which is entirely a new phenomenon studied by the authors and the findings are astonishing. The participants, who didn’t know ahead of time whether it was going to be accurate or not, relegated their decisions to AI 93% of the time when it was deemed accurate and 80% when it was deemed faulty. The more interesting find is that System 3 facilitated cognitive surrender.

And when compounded by time pressure, AI accuracy benefited human cognition and decision making owing to the accuracy vs. speed benefits of AI.

What is the take away from all this? In my own experience, I’ve followed the principles of “Co-intelligence” as suggested by Ethan Mollick of: inviting AI to the table, being the human in the loop, treating AI like a person and assuming it’s at the worst point in its life cycle, I have been skeptical of the results from AI and worked to ensure accuracy. However, Agentic AI is changing this entire approach from Co-intelligence to “Co-existence”, where increasing amounts of decision making is being delegated to the AI vs. being a human in the loop.

Knowing all of the above, Agentic Coding is a place where cognitive surrender I believe has already happened very quickly, as the agent can verify the actions that it takes through a feedback loop. And when the AI is accurate, we have less of a need to go back and look at it. As long as it “just works”, we worry less about the internals of a device. It’s easier to surrender when you don’t really have enough expertise in an area and you’re looking for answers quickly.

The agentic AI thesis is that whole companies can be put out of business quickly. Let us say that I believe there’s a market need for a search engine that doesn’t use AI, however small that market may be. To expand this market hypothesis, in a recent experiment, I asked AI to create a simple search engine using PageRank and Bloom Filters. The goal of this search engine would be to create a very simple interface without any AI assisted search. While this is just me kicking around the tires to see if “goal oriented” AI recursion can do the job for me entirely. I’ve used a lot of search engines over the years, and worked with enterprise search engines such as Elasticsearch and Solr, and while I’m not a domain expert on scaling search engines, I wondered how far I could go. It was done with Antigravity & Gemini 3.5 Flash with some clear guidelines for the AI around engineering practices. I’m sure Claude or ChatGPT will claim that they could do a better job, but models and code agents are supposed to be commodities in the long run, so you can’t blame the consumer yet.

I accepted the implementation entirely. It was written in Go, and knowing that building the search index could take time, concurrency was a key focus. The AI knew and suggested how to build it using BoltDB, a fast key-value store implementation, inverted search index, and a bloom filter with mutex locks to ensure safe database operations. I tried the search engine out, and I saw some interesting base results. Why there’s a UFO reference in a note from 2001 on XML signatures by W3.org (a standards body for the world wide web) begs some questions, but I digress.

Then I dove into the code a little bit, and even though I again knew not enough about the details of a search engine, I found something interesting. To understand this further, a Bloom filter ideally uses multiple hash functions, but with the FNV hash, you can avoid having to do k hash functions that increase time complexity. One of the parameters of the FNV hash is a prime number. And when I looked at the code for this implementation, I saw this constant called

Anyone with elementary school math knows that’s not a prime number, so I asked the AI, “why is this a composite number?” and out came this answer.

The point of the above anecdote, in conjunction with cognitive surrender, is hopefully clear. Relegated decisions (cognitive surrender) such as the above (granted I’m not ready to give Google a run for their money yet) could have risks buried deep in the codebase. I’m sure a more seasoned domain expert in search probably could have more substantial findings. Spotting these could be another AI reviewer’s job, insofar as domain expertise has been extracted and embedded into an AI. With larger and larger LLMs, the hope is that it will be, but what about all the proprietary knowledge inside of orgs around domains? How much surrender has already happened inside of frontier labs like Open AI and Anthropic? No one outside could know concretely, but based on their articles, the indication significant percentage, especially if any company’s output raises by 8X in terms of software. Cognitive surrender is here, and we all have to prepare by asking ourselves questions about where we should, and where we should not. The answers to those questions will help guide people toward their purpose, and avoid building a false sense of confidence based on a surrendered self.