Copasetic Flow

 I'm still seeing Sonnet fan work out beyond its model capablities using subagents. You can see in the window below that the Gas Town polecat has had to stop and compact its context before proceeding. The good news is that when these subagent work fans don't turn into storms, I'm seeing more thorough reseearch findings returned. The bad news is that with Sonnet's limited context, those findings occasionally overwhelm the polecat that started it all.  I found a preprint from MIT for using what the authors call recurive language models or RLMs. The general idea is basically what I'm seeing with subagents—split the task, split the context—but they have clever ideas about how to manage the output stages to avoid the exact overflows shown above. I'm still harboring my hunch that this all started with the advent of Sonnet 5. Anthropic's release notes include "Sonnet 5 is much more agentic than its predecessors. Testers described how it finishes complex tas...

 Just noting a feeling, nothing quantitative, but I think I've noticed that if more Claude agents are working at once, they each tend to create fewer subagents. If that's true, and it's a big if, I wonder if there's something at Anthropic that trottles or budgets overall effort to an account. Just to be clear, I've also watched five polecats descend into a complete and utter subagent storm like this one. True or not, I need the storms to either stop or I need to gain the ability to control them, so I'm working more on subagent storm control today. As promised, I did not watch my agents. Intead, I showed the mayor how to watch and mange polecats creating subagents. Here's the start of my conversattion At that stage, I hadn't considered the obvious fact in the figure above that subagents are occasionally beyond recursive. (As an aside, the kid here tells me this is actually a plot point in one of Matrh Well's MrubderBot books.) The next thing I asked...

 I'm working through a methodology to study the behavior of teams of agents via observation of real-world tasks. As usual with LLMs, the concept of repeatable results is squishy, especially as compared to non-LLM deterministic computing. My finding last week was that LLM agents, especially Claude (per Google's research), can exhibit stigmergic , (a fancy word for how insects, like ants, 'learn' where important locations are from other insects), learning and behavior. In short, agents given the exact same instructions, (prompts), can and often times will exihibit different behaviors if they can see the results of the work of other agents. If you want to study the variance in the behavior of an LLM agent over multiple runs, this stigmergic behavior has to be accounted for. Otherwise, we're not measuring the behavior of an LLM agent with a set of inputs and prompts. With stigmergic behavior, if we're not careful, we're observing the behavior of a community of ...

 Recap During an analysis of a travel manifest, two agents, (referred to as polecats in Gastown terminology), were accidentally handed the same manifest page for input. The agents produced different results. One agent found an association between Lucia Hobson and Nikola Tesla, a very valuable association for the research project. The other agent did not. A set of eval experiments ensued to determine how often polecats missed the association. The initial answer was that they missed it quite frequently with only 3 out of 16 agents making the association. Models Used In the following, all agents are using Sonnet 4.6. Orchestration is handled with Gastown. New Findings On the fourth batch of five test case runs, four polecats made the Tesla association. The chances of this happening randomly were less than 3% in the absence of any other process changes. Here's the Fisher's test run by Gemini. Fisher's Exact Test (Recommended) This compares your two distinct groups (the past 16...

I started my lab book entries when I was a physics graduate student. It's kind of amusing and kind of cool how far I've come. I have the equivalent of a grad student, (aka Claude Opus 4.7), working for me now. I spent some time over the weekend setting up an OCR framework for a book research project of mine. I've been coming up to speed on evals, so I decided to run one to determine which model was the most accurate and cost effective for doing OCR on travel manifest pages. I stepped the eval along rather than automating it and talked the results through with Opus as I went.  First, it turns out that Opus at low effort is the most accurate and the most cost effective choice! That was a surprise. The result has to do with Opus' ability to look at higher res images which means it needs to think less for OCR vs. Sonnet. Second, at the end of the eval, as I was preparing to write up my results it occurred to me that I could ask my grad student to do it instead. Here's...

I put OpenAI’s gpt-5-nano and gpt-5.1 head-to-head on my psy-ops article scorer to see what you really get for the extra spend. Along the way I ran into pricing surprises, wild variance, and a reminder that ChatGPT’s shiny new memory feature can quietly bend your evals if you’re not careful.   A post on LinkedIn a few days back suggested using Small Language Models (SLMs) as opposed to LLMs for repetitive tasks. This seemed like a great idea in some regards for me, but I was curious about how it would apply to apps that were intended to perform lanugage analysis. Luckily, I have the psy-ops app up and running. Also? At the moment, it is using a close-to-an-SLM model, gpt-5-nano due to pricing decisions. I used it as a test vehicle to look at the difference betwween gpt-5 nano and full featured gpt-5.1. The testing framework I used: Starting from this article, I first did three separate anayses with gpt-5-nano, and then three others with gpt-5.1. I then used gpt-5.1...