Copasetic Flow

Posts

W. E. D. Stokes - Ruddering From LLMs Back Towards Ham Radio

While doing research for a book I'm working on, The Gladych Files , I wondered into the weeds of statistical analysis of LLM AI agent performance which relates to my everyday sort of work in engineering. One of the things I really enjoy about The Gladych Files, however, is that it's never long before the project pulls me back towards ham radio. The statistical analysis project involved determining how often, and with what certainty AI agents could find out that Lucia Hobson was the daughter of Rear Admiral Richmond Pearson Hobson and then make the further link that Nikola Tesla was the best man at Rear Admiral Hobson's wedding. While estimating how difficult this was to do with plain old human operated web searches this morning, I came across W. E. D. Stokes! Stokes came into the picture as Lucia Hobson's husband. What I didn't know was that he was one of the founders of The Radio Club of America. His original interest in radio came from wanting to control a mod...

LLM Agent Research Protocol for Avoiding Stigmergy

I'm working through a methodology to study the behavior of teams of agents via observation of real-world tasks. As usual with LLMs, the concept of repeatable results is squishy, especially as compared to non-LLM deterministic computing. My finding last week was that LLM agents, especially Claude (per Google's research), can exhibit stigmergic , (a fancy word for how insects, like ants, 'learn' where important locations are from other insects), learning and behavior. In short, agents given the exact same instructions, (prompts), can and often times will exihibit different behaviors if they can see the results of the work of other agents. If you want to study the variance in the behavior of an LLM agent over multiple runs, this stigmergic behavior has to be accounted for. Otherwise, we're not measuring the behavior of an LLM agent with a set of inputs and prompts. With stigmergic behavior, if we're not careful, we're observing the behavior of a community of ...

fable-5 down for now per US Government Directive

It was fun getting to use Anthropic's Fable-5 for a few days. Hopefully the chance will come up again. For the moment, the US government has denied access to non-US citizens.

LLM Evals Lab Book: The Importance of Statistics and Also Stigmergy

Recap During an analysis of a travel manifest, two agents, (referred to as polecats in Gastown terminology), were accidentally handed the same manifest page for input. The agents produced different results. One agent found an association between Lucia Hobson and Nikola Tesla, a very valuable association for the research project. The other agent did not. A set of eval experiments ensued to determine how often polecats missed the association. The initial answer was that they missed it quite frequently with only 3 out of 16 agents making the association. Models Used In the following, all agents are using Sonnet 4.6. Orchestration is handled with Gastown. New Findings On the fourth batch of five test case runs, four polecats made the Tesla association. The chances of this happening randomly were less than 3% in the absence of any other process changes. Here's the Fisher's test run by Gemini. Fisher's Exact Test (Recommended) This compares your two distinct groups (the past 16...

Can Agents Think Outside the Box?

With all the work that's been put into making agents "correct" by construction, I gotta say, sometimes I need an LLM agent to take a chance at just being wrong. I'm working on a book project called The Gladych Files . While the book is narrative nonfiction about the history of general relativity research, it explores the liminal space inhabited by very rich fringe scientist speculators of the 1950s who funded mainstream general relativity advances, (more or less on accident.) In those spaces, you'll find Tesla, the architect of the FBI building, Timothy Leary's LSD explorations and many, many other things, institutions, and people. I've accumulated hundreds of pages of historical documents from various archives, and I'm using orchestrated agentic AI, (in the form of Gastown), to review those documents. So far, the analysis has gone well, but last week I saw something that made me look up. I'd accidentally input the same archive page twice, so i...

Gladych Files Lab Book: Document OCR vs LLM Model vs Cost or Opus is Cheaper than Sonnet for OCR!

I started my lab book entries when I was a physics graduate student. It's kind of amusing and kind of cool how far I've come. I have the equivalent of a grad student, (aka Claude Opus 4.7), working for me now. I spent some time over the weekend setting up an OCR framework for a book research project of mine. I've been coming up to speed on evals, so I decided to run one to determine which model was the most accurate and cost effective for doing OCR on travel manifest pages. I stepped the eval along rather than automating it and talked the results through with Opus as I went. First, it turns out that Opus at low effort is the most accurate and the most cost effective choice! That was a surprise. The result has to do with Opus' ability to look at higher res images which means it needs to think less for OCR vs. Sonnet. Second, at the end of the eval, as I was preparing to write up my results it occurred to me that I could ask my grad student to do it instead. Here's...