Copasetic Flow

 Antrhopic's Haiku model can routinely pass —about 300 times so far--the extra class US ham exam based solely on what's contained in its model. (The LLM was not permitted, programatically I might add, to access the interent and therefore the question pool .) I'm studying this to reinforce my grad school business statistics class for use in writing agentic evals. Yesterday, I reasoned that the order the questions were asked might influence the results since the single LLM—essentially a single chat session—might get discouraged if it missed the first question and then project onto a 'poor test taker' part of its model space. I found out I was half right. The question order does seem to matter, but instead of influencing the entire exam, it influenced the results on the first question asked. I ran three 90 exam test fllets, one each with a first question that Haiku always gave the correct answer, the wrong answer, or a mixed result (sometimes correct, sometimes wrong...

 I'm still seeing Sonnet fan work out beyond its model capablities using subagents. You can see in the window below that the Gas Town polecat has had to stop and compact its context before proceeding. The good news is that when these subagent work fans don't turn into storms, I'm seeing more thorough reseearch findings returned. The bad news is that with Sonnet's limited context, those findings occasionally overwhelm the polecat that started it all.  I found a preprint from MIT for using what the authors call recurive language models or RLMs. The general idea is basically what I'm seeing with subagents—split the task, split the context—but they have clever ideas about how to manage the output stages to avoid the exact overflows shown above. I'm still harboring my hunch that this all started with the advent of Sonnet 5. Anthropic's release notes include "Sonnet 5 is much more agentic than its predecessors. Testers described how it finishes complex tas...

 Working with Fable last week, I noticed that it can get very excited about fanning out tasks, (to the point of crashing my WSL session.) I experimented with prompting agents to split out tasks on their own, but haven't found a successful to do this yet. Consequnetly, I reverted back to my original prompt which does not explicitly call for task splits. Then, this morning, running on Sonnet 4.6, another task storm sprung up. I wonder if the new 50% higher token limit  till August is Anthropics way of buying themselves some head  room while they experiment with models creating subagents?  Are other  people seeing the same thing? It's particularly worrisome that Sonnet 4.6 has started a subagent storm becuase it really doesn't ahve the context to deal with the results.

 Thijs is new prior to today, usage was broken out between Sonnet and other models like this It's unclear ot me if I tripped a flag somewhere as I approached my weekly limit, or this dialog is reflective of a new Anthropic usage report. I switched Gas Town over to all Sonnet models a few weeks ago for two reasons. First, to get more usage per week. Second, I've found that for the creative research work I have the polecats (LLM agents in Gas Town parlance), Sonnet works much better than Opus 4.8. The research is for a book detailing the funding of mainstream general relativity by fringe science industrialists in the 1950s. It requires polecats to, for example, see the name Lucia Hobson, and immediately jump to the fact that Nikola Tesla was the best man at her father's wedding. Thus far, Opus 4.8 has been a little to stick-in-the-mud to pull this off, but Sonnet 4.6 makes the association for a lower token cost. Using Codex Instead I switched over to codex to run research an...

 It was fun getting to use Anthropic's Fable-5 for a few days. Hopefully the chance will come up again. For the moment, the US government has denied access to non-US citizens.