Skip to main content

Posts

Showing posts with the label AI Evals

Lab-Book 2026-07-19 How to Get Your Thinking Time back from LLMs

 I've seen a few posts in the last week that talk about engineer's loosing their time to think because they're too busy watching what their LLM agents are doing. Anthropic's models, even under GasTown, have decide to spontaneously fork and find other ways to farm work out to subagents, occasionally causing agent storms on my project in the last week, so I can sympathize a bit more now that I did at the start of the week. There are a few solutions I'd like to propose for this issue. Here goes. How to Get Your Thinking Time Back First, if you're agents are doing marginally what you want, this is pretty simple. Stand up, back away from your desk, and leave . If it makes you feel better, setup a notification so the agent messages you when it's done. Perhaps, something like this If you don't trust your agents for the moment, still don't watch them. Come up with metrics to know if they worked or not and control your agents or modify their activities AUTOM...

Can Agents Think Outside the Box?

 With all the work that's been put into making agents "correct" by construction, I gotta say, sometimes I need an LLM agent to take a chance at just being wrong. I'm working on a book project called The Gladych Files . While the book is narrative nonfiction about the history of general relativity research, it explores the liminal space inhabited by very rich fringe scientist speculators of the 1950s who funded mainstream general relativity advances, (more or less on accident.) In those spaces, you'll find Tesla, the architect of the FBI building, Timothy Leary's LSD explorations and many,  many other things, institutions, and people.  I've accumulated hundreds of pages of historical documents from various archives, and I'm using orchestrated agentic AI, (in the form of Gastown), to review those documents. So far, the analysis has gone well, but last week I saw something that made me look up. I'd accidentally input the same archive page twice, so i...

Deploying a ChatKit Demo for PsyOps Detection

 I deployed the LLM Psy-ops detection app earlier today! For those of you just hopping onboard, the WhyFiles ran an episode highlighting a simple, logical scoring method publicized by NCI for determining if a piece media or new article was emotionally manipulative, (think propaganda), or not.  I was looking for a good app to practice deployment, guardrails, and evals, and this one suggested by a @somethingLethal on reddit seemed promising in all those regards. If you'd like to try it, you can find the app at  https://projecttoucans.com/gladych_files_psy_ops  .  LLMs, Simple Math, and Pricing The Psy-op scoring instrument requires that the model sum the scores for the twenty categories. gpt-4o-mini did not sum any of the scores correctly. It got close, but that was about it. I experimented with the python code interpreter to cure the simple math issue. The code interpreter seemed reasonable at first. I mean, three cents per compute minute , not bad right? Ins...