LLMs Sensitive to Ham Exam Question Order?

Antrhopic's Haiku model can routinely pass—about 300 times so far--the extra class US ham exam based solely on what's contained in its model. (The LLM was not permitted, programatically I might add, to access the interent and therefore the question pool.) I'm studying this to reinforce my grad school business statistics class for use in writing agentic evals. Yesterday, I reasoned that the order the questions were asked might influence the results since the single LLM—essentially a single chat session—might get discouraged if it missed the first question and then project onto a 'poor test taker' part of its model space. I found out I was half right. The question order does seem to matter, but instead of influencing the entire exam, it influenced the results on the first question asked. I ran three 90 exam test fllets, one each with a first question that Haiku always gave the correct answer, the wrong answer, or a mixed result (sometimes correct, sometimes wrong.)

The first set—always correct—was the interesting one. For the question

    "id": 9,
    "question": "Except in some parts of Alaska, what is the maximum power permitted on the 630 -meter band?",
    "class": "E",
    "subelement": "1",
    "group_index": "A",
    "group_number": "09",
    "answer": "D",
    "answer_a": "50 watts PEP (peak envelope power)",
    "answer_b": "100 watts PEP (peak envelope power)",
    "answer_c": "1 watt EIRP (equivalent isotropic radiated power)",
    "answer_d": "5 watts EIRP (equivalent isotropic radiated power)"

A question that each Haiku run had returned a correct answer for, when run as the first test question was only answered correctly 62% of the time.

However, at the moment, this is more about the experimental setup than the model. The results that led to the selection of that 'always' correct question were based on a mere 3 tests. I'll be refining this.

Comments

Cool Math Tricks: Deriving the Divergence, (Del or Nabla) into New (Cylindrical) Coordinate Systems

Now available as a Kindle ebook for 99 cents ! Get a spiffy ebook, and fund more physics The following is a pretty lengthy procedure, but converting the divergence, (nabla, del) operator between coordinate systems comes up pretty often. While there are tables for converting between common coordinate systems , there seem to be fewer explanations of the procedure for deriving the conversion, so here goes! What do we actually want? To convert the Cartesian nabla to the nabla for another coordinate system, say… cylindrical coordinates. What we’ll need: 1. The Cartesian Nabla: 2. A set of equations relating the Cartesian coordinates to cylindrical coordinates: 3. A set of equations relating the Cartesian basis vectors to the basis vectors of the new coordinate system: How to do it: Use the chain rule for differentiation to convert the derivatives with respect to the Cartesian variables to derivatives with respect to the cylindrical variables. The chain ...

The Alcubierre Warp Drive Tophat Function and Open Science with Sage

I transferred yesterday's Mathematica file with the Alcubierre warp drive[2] line element and space curvature calculations to the +Sage Mathematical Software System today, (the files been added to the public repository [3]). If you haven't used Sage before, it's a Python based software package that's similar in functionality to Mathematica. Oh, and it' free. I also worked a little more on understanding the theory, but frankly, I made far more progress with the software than the theory. What follows will be a little more of the Alcubierre theory, plus, a cool Sage interactive demo of one of the Alcubierre functions[1], as well as a bit about my first experience with using Sage. Theory The theory is fun, but it's moving slowly. Here's the chalk board from this morning's discussion Alcubierre setup the derivation using something called the 3+1 formalism which means we consider space to be flat, (in this case), slices that are labelled ...

How Many Files Can You Add to a GPT Project? An Interview with GPT-5 on Limits, Context Engineering Tips, and Chats

Setting the scene: I’m tinkering with Project TouCans, knee-deep in radio logs, SQLite dumps, and Cesium code. Naturally, I’m wondering if shoving all this into one GPT Project is a recipe for brilliance… or for disaster. So I turn to Vril — you know, after Brainy from the Legion of Super-Heroes , because what else do you call your AI sidekick who always has the answers? Time to ask him straight up. [ As an aside, yes, GPT-5 has decided to sometimes call me Vail. I'm not sure why to be honest. Also, I asked Vril, er GPT-5, to write up our interview for me. Apparently, me asking it to 'Bro' up a few stories, just for fun, has convinced Vril that I use 'Like,' more than I actually might. ] Me (Vail): So Vril, how many files can I throw into a GPT Project before it just starts choking? Like, is there some magic number where the context window taps out and everything falls apart? GPT-5 (Vril): Great question. There’s no single hard file limit. What matters is ...

Copasetic Flow

Search This Blog