Skip to main content

Deploying a ChatKit Demo for PsyOps Detection

 I deployed the LLM Psy-ops detection app earlier today! For those of you just hopping onboard, the WhyFiles ran an episode highlighting a simple, logical scoring method publicized by NCI for determining if a piece media or new article was emotionally manipulative, (think propaganda), or not. 

I was looking for a good app to practice deployment, guardrails, and evals, and this one suggested by a @somethingLethal on reddit seemed promising in all those regards. If you'd like to try it, you can find the app at https://projecttoucans.com/gladych_files_psy_ops . 

LLMs, Simple Math, and Pricing

The Psy-op scoring instrument requires that the model sum the scores for the twenty categories. gpt-4o-mini did not sum any of the scores correctly. It got close, but that was about it. I experimented with the python code interpreter to cure the simple math issue.

The code interpreter seemed reasonable at first. I mean, three cents per compute minute, not bad right? Instead though, it turned out to be about 3 cents per use of the page. That didn't even come close to being affordable. When I queried GPT-5 about this further, it agreed. Each page load was causing a server startup which cost about three cents.


The real fix wound up being to switch the model to gpt-5-nano. It costs less than o4-mini across the board and does better math. (All the sums since the removal of code intereter and the model switch have been correct.)



Useful Features and ChatKit Privacy

I thought hard about not using ChatKit at all. As I mentioned yesterday, not using it would cut down on guardrail verification and reduce the attack surface area of the app. However, the history feature of ChatKit was too useful to pass up.


As long as the you're using the same browser, ChatKit will wake back up with a list of your already analyzed news articles. That's pretty cool, and development-time-wise it was almost free.

Earlier in the day, I had the code beloww setup so that all users were denoted as "developer" which would have resulted in everyone seeing everyone else's history along with their own. A little bit of code to create a random string per user fixed this easily enough, and  I was back off and running.

  // Optional: pass a user/device id to help scoping server-side
  // (you can remove this if not needed)
  //body.user = "localdev";
  if(user) body.user = user;


Guardrails and Evals

I'm hoping to use the app to get a better feel for deploying and testing real world guardrails on an AI enabled ChatKit (ala AgentKit) application. Hopefully by the end of the week I'll be running evals on this project. It looks like what I'll be doing is wiring the prompt into a CLI python or JavaScript server to run regressions on it. I'll keep you posted as the week proceeds.


Videos


Labels:

Comments

Post a Comment

Please leave your comments on this topic:

Popular posts from this blog

Cool Math Tricks: Deriving the Divergence, (Del or Nabla) into New (Cylindrical) Coordinate Systems

Now available as a Kindle ebook for 99 cents ! Get a spiffy ebook, and fund more physics The following is a pretty lengthy procedure, but converting the divergence, (nabla, del) operator between coordinate systems comes up pretty often. While there are tables for converting between common coordinate systems , there seem to be fewer explanations of the procedure for deriving the conversion, so here goes! What do we actually want? To convert the Cartesian nabla to the nabla for another coordinate system, say… cylindrical coordinates. What we’ll need: 1. The Cartesian Nabla: 2. A set of equations relating the Cartesian coordinates to cylindrical coordinates: 3. A set of equations relating the Cartesian basis vectors to the basis vectors of the new coordinate system: How to do it: Use the chain rule for differentiation to convert the derivatives with respect to the Cartesian variables to derivatives with respect to the cylindrical variables. The chain ...
10 comments
Read more

The Valentine's Day Magnetic Monopole

There's an assymetry to the form of the two Maxwell's equations shown in picture 1.  While the divergence of the electric field is proportional to the electric charge density at a given point, the divergence of the magnetic field is equal to zero.  This is typically explained in the following way.  While we know that electrons, the fundamental electric charge carriers exist, evidence seems to indicate that magnetic monopoles, the particles that would carry magnetic 'charge', either don't exist, or, the energies required to create them are so high that they are exceedingly rare.  That doesn't stop us from looking for them though! Keeping with the theme of Fairbank[1] and his academic progeny over the semester break, today's post is about the discovery of a magnetic monopole candidate event by one of the Fairbank's graduate students, Blas Cabrera[2].  Cabrera was utilizing a loop type of magnetic monopole detector.  Its operation is in...
1 comment
Read more

More Cowbell! Record Production using Google Forms and Charts

First, the what : This article shows how to embed a new Google Form into any web page. To demonstrate ths, a chart and form that allow blog readers to control the recording levels of each instrument in Blue Oyster Cult's "(Don't Fear) The Reaper" is used. HTML code from the Google version of the form included on this page is shown and the parts that need to be modified are highlighted. Next, the why : Google recently released an e-mail form feature that allows users of Google Documents to create an e-mail a form that automatically places each user's input into an associated spreadsheet. As it turns out, with a little bit of work, the forms that are created by Google Docs can be embedded into any web page. Now, The Goods: Click on the instrument you want turned up, click the submit button and then refresh the page. Through the magic of Google Forms as soon as you click on submit and refresh this web page, the data chart will update immediately. Turn up the:...
Post a Comment
Read more