Copasetic Flow

 I made my first ever pull request on GitHub this morning! The gang and I still mostly develop code on Windows boxes. It has more to do with the machines we have available to us on any given day than anything else. (The just-sit-down-and-type machine in our house is also for playing video games, so... Windows.) This has led to issues when using our Datasette enabled QSO logger and mapping applications .  Datasette on Windows has an issue with finding the SpatiaLite DLL file. The gang and I worked through the issues and documented our really clunky, blunt force, fix . The fix, however, amounted to changing the source code of Datasette itself on a per installation basis. That meant that if I moved to a new Windows machine I had to find the fix and re-implement it. Even worse, that meant that if I wanted to try something that worked only in a newer version of Datasette, I also had to reimplement the fix. That... Was a Lot. So, I was totally psyched when I got a little bit of ti...

 I started a project, ahem, yesterday to 'quickly' see if ChatGPT could read the entire United States general class amateur radio exam question pool into a Datasette instance using the datasette-extract plugin. As of this morning, I haven't been able to coax ChatGPT using the gpt-4-turbo model. I capture my rather raw notes below. The short version of this is that I was never able to get the AI to capture more than 19 questions at a time. I'm hopeful that the pool could be moved into a database table using iterative processes, but for now, I've run out of time for this quick project :)  Occasionally ChatGPT seemed to hallucinate out part of its process into the table Notes Follow I'm going to track how easy it is to get the general exam question pool into a database using the Datasette Plugin. I started this endeavor at 20:37 UTC. Get my already existent OpenAI API key ready to go 20:43: Done. As usual with OpenAI, the hardest part was finding login screens ...

I'm starting to be handy enough with Datasette that I can quickly get answers to my questions! This morning, the question is whether or not the Tuna Topper amplifier combined with the Rockmite is giving better propagation results than the we get with the Rockmite running alone. Since we have a database of all the QSOs and RBN spots of the Rockmite, this is exactly the kind of question Datasette can answer. Here are the commands to run datasette to get an sqlite database from the Rockmite's RBN and QSO data table . I'm listing the commands here because I've been to lazy to simply write a script, and thus far I haven't been able to get the whole assmeblelage to work on Windows, meaning I'm working on a github codespace. Codespaces are super-cool, but also—at least for me... so far—completely unable to keep command history around between launches python -m pip install csvs-to-sqlite python -m pip install pandas==1.5.0  csvs-to-sqlite -dt timestamp rm_rnb_history...

The 12 year-old here, aka Daize in these pages, and I have been teaching ourselves Datasette, a Python based tool for exploring data sets. We came across the tool the first time as members of the San Francisco Microscopical Society when we attended a meeting about their historical papers database. Datasette , a tool created by Simon Willison is being used to serve their historical document database . It looked like a nice tool to know, but I didn't have an immediate use for it until last week when I got a little bit of bandwidth to setup new help page links for the ham radio practice exam app . I rooted around in the Google App Engine dataset pages a bit, then I played with making pages as templates in the Django based app, (did I mention that Simon was also one of the creators of Django?), and then it occurred to me that Datasette might be the most useful way of inspecting and modifying the help page links. In any event, the kid and I would be learning something new. The short ve...

 I had to take a few weeks' break before this one made sense to me. It finally did though, and now when the gang and I look at our QSO map in Datasette, we can see which outing generated a QSO by clicking on the marker of the receiving ham radio station. So, for example, if we're wondering when we finally made that recent QSO with Hawaii? Oh yeah, it was during the California QSO party! We've still got some data entry to do—if I had more tactical time at the moment, I'd use the datasette-write package and a form to make the entries more quickly... Anywway—but eventually, we'll have a pretty slick map. The technical details As for the details of what finally got everything to work for me (on a Windows machine.) You'll need datasette datasette-cluster-map datasette-media datasette-json-html and a metadata.json file. (This was the bit that just kind of hurt me deep down and slowed me up for a few weeks. Once I accepted that there was no way around, and then spent ...

 Reading W6CSN's post on his recent POTA at Mt. Tamlpais got me thinking. How many QSOs had I made from the region, and had I sent out QSLs for all of them, and had I updated our log database to reflect those QSLs? We have a mapable QSO log, so clearly, I could just zoom in and out on the map, but I wanted to do something that felt fancier. I remembered a Simon Willison  blog  posts about Datasette mentioning there was a plugin that allowed querying of databases based on regions drawn on a map. That's the thing I wanted. Actually Doing the Thing So, I wanted to be able to display a map,  draw a region on the map, and then review the QSOs I'd made from that region. Armed with a map of QSOs, I also wanted to be able to click each QSO to see if I'd updated the QSL photo for the occasion in our QSO database.  After working though the issues I discuss below, I was delighted to be able to search for our QSOs from the Pantoll Campground Region like so: Now that we've s...

 I needed to learn Jinja and/or brush up on my Django skills for work. I decided to go the Jinja route because that template engine is used in other tools at work, plus I could write a Datasette plugin while I learned my way around. It's always bugged me a bit that map_qsos.py programatically writes out the kml files that I use to display our ham radio outings on Google Earth. Print statements are the wrong tool for the job, which—at the end of the day—is to create the same kml lines and points substituting in different station names and locations in a loop. The process fairly screams template engine. Another issue is that even though the data is available in a database after its initial entry, with the previous usage model, I had to collect the versions of qso_update.csv I wanted to map and then combine their calls into a single file. This file was then operated on by map_qsos.py. The whole process was cumbersome, and did not lead to me wanting to explore maps of the log unless t...

 We have new software projects to play with here at the KD0FNR/KO6BTY ham shack in December. First, Datasette announced a new toy to play with! Enrichments If you've been regularly reading, or watching the repository for the logging software the kids and I have been building for oursevles, you've seen me mention Datasette before. It's a tool for browsing data sets using SQLite, and then applying various analyses to the data. The kids and I have used it to map  QSOs in a number of different ways. Simon Willison announced that the tool now feautres something called enrichments . What do they do? In short, they allow the user to apply operations to data that can subsequently be written back into a database. The post linked to above demonstrates how enrichments work using a geocoding example. The example uses OpenCage to geocode addresses. We've been using the Google Maps API here to do a similar thing, so the first obvious project is  Creating a Datasette enhancement th...

 The gang—the 12, 11, and 8 year-old kids whose dad I am—and I attended Simon Willison's Datasette office hours on Friday to talk about our ham radio Datasette projects and to start a bit of debug on a Datasette geocoding  enrichment we're working on. During our meeting, Simon asked what the longest QSO we'd made with the radio was, and mentioned that the datasette-haversine  plugin can be used to calculate the distance between two locations on the Earth. Coincidentally, I'd already been looking into this in relation to the SKCC 1,000 Watts per Mile award. Consquently, I set out to apply the Haversine to our QSO logbook in Datasette to find the best set of QSOs to send in with our award application. KO6BTY and I did an initial glance on Friday after speaking with Simon. Here's the query we used: select rowid, id, tx_lng, tx_lat, rx_lng, rx_lat, haversine(tx_lat, tx_lng, rx_lat, rx_lng, 'mi') as haversine, floor((((haversine(tx_lat, tx_lng, rx_lat, rx_...

 As with most of my project, I'm probably having way too much fun with this elevation path thing! I got a new look at our usual Pantoll campground site this morning, and it's really, really cool!  First though, let me point you towards the code that enables my recent fascination with our the layout of the terrain at our POTA activations. I've worked the datasette plugin that returns the coordinate's of a portion of the path between two points code through it's first very simple test case. I do not know if it will install correctly in any installation of Datasette yet, but it install's in mine, and it plays well with the instance in the test case, so here's the repository . Note for new readers: Datasette is a Python tool from Simon Willison , described as: " Datasette is a tool for exploring and publishing data. It helps people take data of any shape, analyze and explore it, and publish it as an interactive website and accompanying API. " The gang ...

 I'm mostly writing this so that the next time I run into this, I'll have a handy set of notes. Many months ago, I found out that Datasette on Windows doesn't like this :             for extension in self.sqlite_extensions:                 # "extension" is either a string path to the extension                  or a 2-item tuple that specifies which entrypoint to load.                 if isinstance(extension, tuple):                     path, entrypoint = extension                     conn.execute("SELECT load_extension(?, ?)", [path, entrypoint])                 else:                     conn.execute("SELECT load_extension(?)", [extension]) in...

 I've outlined the work on the ham radio QSO Google Maps Elevation API Datasette Enrichment  (try saying that 3 times fast) over the last several days.  What I haven't talked about much, but thought about pretty often is what the released elevation enrichment should do. Here goes. The Datasette enrichment will store the JSON data returned from the Elevation format in a database column named 'elevation'. Remember, the enrichment is taking two coordinate pairs that form a QSO like the brown line shown between Villa Nueva, NM and Idaho (KJ7DT's QTH). It then determines the endpoint of a line that only covers the first 200 meters of that path, shown as the blue line below: (Yes, 8 year-old Tawnse and I were sitting on the side of the 'cliff' on the other side of the Pecos River from the campground during that QSO. Yes, Tawnse does have a lower center of gravity than I do and led me up that path. Yes, I did have a heck of a time getting back down once I turned ...

 It's been a busy and fun week again! Most of my time was spenting finalizing what the elevation profile Datasette enrichment would look like. I finally found the last piece of the puzzle laying around in a pile of Jinja template use practice. One of the goals this week will be to release the elevation package. I made my first pull request for an open source project, datasette, in this case. (I make them at workr, but this is my first foray into the wider world.)  ChatGPT provided the final kick to get the elevation maps plotted really eaisly. Subsequently, I learned about the Pyplot package and how to add labels to charts. I got a start back towards writing about dadding. I spruced up some writing I had lying around about cleaning the kitchen with the gang. I found another older piece that I love while checking out old blog entries here: Meerktas and Ravens . Oh, and then this one about what unschooling with toddlers lools like. Goals Release the elevation package Relea...

 I got to play around with spherical trig last week under the guise of improving the accuracy of the model that KO6BTY's and my F2 skip maps utilize.  Today, I took the first steps towards formalizing the algorithm I modeled into a datasette plugin so we can use our kml plugin to create skip maps (I'll probably also add a configuration flag to turn skip mapping on and off.) Using datasette-haversine as a model, I created datasette-haversine-point-path . The new plugin allows users to specify three lat/lng pairs that define the enpoints of a QSO and the location of an ionosonde. Given this information, the newly defined sqlite method will return the shortest distance between the ionosonde and the QSO's propagation path along the Earth. (I still need to add the special case where the shortest distance to the path is in fact between the ionosonde and one of the endpoints. I'll talk about this in my next post.) I re-puprosed the test case from datasette-haversine as wel...

 I released my first Python Package last week! I've been using Datasette to analyze my QSO logs for several months . It's been a really useful tool since it's already setup with quite a bit of mapping fucntionality via various plugins . At the same time, the gang and I have been working with the Google Maps API and kml files to map the paths between amateur radio staions in our QSOs.  A few weeks ago, Simon Willison announced Datasette Enrichments . One of the first exampels was a geocoder (an enrichment that produces the longitude and latitude of an address) using opencage . This opened my eyes to the possibilities of taking the gang's and my original mapping utilities I mentioned above, and implementing them with a number of datasette enrichments and plugins. What was missing however, was a way to do this with Google Maps instead of opencage. Missing until last week that is! The gang and I worked through creating the google maps geocoding enrichment . Then, I worked...

 One of my goals this week is to add elevation profiles between QSO stations as a Datasette enrichment to our ham radio QSO Datasette utility. It's turning into a bit more of a convoluted project that I'd expected, so I'm taking the time to right down the first pass of the steps in the process. Here are the broad strokes: Complete a partial path Datasette plugin to find the coordinates of a small portion of the path between stations to use as the endpoint for an elevation profile path. This is already done . Use those coordinates as an input to a Datasette plugin that returns the elevation profile as json data. The first step of this has been prototyped . (It's looking like installing jq for Python on Windows is going to be a bit of an issue , so I'll find a different way to parse the returned json. I've done similar tasks with the ionosonde data and the rx station geocoding data. Produce charts as images for each row so that the query immediately presents a sum...