Skip to main content

TouCans QSO DB: Lab Book

 Project TouCans just got a major database upgrade! The QSO records that KO6BTY and I log from field activations can now be appended directly from our GitHub CSV files instead of regenerating the entire SQLite database. This new method—based on an approach I first saw in Simon Willison’s video—keeps local data editable while automatically merging in the latest contacts from the cloud.

The project code is here.

The excellent video by Simon Willison that inspired this methodology  is here.

Finally, here's more background on Project TouCans.

The Setup

I've written a qso database that KO6BTY and I use to track our QSOs. Over time, we've built various ionospheric analysis apps with Datasette that work with and add to the data. 

The Problem

I had to create the rm_toucans.db sqlite file from a csv file stored on github. Each time, the old rm_toucans.db was thrown out and an entirely new database file was generated. With that methdology the records in rm_toucans.db were not editable.

At first this was ok, but more and more of our analysis tools are editing rm_toucans.db records. The simplest example involves calculating the launch angle of a transmitted signal. It's computationally intense and the result, once calculated, never changes. Rather than calculate it more than once, we want to be able to just store the result in the qso database along with the rest of the information for that QSO. I lose those record edits with the original methodology. 

The Solution

I created a Python script that accepts a timestamp as an argument. It then reads the csv file inserting records that are after that date to the sqlite database. All the modified records remain in the database. That script can be found here. Here's the documentation.

What's Next

Simply open the sqlqite database and run a query to find the latest date without user input.

This also allows other apps like:
When the height of a QSO transmit location is adjusted, store it in the slice database along with a field height_adjust = 1. Then, use a second script to copy that modified record back over to the original rm_toucans.db. Do the same thing for launch angle and hmF2 per QSO. In that way, I'll get rid of the multiple step process for updating those fields at the moment. (Note: there currently is no process at all for updating the tx location altitude in the database.)

Diagram showing GitHub CSV QSO data being appended into a local SQLite database for Project TouCans, illustrating the new append-based update method.




Labels:

Comments

Post a Comment

Please leave your comments on this topic:

Popular posts from this blog

Cool Math Tricks: Deriving the Divergence, (Del or Nabla) into New (Cylindrical) Coordinate Systems

Now available as a Kindle ebook for 99 cents ! Get a spiffy ebook, and fund more physics The following is a pretty lengthy procedure, but converting the divergence, (nabla, del) operator between coordinate systems comes up pretty often. While there are tables for converting between common coordinate systems , there seem to be fewer explanations of the procedure for deriving the conversion, so here goes! What do we actually want? To convert the Cartesian nabla to the nabla for another coordinate system, say… cylindrical coordinates. What we’ll need: 1. The Cartesian Nabla: 2. A set of equations relating the Cartesian coordinates to cylindrical coordinates: 3. A set of equations relating the Cartesian basis vectors to the basis vectors of the new coordinate system: How to do it: Use the chain rule for differentiation to convert the derivatives with respect to the Cartesian variables to derivatives with respect to the cylindrical variables. The chain ...
10 comments
Read more

The Alcubierre Warp Drive Tophat Function and Open Science with Sage

I transferred yesterday's Mathematica file with the Alcubierre warp drive[2] line element and space curvature calculations to the  +Sage Mathematical Software System  today, (the files been  added to the public repository [3]).  If you haven't used Sage before, it's a Python based software package that's similar in functionality to Mathematica.  Oh, and it' free.  I also worked a little more on understanding the theory, but frankly, I made far more progress with the software than the theory.  What follows will be a little more of the Alcubierre theory, plus, a cool Sage interactive demo of one of the Alcubierre functions[1], as well as a bit about my first experience with using Sage. Theory The theory is fun, but it's moving slowly.  Here's the chalk board from this morning's discussion Alcubierre setup the derivation using something called the 3+1 formalism which means we consider space to be flat, (in this case), slices that are labelled ...
3 comments
Read more

The Valentine's Day Magnetic Monopole

There's an assymetry to the form of the two Maxwell's equations shown in picture 1.  While the divergence of the electric field is proportional to the electric charge density at a given point, the divergence of the magnetic field is equal to zero.  This is typically explained in the following way.  While we know that electrons, the fundamental electric charge carriers exist, evidence seems to indicate that magnetic monopoles, the particles that would carry magnetic 'charge', either don't exist, or, the energies required to create them are so high that they are exceedingly rare.  That doesn't stop us from looking for them though! Keeping with the theme of Fairbank[1] and his academic progeny over the semester break, today's post is about the discovery of a magnetic monopole candidate event by one of the Fairbank's graduate students, Blas Cabrera[2].  Cabrera was utilizing a loop type of magnetic monopole detector.  Its operation is in...
1 comment
Read more