The requirements were pretty clear. An AI Chat UI to query the content of Jack Graham’s Essays using an LLM. Event though I’d never written any code integrating AI functionality and though my Python skills were shallow at best, still, this seemed easy enough… Right?

First off, I thought that since I would be writing code are AI, maybe I should use AI to help me write the code. I immediately downloaded Cursor. I won’t review Cursor here, but let me just say, “Wow!” It’s literally VS Code with an AI assistant that can generate reams of pretty darned good code. Of course, that code needs to be vetted, sometimes corrected (with the aid of the AI assistant?)… If you haven’t already, I suggest you give it a whirl for a free two-week trial. There is an always-free model in addition to subscription models.

Next, I set up Supabase locally as suggested in the requirements.

With Cursor and Supabase installed, I asked it to generate a Python script for data ingestion. WOW! I had it “working” in about 30 minutes. Now, granted, I rewrote that script several times as my understanding of tools and technical issue grew, but the last version still bears a striking resemblance to the AI-generated original.

Cursor introduced me to the Python “requests” module and Beautiful Soup for web scraping. As I dug into the world of Tokens, Vectors, and Embeddings for documents, learning what each of these concepts represents, I discovered — and learned about — the Supabase (postgreSQL) pgvegtor extension, Supabase migrations, LangChain’s WebLoader, RecursiveCharacterTextSplitter, OpenAIEmbeddings and SupabaseVectorStore components. FWIW, LangChain was a god-send.

As I moved on to the backend app, Cursor did a yeoman’s job of scaffolding Node.js and Express.js to implement a REST API. That code required some tweaking along the way to make use of the JavaScript versions of LangChain’s components and the Supabase client, but it basically did the basic job right from the start.

Then I had Cursor scaffold the Chat API using Next.js and Shadcn UI components. This was a reasonably good UI. I did have to hand-code the integration between the LLM search and the match_documents function that I’d set up early on. This also required some refactoring of match_documents and the Python data ingestion script to ensure that the essay titles were always captured when the essay page did not contain a <title/> element.

Lastly, there was some clean-up and documentation to do. Cursor helped with both.

The final project, PaulGrahamEssays, is publicly available on GitHub. If you feel the need to comment, remember, this was a learning project, not the product of years of experience…

In all, I would say this project was about 60% learning new concepts, tools, and techniques and about 40% writing code.

In the end, even if I don’t get a job offer from this company, I have demystified the basics of developing apps with AI integration for myself. This widens the scope of jobs to which I will be applying. That’s all good!

If you have read my post, Dev Retro 2022: What I learned in my 43rd year..., you know I've been in the software development game for a long time. However, it's only recently -- the last three years or so -- that I have used Javascript and Typescript professionally -- and exclusively -- in my role as a full-stack MERN engineer. (Note: Hereinafter, when I mention Javascript or JS, know that I'm including Typescript (TS) unless I clearly state otherwise.)

There are days when I love JS/TS. The language's rich prototypes for objects and arrays; the wealth of utilitarian methods available on those prototypes; the economy of expression that is possible in one's code; the freedom to use Type-based code (ES6+/Typescript) when it serves a REAL purpose; the freedom to eschew type-based code when it becomes a limiting factor. All of these aspects of the language are useful and, perhaps, are among the reasons that Javascript has become the language of choice for many Web apps both on the client and server side.

...but "da devil be lurkin' in dem details..."

This particular devil bit me this week. I managed to pollute the Array constructor in one mistyped line of code that brought testing to a halt for several hours while a handful of extremely experienced engineers -- including me -- puzzled over an error being thrown deep inside the code of a third-party package: Array.isArray is not a function.

It took several hours to identify the culprit -- an errant line of code in the app itself. Here's what the code looked like, essentially.

function wasWorkingUntilRecentChanges(...args) {

    // do some stuff for 10-15 lines of high-density code

    const thisArray = Array<Record<string,any>> = 
        thatArray.filter((i) => i.property);

    // do some more stuff for 10-15 lines of high-density code

    // Suddenly, this line of code has begun throwing an Error
    // with the message `Array.isArray is not a function`
    const myResult = em.find(MyType, {});

Do you see it? I suppose the problem might be incredibly obvious here since I have elided the surrounding code that obfuscated the mistake.

Example 1 - The simple line of code that should have been

const thisArray: Array<Record<string,any>> = 
        thatArray.filter((i) => i.property);

This was what was intended -- declaring an array with a specific type to catch the result of thatArray.filter().

Example 2 - The similar-looking, though monstrously fateful line of code that was

const thisArray = Array<Record<string,any>> = 
        thatArray.filter((i) => i.property);

This was the actual code that was committed -- a multiple assignment statement that polluted (over-wrote) the Array constructor object.

Note that the damage caused here didn't result in any errors or warnings from Javascript, Typescript or ESLint in VS Code. They all silently approved.

There was no complaint from the Typescript compiler when the code was built.

Only when the code (a CLI) was run in Node.js did the monstrous result become manifest: Array.isArray is not a function.

Wha...?!?!?!

Array is the constructor function for every array in Javascript (see Array on MDN). It is provided as part of the Javascript engine in the execution environment -- Node.js, the Browser, etc. When you write const myArray = []; The Array constructor is used to create that object.

In addition, both Array and Array.prototype provide a number of utilitarian methods for handling arrays including Array.isArray(); the ubiquitous methods Array.prototype.map() and Array.prototype.filter(); and a host of others.

However, Array is not an immutable (read-only) object because there is no such thing in Javascript. In fact, the JS Array function is just like any object in Javascript. It is writable. Herein lies the problem.

So, the perfectly legal, mutiple-assignment statement in Example 2 runs afoul of these JS "features". Here's what happens.

1. thatArray.filter(...) is called.

2. The result of the function call is stored in Array<Record<string, any>> which actually means Array.

3. The value stored in Array (i.e., now the value returned by the function) is stored in thisArray.

The net result here is that the Array function has been obliterated (over-written) by the results of thatArray.filter(...). In this case, that was just a POJO from an imported .json file.

The consequence is that, from this point forward, any JS code in this routine -- including code in packages -- that attempts to use Array in its official capacity will fail.

• const myArray = [] will fail because it attempts to call the Array function behind the scenes.

• Array.isArray(myObj) will throw Array.isArray is not a function because... well... it is no longer a function.

• myArray.map((a) => ({ ...a, ...b}) will continue to work because these functions are on Array.prototype which is a different object than Array and any existing Array already has a reference to Array.prototype in it's own .prototype property.

The truly heinous thing here is that the problem may not show up immediately. Until the program reaches a line of code that must create an array or check whether an object is an array, the devastation will be buried.

...and when the program runs headlong into this pit of devastation -- which it almost certainly will -- the error messages, though helpful, will be confusing. "How can Array.isArray not be a function? It's always a function..."

Well, until it's not. 😉

If this happens to you, go looking for something like Example 2 above in your own code -- not in the package that threw the error.

Your first instinct might be to search for this error message related to the package that threw it. I did... for several hours... which was both fruitless and frustrating.

Remember. The package that threw the error is an innocent victim. Your own code is the villain.

In November of last year, my older daughter proposed that she, her brothers and I participate in Advent of Code (AoC). While I didn't know anything about AoC except what my daughter described, this sounded like it could be fun. Each of us is a professional software developer (engineer?) with a strong streak of curiosity and a competitive spirit.

If you aren't familiar with AoC, it is an annual software-development-focused, problem-solving competition that runs from Dec 1st to Dec 30th -- the season of Advent, hence the name.

During the competition, part one of that day's challenge is published on the AoC website at midnight EDT. Using any technology stack, development language, or computation tool, each competitor must solve the problem -- Part One and Part Two -- as quickly as possible. The description of the problem and competitor-specific input data are provided by AoC. The criteria for success is entering the answer -- always a single number -- into the AoC website page. When the correct solution to Part One has been entered, AoC provides Part Two of the problem which generally increases the complexity of the Part One problem.

Past solutions have been crafted in every imaginable development language with the possible exception of IBM System 370 Assembler. In 2022, the leaderboards for AoC were dominated by participants who used generative AI tools (e.g., ChatGPT). Every day, solutions were posted in under a minute, sometimes within seconds. The outcry from "legitimate" competitors was deafening. For 2023, the use of AI tools has been strongly discouraged, at least for the official competition. Yeah. RIght.

...but I digress. Back to the story

For my family, the 2022 challenges were fun for the first week or so. As the difficulty increased, the solutions became more time-consuming... much more time-consuming.

Since we all have real jobs that pay the bills and real lives, one by one each of us found it necessary to drop out. By the beginning of Week 3 (Day 15 or 16), we were done. I will only say that I was the last of our group to drop out... and that finishing AoC could have become a full-time pursuit by the end of the month! In the end, the only clear winner was AoC. It had beaten us! 😳

2023 - Thistory ("this story" 😉)

This year, my daughter and two sons opted out of AoC leaving me and my oldest son, Marshall. Not surprisingly, we had the same idea. Rather than using AoC as an individual competition, we could collaborate on the problems and use AoC as an opportunity to learn (a) a new language and (b) spend more time together. We decided to ignore the "competition" element of AoC -- ignoring the leaderboard completely -- and to move at our own pace since our objectives were decidedly not competitive. After agreeing on this approach and after some discussion, we decided to use Go as our development language this year.

As of today, AoC 2023 has posted 11 daily challenges. We finished the first 4 days' challenges rather quickly, learning some fundamentals of Go coding. Day 5 was a decidedly bigger challenge. We decided to dive deeper into Go and take our time solving this problem. We definitely weren't after the fastest solution, but certainly one that provided the correct answer.

Over a period of 8 days -- off and on -- we wrote some pretty wicked Go code -- delving into objects, separating code into packages, playing with generics, writing go-routines that use channels, etc. Generally, we were shaking down Go based on our experience with literally dozens of other languages (e.g., C, C++, C#, COBOL, Fortran, JavaScript, Rust, LUA, et al).

Our solution to Day 5 was big... and slow 🐌. Part 1 only ran for a few seconds. Part Two ran for 40 minutes. This was after implementing go routines to get parallel processing and consuming 10 AMD Epyc CPUs (on a 24-CPU Linux host). This was the epitome of the "Brute Force" approach 🤣. Thankfully, the answer it produced was correct 😮‍💨.

Go - Hysteri-a

Now, for my thoughts on Go. Already, it's a love-hate relationship -- on many levels.

While Go syntax is more or less like other members of the C/C++/C#/Java/Javascript family, there are some departures. In general, it feels like Go does not provide the economy of expression of some other languages. Take, for example, another, more performant solution for AoC 2023 Day 5f, I mean that equivalent functionality in Go and, for example, Typescript

The syntax for defining objects, methods, and interfaces took some time to feel "normal". In particular, the lack of (a) function overloading, (b) optional function arguments, and (c) default argument values means many additional, uniquely named functions to cover the various signatures required in a library. If you are the library's author, that can be a lot of extra code to maintain. If you are the library's consumer, it can mean a confusing list of annoyingly similar but different method or function names to navigate.

Error handling in Go is also idiosyncratic. The idiomatic practice of a function returning a second value to report an error -- or just ok -- makes it all but impossible to chain function and/or method calls together in the style of so-called fluent APIs. This is touted as one of Go's strengths, but I find it somewhat annoying. It certainly creates more verbose code.

The highly opinionated Go toolchain also takes a bit of settling in. You just have to get used to the fact that your IDE with Go tools set up is going to reformat your code every time you save your code... and your code won't run if there's any kind of an error in syntax, usage, type-matching or whatever. You simply MUST fix these things.

On the other hand, the efficiency and speed of the GO toolchain and your compiled programs -- WOW! It's just amazing. Likewise, the small size of your compiled programs.

Finally, Go has been around a while (see the Wikipedia article in the footnotes) and the amount of information available from the official website, StackOverflow and other independent sources is pretty substantial. A Google search for a go-related issue generally turned up a solution in a minute or two.

Conclusions - Mastery

In the first 10 days, it is clear that we are already achieving our objectives for AoC 2023. We have learned a lot about Go, spent more time together, and committed to continuing our efforts on the AoC 2023 challenges into 2024, albeit at a more leisurely pace. We'll still be using Go to develop more proficiency, but we'll move forward at a more leisurely pace.

If you are a professional developer these days, knowing something about Go is, at the very least, helpful. For some, it's required knowledge. The good news is that, in a week or so, you can get a pretty solid foundation in Go and be off to the races.

Links

• Day 5 by Brute Force - Our AoC 2023 Day 5 solution on GitHub

• Advent of Code Website

• Marshall on GitHub

  • My oldest son

  • Space software entrepreneur extraordinaire

• Go language

  • The official Go website

  • Go Wikipedia article

"[C. S. Lewis] showed me that newness is no virtue and oldness is no vice. Truth and beauty and goodness are not determined by when they exist. Nothing is inferior for being old, and nothing is valuable for being modern. This has freed me from the tyranny of novelty and opened for me the wisdom of the ages."

-- John Piper, How C. S. Lews Awakened John Piper to the "Realness" of Life, April 02, 2018.

**⁹**What exists now^[ah] is what will be,^[ai]
and what has been done is what will be done;
there is nothing truly new on earth.^[aj]
10  Is there anything about which someone can say, “Look at this! It is new”?^[ak]
It was already^[al] done long ago,^[am] before our time.^[an]
11  No one remembers the former events,^[ao]
nor will anyone remember^[ap] the events that are yet to happen;^[aq]
they will not be remembered by the future generations.^[ar]

-- Ecclesiastes 1:9-11, New English Translation

So, why am I quoting a renowned Christian author and the Old Testament here? Because, even in technology, in software development, there is nothing new under the sun. Let me explain...

In the last two years, I have been learning about MongoDB. Now, if you have any current knowledge of software development tools, you probably know something about MongoDB... unless you are a mainframe COBOL programmer with no intellectual curiosity whatsoever. ;) Of course, the most well-known fact about MongoDB is that it is a "NoSQL" database which is to say that it is decidedly not relational.

For many years before my introduction to MongoDB, I was a relational database junkie using on or the other RDBMS as the backing store for my applications. Over the years, I believe that I had become -- and still am -- a pretty darned decent relational database architect, designer and developer.

As I began to explore the world of MongoDB (my own personal "Oz"), many of the relational database tenets I have long held dear and perhaps even inviolate have caused twinges and cringes and gasps ("Oh, my!").

For example, joining a document (a row?) from one collection (a table?) with one or more documents in another collection is discouraged. Rather, one is encouraged to create sub-documents in one of the collections containing the document(s) from the other collection which thereby eliminates the need for a second collection.With joins greatly discouraged, it was no surprise to find that 3NF is not a thing, either.

"What is 3NF?", you say? You know... 3rd Normal Form... where "Every column is dependent on the key, the whole key and nothing but the key, so help me, Codd!" What? You never heard of that? Right. Anyway... 3NF is not a thing in the non-relational world. Well, actually, it is a thing but seemingly a bad or undesirable thing.

The more I learn about MongoDB, the more it starts to feel familiar... like... I've seen something similar to this before. This "knowing" continued to grow for a while and then, one day... click... whir... there it was! Indexed Files.

Do you remember Indexed Files? This was a "flat file" with indexes to directly access records in the file. There were variations of this theme on every operating system. IBM's mainframes had ISAM and VSAM files; DEC's VMS, RMS Indexed files; HP's MPE, KSAM files; MS-DOS, Btrieve (3rd party)... the list goes on and on. Generally, these indexed file systems required one unique index for the records while supporting one or more additional indexes that might or might not be unique. However, the format of individual records was not required to be consistent.

Document databases are very similar. Sure, there is general support for JSON input and output and the fields in each document are named, but the concept of a document schema is very, very loose. Validation is generally provided by external code or logic (in apps or ORM/ODM definitions). Their forté is lightning-fast access to data -- perhaps multi-level documents.

Relational databases -- which provide similarly flexible indexing options -- provide a much stronger definition of the data and relationships being stored. Used properly, this ensures that an application cannot store inconsistent data in the database.

In the end, it seems to me that a document database sits somewhere between the archaic indexed file systems -- which left everything except the indexing up to the application -- and relational databases -- which provide highly evolved facilities for data validation and integrity.

To my way of thinking, there is not that much new about document databases. They are a mashup of indexing facilities and "roll-your-own" content with the server-side processing provided by most relational databases.

Getting back to my original point...

Relational databases -- old and antiquated though they may be -- are not necessarily "bad". In fact, a decent RDBMS provides validation, constraints and integrity that are meager to non-existent document databases (i.e., MongoDB)... unless you use a third-party library (e.g., Mongoose or MikroORM).

For that matter, indexed file systems though older than relational databases, are not necessarily "worse". For applications written in certain environments in legacy languages, they may be the best choice.

Likewise, a document-oriented (NoSQL) DBMS, though the newest of all, are not necessarily "better" or "best" for every circumstance. Certainly, though, there are many use cases where NoSQL reigns, particularly in Web-based apps.

Lastly, I am not here to argue that a good RDBMS (e.g., PostgreSQL, SQL Server, Oracle, etc.) is the "right" data store for every purpose. However, there are many line-of-business applications where a relational -- or object-relational -- data store provides data integrity features that are beyond compare.

When it comes to data storage, old and new technologies alike have strengths and weaknesses. Choose your platform(s) carefully.

There was no database. I hand-crafted indexed file management routines. There was no GUI. Every prompt & input was carefully placed on the 24x80 monochrome screen. There was no virtual memory. All of my code had to fit into 64KB or less.

In the years since 1979, I wrote code in more languages, accessed data in more data storage technologies, used more methodologies and sold to and consulted with more customers than I could possibly count. I even co-founded, managed and sold a consulting company. It's been a grand journey.

From 1997 to 2020, I made my living with Microsoft Development Tools and Technologies. Not so much out of preference as out of practicality. That's where the opportunities were for me as an engineer.

Up until 2020, I'd never used a document database. I looked down on Javascript development. "Can't we use a real OOP language?" Express and Node.js were just tools in the Javascript ecosystem that I had no interest in learning.

That all began to change in late 2020 when I joined a startup selling a DevOps platform that was built on the MERN stack. I joined as a pre-sales DevOps Engineer, but there was a healthy dose of consulting with customers to build solutions around the tool. In early 2021, a cash-flow crisis resulted in the departure of several members of the development team. I got asked to fill in. I never looked back.

In the 18 months since that time -- in the 43rd year of my software development career -- I have learned a bit about Javascript development, MongoDB, React, Node.js, AWS APIs and... well, a bunch of stuff I never thought I'd learn.

The message here is that, even after 43 years of software development, I am still learning. If you hang around for 43 years, I bet you'll still be learning, too.