Scattered Thoughts on LLM Technology

In a sense, this is maybe part of the blog post I promised back in January 2023 but never wrote, because I couldn't ultimately wrap my head around LLMs enough to form coherent thoughts.

I am less skeptical of this than I used to be, and I find a number of arguments about it less compelling. For a longer, more compelling, less scattershot version of this post (not all of it, but some of it), please read The average AI criticism has gotten lazy, and that's dangerous by Danilo Campos.

  1. On the energy consumption case, I have to remind myself sometimes that I am not inherently against energy consumption. It must be done thoughtfully, and renewably, but energy use by itself is not disqualifying. Furthermore, there is going to be huge amounts of financial and competitive pressures for LLM technology to get leaner, using less compute.

    This is a contrast to blockchain, where in proof-of-work systems, resource consumption is the point. Efficiencies are automatically consumed by difficulty scaling, and so individual actors can find efficiencies and profit from them, the system as a whole cannot.

  2. On the training data problems: This is a legitimate concern; yet it strikes me that it is not impossible to overcome; Similar to how LLMs will have strong downward pressure on their compute and energy consumption, LLMs are experiencing downward pressure on their data requirements. While huge data and huge compute are required today, I think it's short sighted to assume the same will be true tomorrow. The idea of a small model that is specialized to mozilla-central is kind of appealing!

  3. It feels like there has to be a bit of a cost reckoning; If OpenAI isn't profitable (and I don't know that we know if that's true or not), we could see costs climb for the deployment of AI models. Spend some time playing with the OpenAI cost calculator, and it's pretty clear that at scale it's already reasonably expensive to use. I think we will see three trends here: 1) Companies that are just thin wrappers of OpenAI’s APIs will slowly disappear; why go third party when OpenAI will serve you (for cheaper almost certainly). 2) You’ll see at least two hardware startups blow up on their ability to dramatically reduce cost for running these open models. E.g. play with Groq demo. Partially driven by cost, I also think

  4. The “just throw a chatbot on X” model of deployment we’re seeing now will fade away; a lot more usage will be LLM-as-API; I expect this also means API focused models. Ones where output is always JSON, and they are task trained more than general purpose. I’ll bet we even see input stop being “prompts”, as the technology adapts to traditional programming over time.

  5. This space is going to change remarkably over the next few years; we're in for a period of dramatic change here. I've no idea how it's going to go, but it's going to look wildly different in five years I suspect. I work at Mozilla, and the Mozilla Foundation has been thinking about this a lot.

  6. I worry so much about bandwagoning, and bad choices driven by FOMO; I think so many of the "rub-generative-ai-on-it" projects are gross, and many of them undermine the value proposition of the organizations deploying them. Yet, I am increasingly convinced that I can't put a blanket over my head and hope this all blows over... it's not blowing over.

For more, be sure to also read Campos' What if Bill Gates is right about AI.

Books of Early 2024

Let's talk a bit about some books I read early this year (some are also waiting for a longer blog post too).

I'm loathe to publicly call out bad books, but I will say, one book not represented here is one I paid good money for, and hated. Everything about it says I should have loved it; a well-loved author, genre and subject a lovely mishmash that should have worked for me, and yet less than a third of the way into the book I had to stop. I haven't hated a book like this in a long time. All the reasons why I should have loved it made the pain of it being bad all the more painful.

Three Men in A Boat

I am gently shocked I didn't write a blog post about this book previously; this was actually my second read through. Published in 1889, it's a humour book which holds up ridiculously well for something which is 135 years old. I first discovered it after it was mentioned/tied gently into Connie Willis' To Say Nothing of the Dog, so it's likely my first read through of Three Men in a Boat was roughly 2018.

In theory it is the story of three young men taking a break from their lives to take a vacation by boat up the river Thames, and it is that; however, a good chunk of the book however is humorous digressions. One of which is very prescient reads on our contemporary character:

To go back to the carved-oak question, they must have had very fair notions of the artistic and the beautiful, our great-great-grandfathers.  Why, all our art treasures of to-day are only the dug-up commonplaces of three or four hundred years ago.  I wonder if there is real intrinsic beauty in the old soup-plates, beer-mugs, and candle-snuffers that we prize so now, or if it is only the halo of age glowing around them that gives them their charms in our eyes.  The “old blue” that we hang about our walls as ornaments were the common every-day household utensils of a few centuries ago; and the pink shepherds and the yellow shepherdesses that we hand round now for all our friends to gush over, and pretend they understand, were the unvalued mantel-ornaments that the mother of the eighteenth century would have given the baby to suck when he cried.

Will it be the same in the future?  Will the prized treasures of to-day always be the cheap trifles of the day before?  Will rows of our willow-pattern dinner-plates be ranged above the chimneypieces of the great in the years 2000 and odd?  Will the white cups with the gold rim and the beautiful gold flower inside (species unknown), that our Sarah Janes now break in sheer light-heartedness of spirit, be carefully mended, and stood upon a bracket, and dusted only by the lady of the house?

Others instead poke fun at the manner and mores of the day; visit the Project Gutenberg ebook, and read from "Have you ever been in a house where there are a couple courting?".

Or perhaps if you're in for the fantastic story of a ripe cheese, read from "For other breakfast things, George suggested eggs and bacon".

Persuasion

When I got sick recently, I discovered all I wanted was Austen. So we watched Sense and Sensibility, Pride and Prejudice , Mansfield Park (Grim) and I read Persuasion. First time reading it, though I was familiar with the story beats, having watched an adaption last year.

Persuasion is an interesting book; tension wise, it is a slow burn, building to its crescendo only in the final chapters. It's a good love story, told well, in a world that feels lived in.

I was struck however by the characters. Last year I read Middlemarch (which I apparently never wrote about?? Something I find gently shocking considering how highly I now regard it!), and what I find having read the two near each other is how much more alive the characters in Middlemarch feel -- their emotions more clear, their strains more communicated.

I am very happy I read Persuasion, but I think I preferred Pride and Prejudice when I read it a few years ago.

The Very Secret Society of Irregular Witches

This was just a very cozy story of found family in a gentle magical world just behind the veil of our own. It was warm and fuzzy (with a small amount of sex, including one ??? moment for me), and it was just the right thing for a cold winters day.

A Few Cozy Mysteries

I lucked out and got to read two lovely cozy mysteries over this Christmas break:

Vera Wong’s Unsolicited Advice For Murderers

A man dies in a lonely teashop, and the proprietor decides that she is going to solve this mystery. I loved this book, and it was the perfect read for my holiday. This joins the Thursday Murder Club for great mysteries involving older people solving mysteries.

You may have heard of the idea of love languages: Vera Wong’s love language is food and I found myself starving after almost every chapter.

Aunt Dimity’s Death

Not a murder mystery, but more fantastical mystery, I nevertheless was mostly charmed by this book. However, a misapprehension gained early in the book led me down a wild goose chase that never paid off; the book is for the better not paying my hunch off, but my wife went down the same road, which suggests this is either intentional or at least a flaw others have tripped on.

Still. This was a cute read, and could probably be comfortably consumed a weekend or less, if you don’t have children clambering across you.

Ten Years of Blogging

It turns out, that today is the ten year anniversary of this blog. I opened this blog with a description of my history with blogging, and started with a decision to make the blog open without topics. That lasted until 2016, when I split my technical blog off.

I’m super happy to have made it to ten years. To ten more years!

Some Favourite Posts:

I also wrote some work related blog posts that I really was happy with:

Year after year, my most popular blog post remains Some Notes on CMake Variables and Scopes. I totally get this -- CMake scopes and variables are bizarre.

This year, my number two most popular blog post has been Polybius: The Rise of The Roman Empire, which I find gently baffling.

I've managed to hit the front page of Hacker News, with Faster Ruby: thoughts from the outside.