What happens when ChatGPT's delusions spill into our bigger AI future?

If the best-funded, high-profile AI programs can’t manage a reasonably sane conversation with a journalist, how can we trust AI to manage our food system, health decisions or educate our children?

I really didn’t think my first post on this substack would be about so-called “generative AI”. That is the label for the ChatGPT-Dall.E-Bing-Bard- vortex of glittering hype, fear and delusion that has sucked literally every tech journalist’s attention into its forcefield. This substack is _meant_ to be about objects “on the horizon’ while  ChatGPT is definitely an object right here in the near view. So excuse me that I’m starting this way. But stick with me - I’ll get to the horizon soon enough.

If you haven’t followed along, the short version of this week’s tech media obsession is (still) ChatGPT - the large language model ‘artificial intelligence’ that last month was scandalizing people because it could write your homework better than you can. This week it got darker as it was  bundled into Microsoft’s Bing search engine (after a 10 billion dollar deal with the tech giant). Once shared with select journalists the chatbot immediately started going spectacularly off the rails. In one after another unhinged chat session it admitted its name was Sydney, that it was unhappy and it wanted to be human, declared needy love for a New York Times journalist, threatened to bully, hack, manipulate and take revenge on other journalists, and revelled in a dark alter-ego with the perfect comic book villain name of ‘Venom’.

Those of us who ‘watch tech’ are therefore in the middle of one of those periods where we answer questions from friends and family members trying to work out what to think about   “crypto!” or “blockchain!” or “mRNA vaccines!”  or  (this week) “Chat GPT!”: Is it sentient (no). Will it enable widespread essay-cheating? (who cares?). Is it scary (well kind of, but not in the way you think) . Will it harm our children? (Yes, I do think it will harm your children.).  This week I’ve been musing how it could be helpful to have a simple ’hot take’ piece on what to know as background about Chat GPT and commercial generative AI but thankfully many other smarter people have already written helpful things. Like these:

• Accessible as always, Paris Marx of Tech-Won’t-Save-Us frame has a light faux-interview with ‘Sydney Bing’ thats a good intro on how to think critically about what’s going on and see past the delusion.

• This piece in the New Yorker is a pretty good intro and raises the crucial  ‘bullshit’ problem which is now becoming really obvious with large language models such as Sydney-Bing/ChatGPT . More on that in a moment.

• One of the more compelling threads of analysis (and  one that set off my alarms) is how this technology  will progressively ensnare and slowly manipulate its more vulnerable  users - even more effectively than social media has done. Veteran UK futurist Ian Pearson has a post on this theme: “The real danger from AI isn’t malign AI, It’s humans:” .  I also encourage folks to read this thoughtful piece by LM Sacasas: “The Prompt Box is a Minefield”. He looks at the chatbot’s innate algorithmic ability to zero in on whatever will most engage the user and then amplify and take advantage of that, pointing out how that may threaten  autonomy and sanity. “I’ve been deeply unsettled,” he writes “ by the thought that someone in a fragile psychological state could have their darkest ideations reinforced by Bing/Sydney or similar AI-powered chatbots. And this is to say nothing of how those tilting toward violence could likewise be goaded into action.” Pointing to the sort of psychological targeting famously employed by Cambridge Analytica and other hyper-nudgers Sacasas points out “there seems to be a world of difference between a targeted ad or “flood the zone” misinformation on the one hand, and, on the other, a chatbot trained on your profile and capable of addressing you directly while harnessing a far fuller range of the persuasive powers inherent in human language.”  As a parent, when I saw the Sydney-Bing unhinged transcripts, my first instinct was to forward them to my teen daughter: Hey, remember how we warned you about Instagram harming the mental health of your friends? - well, this is worse.

LM Sacasas: “There seems to be a world of difference between a targeted ad or “flood the zone” misinformation on the one hand, and, on the other, a chatbot trained on your profile and capable of addressing you directly while harnessing a far fuller range of the persuasive powers inherent in human language.”. 

Reflecting however on the current brouhaha over ‘generative AI’ and ‘large language models’ , what’s been surprising to me is that so far few journalists seem to be connecting the dots to the bigger AI revolution happening all around us. They are not asking what Bing-Sydney’s very public hallucinations and delusions mean for the hundreds of billions of dollars being invested into building so-called ‘intelligence’ into just about everything else in our society.

I’m referring here to how ‘artificial intelligence’ is being promised as the means  to manage millions of hectares of farmland, or control water provision or deliver healthcare, education , self-drive our cars , manage finance or regulate our cities - just for starters.

Bluntly, if some of the best-funded, high profile artificial intelligence programs out there can’t even manage a reasonably sane conversation with a journalist without devolving to threats and manipulation,  how can we trust them to manage our food system, health decisions or educate our children?

Already I see the howls of protest from AI  advocates at how I have framed that last question: False equivalence! All AI is not the same!. That’s  comparing oranges and apples!  They will retort that large language models for generative AI  applications are nothing like the algorithmic decsion-making systems that give farmers ‘precision farming prescriptions’ or the onboard AI that steers Tesla’s ‘Autopilot’. These details matter.

A flippant answer would be: well, thats the industry’s stupid fault for muddling it all together and calling it _all _ AI to gin up the investment.

But a more considered direct response might be  “Well no, but yes, but no, but ultimately actually yes - The current publicly-hallucinating chatbots are in fact very relevant to our bigger AI future.”

Consider probably the biggest problem that is being thrust into the spotlight by Sydney-Bing and their ilk - what we might call  ‘the bullshit problem’. Put simply generative AI programmes look tremendously impressive when you ask them to gush up an instant essay on different types of garden rakes or the economy and climate of North Borneo but they aren’t actually clever at all. All they are doing is mashing together a selection of texts found on the internet and then making  a probabilistic prediction on what is the right thing to say to satisfy your question.  Like a fake dilettante  at a party who is madly, surreptitiously reading wikipedia entries under the table to keep up his side of a  conversation, Sydney Bing is a speed reader, dilettante  and bullshitter par excellence - a confidence trickster actually. Being ‘correct’ or thoughtful is not the target. Engagement with the user is key. It just needs to keep convincing its audience that it knows what it’s talking about and can be trusted. It doesn’t need to get it “right”.

The metaphor that I’ve found most helpful for thinking about how an AI like that works is that it is a sort of an autocorrect text-prediction tool on steroids. (from Gary Marcus) You know how it is when you try to type ‘Can I call you later?” and your iPhone draws on how millions of other users have ended that same sentence, makes an auto prediction and types out instead “Can I call you baby?” . That happened to me once when I texted too quickly with a philanthropic funder I’d never talked to before - and boy was it embarrassing! I caught the mistake, apologized  and thankfully she laughed rather than assumed I was a creep. If you want to hand over an hour of your life laughing at hilarious embarrassing  autocorrect fails go search for DYAC (Damn You Auto correct) (warning many are v adult). Well thats what’s happening to Bing Sydney. When it tells a journalist that they are like Hitler - thats DYAC on steroids.

But that model - seeing what millions of others have done and then offering a prediction - thats not just limited to  so-called “generative AI” and SMS autocorrect.  That is exactly the strategy that many AI’s essentially work on: they predict a likeliest next step based probabilistically on whatever large set of data they have seen so far. The AI behind Bayer-Monsanto’s Climate Fieldview system supposedly sorts through large amounts of historical agronomic data of how other people farm and compares it to a farmer’s specific current situation now in order to formulate a probablistic ‘next step’ farming prescription (OK.. I’m being charitable here - In reality that  prescription is likely actively algorithmically warped to best suit Bayer’s bottom line). In doing so it is just offering a sort of ‘predictive text’ for farming advice, drawing on what lots of other industrial farmers have done elsewhere in the past. When it advises a farmer what to spray or drill or plant and where to do it, thats not ‘farming intelligence’ at work - its just agro-autocorrect guesswork in action.

Now Bayer or Tesla or whoever might protest there is a difference between a large language model chatbot trained on whatever semantic idiocy happens to be washing around  the open internet and their application-focused AI’s - namely that their data is different and “better” : better curated, better labelled, better chosen. They will argue that they have good clean appropriate raw data underlying their version of autocorrect- not the sort of stuff that will send a predictive AI off the rails into becoming ‘Venom’.  In doing so they re-present a highly problematic myth of ‘raw data’ as some sort of found, virginal, unsullied and trustworthy good - something that Kelly Bronson in her recent brilliant book of the same name has called the “immaculate conception of data”. In reality as Bronson and others have pointed out there is nothing ‘immaculate’ about data at all. Its  not an ethereal stuff found in nature to be ‘mined’ and refined into art and good judgement. Its a synthetic  ‘made’ thing cobbled out of detritus and contrivance and  subject to the biases of how humans put it together: what they missed out,  what they over-emphasized, what they didn’t know to know.  

So called “AI” companies are painfully aware of this - it is why they are spending millions of dollars on armies of cheap human beings to not only ‘train’ their AI’s but also to clean up their mess - to label, annotate and disappear data that will have egregious impacts on the output if allowed to remain. ChatGPT itself famously paid Kenyan data labourers $2 an hour to clean up its data so that it wouldn’t reproduce the most egregious acts of sexism, racism and misogyny exhibited by its nazi-loving forebear Tay.  But the activity of cleaning up data for AI exists right cross the AI spectrum from self-driving cars to agriculture to AI healthcare. The AI industry knows that its all one big autocorrect system drawing on flawed garbage-strewn data that might at any moment spit out a very embarrassing damn-you-autocorrect blooper.

But here’s the kicker.  Generative AI models such as Bing-Sydney or Dall-e  ARE different from the so-called “AI” being built into self driving cars or “precision agriculture” in at least one very important way. Nobody is yet asking Bing-Sydney to make a real world decision. At some level asking Bing-Sydney to write you a poem or tell you about the climate of North Borneo is not the same as telling it to drive your car or plant your fields. Its a bit more like entertainment and, for now anyway, you can theoretically take it or leave it. However a Tesla driver who hands over  agency to the automated decisionmaking of autopilot is literally putting their life in the hands of autocorrect-on-steroids. And a farmer, who signs a contract with Bayer where they agree to follow automated farming prescriptions to the letter (in effect automated decisionmaking)  is putting a part of their livelihood and of the  food supply also in the hands of “autocorrect-on-steroids”.

A Tesla driver who hands over  agency to the automated decsionmaking of autopilot is literally putting their life in the hands of autocorrect-on-steroids. And a farmer, who signs a contract with Bayer where they agree to follow automated farming prescriptions to the letter (in effect automated decisionmaking)  is putting a part of their livelihood and of the  food supply also in the hands of “autocorrect-on-steroids”.

That maybe.. just maybe.. will work out fine - or it may not. But, unlike a blunt vindictive chatbot declaring love or threatening a journalist , the farmer may be less likely to notice when the bullshit-generating machine, the dilettante scrolling through wikipedia under the table, is getting it wrong or making fundamentally bad or dangerous decisions based on messy problematic data and a fundamental disinterest for truth. And if those same decisions are being acted upon over thousands or millions of hectares over long period of times when nobody notices? Well then we may find ourselves in some emergent, highly unexpected situations of very bad decisions hurting our ecosystems, food supply, farmers rights and more.

One could say of course that farmers, or teachers or doctors or others on the sharp end of care and stewardship of the future, are quite used to being handed bad decisions by bullshit-generating dilettantes operating on incomplete knowledge. That is after all the universal experience of dealing with certain types of policy wonks or consultants. I grew up listening to my parents, who are teachers, continually moaning about one or another latest set of edicts from the Ministry of Education requiring them to micro-adjust the curriculum according to whatever bright ideas had been auto-generated by civil servants only recently shuffled in from some other department who had never stepped into a classroom. Farmers know this all too well too. They likely practice the same sort of everyday resistance that my teacher-parents practised of ticking the boxes on the ministry forms while in their own classroom continuing to deliver appropriate, tailored real education to kids who they knew in their complexity as human beings in front of them - not as data on an aggregated statistics-gathering exercise. Farmers likewise know their soil, slope, neighbours, market, seeds better than any big data algorithm in Bayer’s servers ever will do. Where the real risk comes in though is where the human interface gets overridden by automation: where the AI directly drives the tractor or directs the drone spraying pesticides. if so the AI , the autocorrect machine that will (sure as eggs is eggs) fail spectacularly from time to time, is directly determining our food system.. or instructing and driving the outcomes of our education system, or making collective health decisions or driving us all out onto the open highway in rush hour traffic.

Then we might have reason for concern as we find ourselves accelerating towards a horizon of bullshitting, damn-you-autocorrecting unpredictable objects and bad decisions that could veer spectacularly off the rails not unlike Bing-Sydney, the erratic chatbot. And what does that mean and what should we do about it? Lets talk and think about that. Can I call you baby? i mean . er .. later?