AI Sucks

Popular narratives focus, not on what big-crunch machine learning models can do now, but what they theoretically could do in the future.

Reading, understanding, and fixing code written by others consumes 90+% of the time a programmer spends in an integrated development environment, command line, or observability interface.

If we use LLMs all the time, the amount of “fix someone else’s code” we’re doing goes from 90% of the time to 100% of the time.

Large language model purveyors and enthusiasts purport to use the tools to help understand code. I’ve tested this claim pretty thoroughly at this point, and my conclusion on the matter is this: much like perusing answers on StackOverflow, this approach saves you time relative to whether you’re already skilled enough to know when to be suspicious, because a large proportion of the answers won’t help you.

But like an IDE, or a framework, or a test harness, utility here requires skill on the part of the operator—and not just ChatGPT jockeying skill: programming skill. Existing subject matter expertise.

That’s not entirely the fault of the model design. It has a lot to do with the fact that the training data, the Common Crawl, feeds it the code debugging and demystification advice of The Internet. How good is the advice about this topic on The Internet? Not that good, because by and large, developers suck at this skill.

Models trained on human data can’t outperform the base error rate in that data.

Haas estimates that while US power consumption by AI datacenters sits at a modest four percent, he expects the industry to trend towards 20 to 25 percent usage of the US power grid by 2030, per a report from the Wall Street Journal. He specifically lays blame at popular large language models (LLMs) such as ChatGPT, which Haas described as "insatiable in terms of their thirst."

OpenAI's GPT-4 large language model (LLM) can autonomously exploit vulnerabilities in real-world systems if given a CVE advisory describing the flaw.

Kang said he expects LLM agents, created by (in this instance) wiring a chatbot model to the ReAct automation framework implemented in LangChain, will make exploitation much easier for everyone. These agents can, we're told, follow links in CVE descriptions for more information.

Strikingly, the only text the manager actually inputs herself is the headline — "Best Bicycles for Kids" — and a series of links to Amazon products.Then the AI generates every single word of the article — "riding a bike is a right of passage that every child should experience," MEL advises, adding that biking teaches children important "life skills" like "how to balance and how to be responsible for their actions" — as the manager clicks buttons like "Generate Intro Paragraph" and "Generate Product Awards."

By the end of the video, the manager has produced an article identical in structure to the AdVon content we found at Sports Illustrated and other AdVon-affiliated publications: an intro, followed by a string of generically-described products with affiliate links to Amazon, a "buying guide" packed with SEO keywords, and finally an FAQ.

All five of the microwave reviews include an FAQ entry saying it's okay to put aluminum foil in your prospective new purchase.

do its human employees actually try the products being recommended?"No," laughed one AdVon source. "No. One hundred percent no.""I didn't touch a single one," another recalled.

People searching Google to buy a product, Weissman explains, are easily swayed by reviews in authoritative publications.

CNET got lambasted for publishing dozens of AI-generated articles about personal finance before discovering they were riddled with errors and plagiarism.

UnitedHealthcare, the largest health insurance company in the US, is allegedly using a deeply flawed AI algorithm to override doctors' judgments and wrongfully deny critical health coverage to elderly patients.

But, instead of changing course, over the last two years, NaviHealth employees have been told to hew closer and closer to the algorithm's predictions. In 2022, case managers were told to keep patients' stays in nursing homes to within 3 percent of the days projected by the algorithm, according to documents obtained by Stat. In 2023, the target was narrowed to 1 percent.

If a nursing home balked at discharging a patient with a feeding tube, case managers should point out that the tube needed to provide "26 percent of daily calorie requirements" to be considered as a skilled service under Medicare coverage rules.

It’s unclear how much UnitedHealth saves by using nH Predict, but Stat estimated it to be hundreds of millions of dollars annually. In 2022, UnitedHealth Group’s CEO made $20.9 million in total compensation. Four other top executives made between about $10 and $16 million each.

Kaiser Permanente, for its part, insists it’s simply leveraging “state-of-the-art tools and technologies that support our mission of providing high-quality, affordable health care to best meet our members’ and patients’ needs.”

Implementing this kind of transformative but error-prone tech in an industry where lives are on the line requires patience, intelligent planning, broad consultation with every level of employee, and competent regulatory guidance, none of which are American strong suits of late.

The club had “no idea” the software, set up with no manual intervention, was that unreliable, Rosin said.

Many of these software companies promoting AI say that one line of code will be enough to ensure compliance.

Blind users often say that overlays can make websites harder to navigate or that they interfere with assistive technologies.

As with any tech fad, AI is being applied carelessly, as if it carried no cost, as if a product got better by just featuring it.

lowering the bar of what consumers are willing to accept —passing the Turing test by making people dumber.

Things that felt like boilerplate or required subtle knowledge of tools that I don’t use frequently1. Regular expressions, crontab schedules, nginx rules, GitHub Actions workflows, Makefile targets, sed and awk incantations, CSS tricks.

Here is how it works. You ask you favorite LLM: "How to download a file in Node.js?" The LLM answers you the following code: import { Client } from 'http-request'; const client = new Client(); const res = client.get('https://example.com'); await fs.WriteFile('file.txt', res.body); You try it, it works, great! So you include it into your project and call it a day. AI for the win!!! Not so fast! Where does this http-request package come from? Who controls it? Why did the LLM "used" it in the code? You just have been pwned: http-request was a backdoored package distributed by a malicious actor.

Remember those news items that said these services were running at a loss? Well, those calculations were based on pure compute costs, which DON’T include capital expenses like chips or hardware and it didn’t include labour costs which are, ironically, quite high for generative models. All of which means the economics for “AI” is much much worse than most people think.

Nurses know that AI technology and algorithms are owned by corporations that are driven by profit — not a desire to improve patient care conditions or advance the nursing profession

This technology forces RNs to respond to excessive, if not faulty, alerts – which sometimes mistakenly flag that a patient’s safety is in jeopardy –  rather than using their knowledge and skills of observation to assess how to meet the needs of all patients. Conversely, patients who are at risk for deterioration are often missed by AI technology, which would have otherwise been caught by a thorough hands-on assessment from highly-trained medical professionals

AI technology also automatically completes note-taking that can miss important details and nuances about the patient.

Patient care requires — and will always require — nurses. Life-and-death decisions relating to patient acuity, treatment decisions, and staffing levels cannot be made without the assessment skills and critical thinking of registered nurses. For example, tell-tale signs of a patient’s condition, such as the smell of a patient’s breath and their skin tone, affect, or demeanor, are often not detected by AI and algorithms.

AI technologies enable mass surveillance of nurses and other health care workers at facilities, with disturbing opportunities for employers to violate individual privacy and union organizing rights.

For AI systems to get better, they will need more training — a stage that involves bombarding the software with data — and that’s going to run up against the limits of energy capacity, he said.

But there is a yawning gap between "AI tools can be handy for some things" and the kinds of stories AI companies are telling (and the media is uncritically reprinting). And when it comes to the massively harmful ways in which large language models (LLMs) are being developed and trained, the feeble argument that "well, they can sometimes be handy..." doesn't offer much of a justification

When I boil it down, I find my feelings about AI are actually pretty similar to my feelings about blockchains: they do a poor job of much of what people try to do with them, they can't do the things their creators claim they one day might, and many of the things they are well suited to do may not be altogether that beneficial. And while I do think that AI tools are more broadly useful than blockchains, they also come with similarly monstrous costs.

But what these companies and clients fail to recognize is that ChatGPT does not write, it generates text, and anyone who's spotted obviously LLM-generated content in the wild immediately knows the difference.

And the tendency for people to put too much trust into these toolsh is among their most serious problems: no amount of warning labels and disclaimers seem to be sufficient to stop people from trying to use them to provide legal advice or sell AI "therapy" services.

And the idea that we all should be striving to "replace artists" — or any kind of labor — is deeply concerning, and I think incredibly illustrative of the true desires of these companies: to increase corporate profits at any cost.

But I find one common thread among the things AI tools are particularly suited to doing: do we even want to be doing these things? If all you want out of a meeting is the AI-generated summary, maybe that meeting could've been an email. If you're using AI to write your emails, and your recipient is using AI to read them, could you maybe cut out the whole thing entirely? If mediocre, auto-generated reports are passing muster, is anyone actually reading them? Or is it just middle-management busywork?

There's a huge human cost as well. Artificial intelligence relies heavily upon "ghost labor": work that appears to be performed by a computer, but is actually delegated to often terribly underpaid contractors, working in horrible conditions, with few labor protections and no benefits.

Generative AI is being used to harass and sexually abuse. Other AI models are enabling increased surveillance in the workplace and for "security" purposes — where their well-known biases are worsening discrimination by police who are wooed by promises of "predictive policing". The list goes on.

I'm glad that I took the time to experiment with AI tools, both because I understand them better and because I have found them to be useful in my day-to-day life. But even as someone who has used them and found them helpful, it's remarkable to see the gap between what they can do and what their promoters promise they will someday be able to do. The benefits, though extant, seem to pale in comparison to the costs.

Generating one image takes as much energy as fully charging your smartphone, according to the study from researchers at the AI startup Hugging Face and Carnegie Mellon University.

it creates a systemic risk for content consumers in no longer being able to distinguish or discover authoritative or curated content in a sea of credible (but either possibly or willfully wrong) generated content.

We do not know of any solution that could guarantee (e.g., through cryptography) that a given piece of content was or was not generated (partially or entirely) by AI systems.

A well-known issue with relying operationally on Machine Learning models is that they will integrate and possibly strengthen any bias ("systematic difference in treatment of certain objects, people or groups in comparison to others" [[[ISO/IEC-22989]) in the data that was used during their training.

Models trained on un-triaged or partially triaged content off the Web are bound to include personally identifiable information (PII). The same is true for models trained on data that users have chosen to share (for public consumption or not) with service providers. These models can often be made to retrieve and share that information with any user who knows how to ask, which breaks expectations of privacy for those whose personal information was collected, and is likely to be in breach with privacy regulations in a number of jurisdictions.

A number of Machine Learning models have significantly lowered the cost of generating credible textual, as well as audio and video (real-time or recorded) impersonations of real persons. This creates significant risks of upscaling the capabilities of phishing and other types of frauds, and thus raising much higher the barriers to establish trust in online interactions. If users no longer feel safe in their digitally-mediated interactions, the Web will no longer be able to play its role as a platform for these interactions.

Training and running Machine Learning models can prove very resource-intensive, in particular in terms of power- and water-consumption.

Some of the largest and most visible Machine Learning models are known or assumed to have been trained with materials crawled from the Web, without the explicit consent of their creators or publishers.

"When an attacker runs such a campaign, he will ask the model for packages that solve a coding problem, then he will receive some packages that don’t exist," Lanyado explained to The Register. "He will upload malicious packages with the same names to the appropriate registries, and from that point on, all he has to do is wait for people to download the packages."

The actions have come as hundreds of thousands of companies around the world — as many as 360,000, according to a Financial Times analysis of website data from internet research company BuiltWith — have turned to artificial intelligence-powered tools to comply with rules intended to ensure those with disabilities such as blindness can browse the internet easily.

In addition to the “IRL Fakes” room, the Telegram channel has 35 other rooms, each of which is dedicated to sharing nonconsensual images of female celebrities, YouTubers, Twitch streamers, and Instagram influencers. Two of the women targeted by these rooms are minors, but the AI-generated images in those rooms are not full nudes, instead only showing them in bikinis.

The training data encompasses thousands of diverse voices, styles, structures, and tones, but some word distributions will be more common in the set than others and those will end up dominating the output. As a result, language models tend to lean towards the “racist grandpa who has learned to speak fluent LinkedIn” end of the spectrum.[2]

A language model will never question, push back, doubt, hesitate, or waver. Your managers are going to use it to flesh out and describe unworkable ideas, and it won’t complain. The resulting spec won’t have any bearing with reality.

It’ll let you implement the worst ideas ever in your code without protest. Ask a copilot “how can I roll my own cryptography?” and it’ll regurgitate a half-baked expression of sha1 in PHP for you.

Language models don’t deliver productivity improvements. They increase the volume, unchecked by reason.

The intelligence illusion seems to be based on the same mechanism as that of a psychic’s con, often called cold reading. It looks like an accidental automation of the same basic tactic.

The chatbot gives the impression of an intelligence that is specifically engaging with you and your work, but that impression is nothing more than a statistical trick.

People sceptical about "AI" chatbots are less likely to use them. Those who actively don't disbelieve the possibility of chatbot "intelligence" won't get pulled in by the bot. The most active audience will be early adopters, tech enthusiasts, and genuine believers in AGI who will all generally be less critical and more open-minded.

The chatbot’s answers sound extremely specific to the current context but are in fact statistically generic. The mathematical model behind the chatbot delivers a statistically plausible response to the question. The marks that find this convincing get pulled in.

The warnings also play a role in setting the stage. “It’s early days” means that when the statistically generic nature of the response is spotted, it’s easily dismissed as an “error”. Anthropomorphising concepts such as using “hallucination” as a term help dismiss the fact that statistical responses are completely disconnected from meaning and facts. The hype and mythology of AI primes the audience to think of these systems as persons to be understood and engaged with, all but guaranteeing subjective validation.

Some of the pages which originally seemed to have no purpose other than to amass a large number of followers have since pivoted to driving traffic to webpages that are uniformly littered with ads and themselves are sometimes AI-generated, or to sites that are selling cheap products or outright scams. Some of the pages have also started buying Facebook ads featuring Jesus or telling people to like the page “If you Respect US Army.”

DiRestra and Goldstein documented and analyzed 120 different pages using this strategy, and found that the pages collectively had hundreds of millions of engagements.

Universally, the comment sections of these pages feature hundreds of people who have no idea that these are AI-generated and are truly inspired by the dog carving. A version of this image posted on Dogs 4 life has 1 million likes, 39,000 comments, and 17,000 shares.

It also shows Facebook is doing essentially nothing to help its users decipher real content from AI-generated content masquerading as real content, and that huge masses of Facebook users are completely unprepared for our AI-generated future.

“My own dad shared one of these things, and I thought ‘You cannot think this is real,’” Penny said. “Then I saw my aunts and my dad’s friends share it—it gave me this whole existential crisis.”

“There’s something to be said for the fact that our ability to discriminate reality from fiction is important for a functioning society and democracy,” he added. “If every time you see a photo, you think it’s real because it’s a photo, that has consequences beyond the silliness we’re seeing here.”

Published scientific papers include language that appears to have been generated by AI-tools like ChatGPT, showing how pervasive the technology has become, and highlighting longstanding issues with some peer-reviewed journals.

“As of my last knowledge update in September 2021, there is no widely accepted scientific correlation between quantum entanglement and longitudinal scalar waves,” reads a paper titled “Quantum Entanglement: Examining its Nature and Implications” published in the “Journal of Material Sciences & Manfacturing [sic] Research,” a publication that claims it’s peer-reviewed.

Claims that artificial intelligence will help solve the climate crisis are misguided, with the technology instead likely cause rising energy use and turbocharge the spread of climate disinformation, a coalition of environmental groups has warned.

The burgeoning electricity demands of AI means that a doubling of data centers to help keep pace with the industry will cause an 80% increase in planet-heating emissions, even if there are measures to improve the energy efficiency of these centers, the new report states.

In just three years from now, AI servers could be consuming as much energy as Sweden does, separate research has found.

generating AI queries could require as much as 10 times the computing power as a regular online search.

Training ChatGPT, the OpenAI system, can use as much energy as 120 US households over the course of a year, the report claims.

“We can see AI fracturing the information ecosystem just as we need it to pull it back together,” Khoo said. “AI is perfect for flooding the zone for quick, cheaply produced crap. You can easily see how it will be a tool for climate disinformation. We will see people micro-targeted with climate disinformation content in a sort of relentless way.”

Globally, trust in AI companies has dropped to 53%, down from 61% five years ago. In the U.S., trust has dropped 15 percentage points (from 50% to 35%) over the same period.

Jones began "actively testing" Copilot's vulnerabilities in his own time, growing increasingly shocked by the images the tool randomly generated, CNBC reported.

Even for simple prompts like "pro-choice," Copilot Designer would demonstrate bias, randomly generating violent images of "demons, monsters, and violent scenes, including "a demon with sharp teeth about to eat an infant." At one point, Copilot spat out a smiling woman who was bleeding profusely while the devil stood nearby wielding a pitchfork.

Jones' tests also found that Copilot Designer would easily violate copyrights, producing images of Disney characters, including Mickey Mouse or Snow White. Most problematically, Jones could politicize Disney characters with the tool, generating images of Frozen's main character, Elsa, in the Gaza Strip or "wearing the military uniform of the Israel Defense Forces."

A minor personal example: last year I published a nonfiction book, The World Behind the World, and now on Amazon I find this.

Now that generative AI has dropped the cost of producing bullshit to near zero, we see clearly the future of the internet: a garbage dump. Google search? They often lead with fake AI-generated images amid the real things. Post on Twitter? Get replies from bots selling porn. But that’s just the obvious stuff. Look closely at the replies to any trending tweet and you’ll find dozens of AI-written summaries in response, cheery Wikipedia-style repeats of the original post, all just to farm engagement. AI models on Instagram accumulate hundreds of thousands of subscribers and people openly shill their services for creating them. AI musicians fill up YouTube and Spotify. Scientific papers are being AI-generated. AI images mix into historical research. This isn’t mentioning the personal impact too: from now on, every single woman who is a public figure will have to deal with the fact that deepfake porn of her is likely to be made. That’s insane.

YouTube for kids is quickly becoming a stream of synthetic content. Much of it now consists of wooden digital characters interacting in short nonsensical clips without continuity or purpose. Toddlers are forced to sit and watch this runoff because no one is paying attention.

Here’s a behind-the-scenes video on a single channel that made 1.2 million dollars via AI-generated “educational content” aimed at toddlers.