Perplexity's AI Research Assistant: Can Journalists Trust It?

Some of my fellow journos have been talking up Perplexity.ai as an alternative to ChatGPT, so I checked it out. Also: latest #aiwriter news and opinions.

A Royal Portable typewriter that belonged to New York Times reporter Lester Bernstein (1920-2014).

I recently came across an article by Message Lab, a content marketing company teamed by experienced journalists like Miguel Helft and Laura Rich (both ex-New York Times). The article was entitled “Drowning in a Sea of Mediocre AI-Generated Content” and was written by Peter Burrows, an Editorial Director at Message Lab. Despite the skeptical-sounding headline, the theme of the piece was how to use AI to help research articles. Burrows first explained that, pre-generative AI, he used to google things when he needed to “get-smart” about a topic. But now he uses AI:

I won’t spend that many hours Googling again. My get-smart process now starts with ChatGPT or Perplexity.ai, my preferred AI chatbot (in part because it clearly labels the sources it drew from, easing my concerns about hallucinations or bias). I haven’t tracked the time spent to see how it compares to the “before times,” but I’m clearly learning more per hour with generative AI — and am more confident that I can quickly get answers as new questions come up.

I’d seen the name Perplexity.ai before — my old ReadWriteWeb colleague Marshall Kirkpatrick recently said it’s “a chatbot platform I've been using a lot lately” — but I hadn’t yet played with it. Burrows added in a footnote that “Perplexity includes sources and citations along with its answers, raising our confidence in the accuracy of its findings.”

I signed up to Perplexity on the free tier to check it out. In its FAQ, Perplexity calls itself an “AI research assistant” and says that its goal is “to make searching for information online with Perplexity feel like you have a knowledgeable assistant guiding you.”

Although not mentioned in its FAQ, Perplexity appears to use a fine-tuned version of OpenAI’s GPT 3.5 model, per this recent tweet endorsed by OpenAI’s CEO:

Pro Perplexity accounts (which start at $20 per month) allow users to “upgrade to Claude-2 or GPT-4, boost your Copilot uses, and upload more files.”

Ok, onto my tests. I’m currently planning an article for my employer The New Stack (a tech media publication) about serverless technologies in frontend development, so I decided to do some research on that in Perplexity. I started out with this question:

The response was a nice overview of serverless in frontend, along with 6 links to what seemed to be reputable sources.

Underneath the response were a few suggestions for “related” questions. I clicked the following one, since I did in fact want to see some specific examples: “what are some examples of serverless technologies used in frontend web development”

There were 7 example products listed, including AWS Lambda and Netlify Functions. I had both of those on my initial list to research more, however Perplexity did not mention the third product on my (manually created) shortlist: Vercel Functions. So I decided to ask a follow-up question about that, specifically in relation to Vercel’s arch-nemesis competitor, Netlify.

At first glance it seemed like a useful response, although as a tech journalist a couple of things immediately jumped out at me:

1. I wasn’t familiar with most of the sources it used, so I quickly realized I’d have to check all six links and decide for myself how reliable they were as sources.

2. The Perplexity response included a lot of industry jargon that I would need to dig more into; e.g. “Vercel features Edge caching, while Netlify Edge offers asynchronous procedures”.

Also, on closer inspection, I noticed that Perplexity was a little confused about some details. For instance, this sentence: “Additionally, Vercel supports Next.js, Nuxt.js, and Gatsby, while Netlify supports JavaScript, TypeScript, Go, and more.” This implies that only Netlify supports JavaScript in its serverless product. But, in fact, if you go to the official documentation for both Netlify and Vercel, you’ll discover that Vercel supports JavaScript in serverless just as much as Netlify does. (Even though frontend developers will almost certainly know that Next.js, Nuxt.js, and Gatsby are all JavaScript frameworks, this wasn’t made clear by Perplexity and so it’s potentially misleading for a journalist.)

One way for Perplexity to improve this situation going forward is to use official documentation for software among its sources, when dealing with technical questions like mine.

Vercel’s documentaion (emphasis mine)

I challenged Perplexity about the JavaScript detail in Vercel’s documentation, just to see how it would respond. However, similar to my experiences with ChatGPT, Perplexity’s “apology” didn’t seem very sincere — and, more importantly, didn’t make me trust that it had gotten other details correct.

Like other generative AI products, Perplexity clearly has some issues with accuracy and the framing of technical material. So to answer my own question in the headline of this newsletter, no I don’t entirely trust Perplexity. But it is still very useful as a journalism tool, primarily because it shows you its sources. Of course, as a journalist I will be checking those sources (and the sources not suggested!), but that’s how it should be for my profession.

I’ll add that the Perplexity user interface is slick and the “related” prompts are a classy touch. I’m even tempted to upgrade to the Pro tier, so that I can see whether using Claude-2 or GPT-4 improves the responses.

News 📰

• Perplexity has acquired Spellwise, an AI-powered keyboard writer for iOS. (RM: Perplexity also has a nifty iPhone app)

• Several writers made TIME magazine’s list of the “100 Most Influential People in AI”: Ted Chiang (scifi author), Kate Crawford (author of Atlas of AI), Charlie Brooker (of Black Mirror fame), Rootport (an author of Japanese manga).

• LangChain announces a hub for prompts; while LangChain is a developer tool, the company recognizes that “non-technical team members are participating in the process of editing and refining prompts.” Presumably that includes content folks.

• Matt Shumer, CEO of HyperWrite, announces gpt-author v2: “One prompt -> an entire fantasy novel! Just describe the high-level details, and a chain of AI systems will write an entire book for you in minutes.”

• HubSpot announces AI Assistants for marketing: “AI Assistants will work across the entire HubSpot platform to help teams draft content, create images, generate blog ideas, build websites, and develop reports—instantly.”

Perspectives 🗣️

• Steven Levy’s Wired cover story on OpenAI provides more evidence of the alarming world view of the company’s founders: “These are people who do not shy from casually using the term “super-intelligence.” They assume that AI’s trajectory will surpass whatever peak biology can attain.”

• Ted Chiang weighs in on AI super-intelligence: “The idea of a computer that can make itself smarter and become super intelligent, that is a really interesting story idea. But to hear people talk about it as if it’s something that is actually going to happen, that’s just jarring.”

• In the latest extract from Walter Isaacson’s upcoming bio of Elon Musk, in 2013 Google’s Larry Page apparently accused Musk of being a “specist” because he valued humans over AI. Musk responded, “Well, yes, I am pro-human. I f-cking like humanity, dude.” (RM: I’m team Musk on this one!)

• Elif Batuman discusses Proust with ChatGPT: “Didn’t I help train it in acting like a resentful servant, since I was behaving like an exasperated master?” (RM: I think we can all relate to these kinds of ChatGPT conversations!)

• Google CEO Sundar Pichai briefly mentions writing in his public memo on Google at 25: “We’re just beginning to see what the next wave of technology is capable of and how quickly it can improve. One million people are already using generative AI in Google Workspace to write and create.”

Reference 🧠

• Ethan Mollick embraces weirdness and gives his views on what it means to use AI as a writing tool (RM: great tips here, including to “simulate a reader” and doing “synonyms for paragraphs”).

• Tom Johnson on using AI for glossary definitions.

See you next week, my fellow augmented writers!