I might be exaggerating slightly, but if you look at the few new evaluation matrices for AI-powered search circulating, “relevancy” is often just one of several categories, evaluated in a highly subjective and “I-know-it-when-I-see-it” manner. This is baffling, given that a search engine (AI-powered or not) lives and dies on its ability to retrieve relevant results.

Also call for Sponsors!

Quick catch-up to what I have been writing and thinking about

Elicit.com, Consensus, and Undermind.ai are among the new leading comprehensive cross-disciplinary “AI-powered academic search engines” today. Thanks for reading Aaron Tay's Musings about Librarianship! Subscribe for free to receive new posts and support my work.

Introduction

Introduction In my last post, I argued that Deep Search—iterative retrieval that blends keyword, semantic, and citation chasing with LLM-based relevance judgments—is the real breakthrough behind today’s “Deep Research” tools. It consistently beats one-shot embedding search in recall/precision, and in hindsight, it’s what I loved all along (the “generation” step just came bundled). The price?

Back in 2022, I was hyped about Retrieval-Augmented Generation (RAG).The novelty of seeing a search engine spit out a direct answer — with citations! — in tools like Elicit and Perplexity felt like the future. I even predicted that this “answers-with-citations” model could become the prominent paradigm for academic search. Three years later, that prediction has partly come true.

I recently gave a 30-minute talk at the Librarian Futures Virtual Summit, and for the topic of "AI-powered search," I decided to play devil's advocate.

Disclosure : I am currently a member of Clarivate Academia AI Advisory Council but I am writing this in my personal capacity. Imagine a first‑year student typing “Tulsa race riot” into the library search box and being greeted with zero results—or worse, an error suggesting the topic itself is off‑limits. Thanks for reading Aaron Tay's Musings about Librarianship! Subscribe for free to receive new posts and support my work.

One of the most interesting things about teaching is that the best questions come after I’ve finished my talk. Yesterday, during Day 2 of my three-hour crash course on AI search at FSCI 2025, a participant looked at our side-by-side demo of Scopus (not Scopus AI), SciSpace (in standard, non-deep search mode), and AI2 PaperFinder and asked (paraphrased): Thanks for reading Aaron Tay's Musings about Librarianship!

Though my blog focuses on academic discovery and retrieval, these days you can’t really understand those topics without grappling with concepts like Transformer models, agents, and reasoning.