Why this game?

Who's that LLM? started with something a friend said:

“Over time I think I could tell an Opus 4.5 from a 4.7 and a 4.8. Like a wine...”

I laughed, and then I couldn't stop thinking about it. Can people actually tell models apart from just a few messages? Especially models from the same family? Or do we only think we can?

I'm a data scientist, so my first instinct was to collect data. That's really what this game is for. Sure, it's fun to play, but every round also tells me something: which models get confused with each other, whether big models are easy to spot next to small ones, whether a model's origin shows through, and plenty of things I haven't thought of yet.

I plan to share this data openly for research later on. Until then, the statistics page shows what the data says so far.

What data do I store?

I keep your username and your game data: the messages, token counts, your guesses, which model you were talking to, and the other details of each round. That's it. I don't collect any other personal information. Signing in is handled by Supabase, and by Google if you choose it. Your email is stored there only so you can log in, and I never use it for anything else. I never see your password.

Your messages are sent to OpenRouter and to the company that runs the model you are talking to, so they can write the reply. Their own policies apply to what they receive. The site is hosted on Vercel, which also counts page views with Vercel Analytics. That count does not use cookies and is not tied to your account.

You can delete your account at any time from your profile. Your username comes off the leaderboard. Finished games stay in the statistics without your username.

Who am I?

I'm Gael, a data scientist based in Fribourg, Switzerland, with a PhD from EPFL. Funny enough, my master thesis was about beer reviews: do people rate a beer for what it is, or because of what the community already said about it? So the wine comparison that started this game hit close to home. It's the same kind of question: can we really taste the difference, or do we just think we can?

By the way, if you want to chat about the game, the data, or anything else, you can find me on LinkedIn and GitHub.

Do you want to contribute?

Who's that LLM? is a small side project, and it gets better with help. If you have an idea, want to contribute, or found a bug, you're very welcome to open an issue on GitHub. Whether it's a typo or a big suggestion, I'd love to hear about it.

Do you want to support the project?

I built this game on my own time, and I pay for the API calls myself. If you enjoy it and want to help keep it running, you can leave me a small tip on Ko-fi. It's completely optional, but it means a lot.