ChatGPT vs Claude vs Gemini: Who Really Gives the “Best” AI Responses?
Why evaluating AI outputs is subjective, what trade-offs matter most, and how to decide which model fits your use case.
Which Foundation AI Model Gives the Best Responses?
I’ve been thinking about this lately. Ever since the growing popularity and adoption of foundation AI models, what is the real benchmark for saying that a model gives very good responses?
Foundation models like ChatGPT, Claude, and Gemini now support a wide range of use cases, both for personal use and for building AI-based software.
But how can we properly evaluate the responses from these models?
Other AI models, such as classifier models, are quite binary in their use. For example, with a model trained to detect spam emails, it is very easy to determine whether the model works optimally or not. You can test it on 10,000 sampled emails and immediately see which ones are spam and which are not. Easy peasy, lemon squeezy.
From those results, you can determine how efficient the model is: how many emails it correctly classified as spam and how many it wrongly classified.
However, when using foundation models like ChatGPT, Gemini, or Claude for personal use, it becomes much harder to determine whether a response is satisfactory or not.
Let’s do a quick experiment. I’ll give the same prompt to three foundation models and evaluate their responses. Imagine I want to create an AI-based service for generating poems, and I want to use an AI model for this feature.
Here is the prompt:
“Write a short poem for me.”
Below are the responses from OpenAI’s ChatGPT, Anthropic’s Claude, and Google’s Gemini.
To be honest, I’m not really a poet, but here are my thoughts on the three responses:
ChatGPT: It knows my name. shy 🙈
Maybe because this is my most-used foundation AI model, the poem honestly feels very personalised. It is almost like it reflects a stage in my life that I’m currently going through. In my opinion, the response is too personalised. Maybe I’m the only one who can tell, because I’m the only one who knows what I’ve discussed previously on the platform.
For context, I’m logged into my personal account for all of these experiments. However, remember that the idea here is to build an AI-based product that generates poems (not necessarily poems for me personally).Claude: Not bad at all, per se.
Gemini: If I needed an AI-generated poem, I would definitely choose Claude’s response over Gemini’s.
Now, this works because we are able to give feedback on the responses. We already have some idea of what a “good” poem might look like.
Let’s take it up a notch.
What if we want an AI response for a task where we don’t know what a good response should look like? What if we need a model to summarise a large body of text that we haven’t already read?
How do we know if it has done a good job?
Da dauuumm…
These are the real issues.
So how can we say which model provides the best response for a specific task? How do we know which model to adopt for personal use or for embedding AI features into applications via APIs?
The truth is that responses from foundation models are currently very subjective. There is no direct or easy method to assess their output quality universally.
However, there are metrics that individuals and developers can apply when deciding which model to adopt for a specific task.
1. Data Privacy
Any information you provide to a model via a prompt can potentially become part of the model’s knowledge base. This can lead to severe data leaks in use cases where privacy is critical.
Think about this from both national security and proprietary business perspectives. Before using a model, you need to understand where the model hosts its data. Are you comfortable with your data being hosted outside your national servers? What is the model’s policy on using prompt data for training and improvement?
2. Data Lineage and Copyright
Although IP laws around AI are still evolving, understanding a model’s training data sources is crucial (especially for commercial use). If a model is trained on copyrighted data and you use it to generate a digital product, can you fully defend your product’s intellectual property?
3. Performance
It has become obvious that many foundation models perform better when you pay for them (internal screaming). Open-source models often do not yet match proprietary ones in performance. Sad but true.
That said, history shows us that open-source software eventually catches up. At some point, open-source foundation models will likely reach parity with proprietary models.
4. Functionality
For me, this should be the defining factor.
Instead of asking which model is “the best,” we should ask how well a model performs the specific task we care about. Reviewing response quality for your particular use case is often the most practical way to decide.
In my earlier example, the experiment was inconclusive because the prompt itself wasn’t well-suited to the use case. I didn’t specify that the poem should be generic enough for something like an electronic billboard in a city centre. Expecting the model to infer that was unrealistic.
5. Control
When ChatGPT launched in 2022, people built entire SaaS products around its API. Wild times.
But what happens when a model is deprecated, its API changes, or access is restricted entirely? There’s a real risk in tightly coupling your product to a single external model. This needs to be considered before committing long-term.
6. Cost
Depending on pricing, rate limits, and SLAs, building your own model can sometimes be the better option.
Okay, calm down. I’m not screaming “build your own foundation model” like it’s a sponge cake. But when you factor in cost per API call and long-term scalability for commercial applications, it’s at least worth considering.
Conclusion
Selecting the appropriate foundation model is still a subjective process. It depends on factors such as cost, training data, privacy requirements, control, and most importantly, whether you’re satisfied with the responses for your use case.
I encourage everyone, especially application developers embedding AI features, to include a clear model-evaluation plan before committing to a foundation model. Being able to critically justify why you chose a particular model is now a best practice, not a nice-to-have.
Let me know in the comments what issues you’ve faced when using foundation models and whether switching models could have mitigated those problems.