How to spot manufactured AI software reviews before you buy
Research shows 3 websites created over 215,000 'best software' pages to trick AI search engines. Recommendations from tools like Perplexity might be based on manufactured data rather than real user experience. Always verify software with a 10 minute hands-on test before buying.
Key Takeaways
- AI engines like Perplexity often cite content farms that mass-produce software reviews.
- 3 sites created over 215,000 pages to dominate AI search results.
- Manufactured reviews create a false consensus that leads to poor software choices.
- Check the Source list in AI tools to spot automated content.
- Use the 10 minute test during free trials to verify utility before paying.
The trap inside your AI search results
Business owners ask Perplexity or ChatGPT for software recommendations every day. It feels like a shortcut to avoid hours of research. You type a prompt, and the AI spits out a list of the top 5 tools for your industry. It looks clean and objective. The problem is the AI often reads from a script written by robots to trick other robots.
A recent investigation found that just 3 websites created 215,128 pages dedicated to best software pages for AI. These aren't deep reviews written by experts who actually used the tools. They're mass-produced pages designed to rank in search engines and feed AI answer engines. When you ask for a recommendation, the AI often cites these exact sites as its primary sources.
If you rely on these results, you aren't getting the best tool. You're getting the tool that was the best at gaming the system. Small business owners waste thousands of dollars on software subscriptions that don't solve their problems because they trusted a manufactured recommendation. You have to look under the hood before you enter your credit card details.
Why AI engines fall for the trick
AI models like Perplexity don't have personal experience. They don't know if a CRM (Customer Relationship Management) tool, which is software that tracks your customer interactions, is actually easy to use. They just look for patterns in data. When a small group of sites floods the internet with 215,000 pages of AI software reviews, they create a false consensus. The AI sees the same names mentioned repeatedly and assumes they're the industry standard.
This is a numbers game. These sites use automation to churn out reviews that look professional but lack substance. They use technical terms and generic praise to fill space. Because they're optimized for search engines, they show up first. When the AI scans the web to answer your question, it hits these content farms (sites that pump out low-quality articles to rank high in search) first and reports their biased data as fact. It's a feedback loop that rewards volume over quality.
The practitioner's reality
Testing these tools in real business environments reveals the truth. A top rated tool on a review site often lacks basic features like a reliable backup or a simple interface. The people writing those 215,000 pages have never had to answer a customer support call using the software they recommend. They're selling clicks, not solutions. You need to verify the utility of a tool by looking at its actual output, not just its star rating on a generated list.
How to verify a tool before you pay
You don't have to stop using AI for research, but you must change how you verify the answers. Treat every AI recommendation as a lead, not a conclusion. Your goal is to strip back the marketing fluff and see if the software actually functions. Here are 3 ways to do that this week.
First, check the Source list in your AI tool. If you see names you don't recognize or sites that review every single software category imaginable, be skeptical. Look for sources from actual practitioners or industry-specific forums where real people discuss their frustrations. If the source list is just these massive content hubs, ignore the recommendation and start a new search.
Second, look for a Changelog or a What's New page on the software's official website. A real software company updates their product constantly. If the last update was 6 months ago, or if they don't have a public record of improvements, it might be a zombie tool. These are often the ones pushed by manufactured review sites because they offer high affiliate commissions (money paid to the reviewer for every sale they generate).
Third, use the 10 minute test during a free trial. Don't just look at the dashboard. Try to perform one core task, like importing your contacts or setting up an automated email. If you can't figure it out in 10 minutes without a manual, the software is poorly designed. No amount of 5 star reviews from a bot can fix a bad user experience. If it feels clunky now, it'll be a nightmare when you're busy.
What to watch next
The volume of these manufactured reviews is going to grow. As it becomes cheaper to generate text, these 215,000 pages will turn into millions. We're entering a period where social proof is easily faked. You'll need to rely more on your own hands-on testing and recommendations from trusted peers rather than search engine results. Watch for AI tools that start to prioritize verified user data over general web crawling. That'll be the only way to cut through the noise.
FAQ
Why does Perplexity recommend bad software?
Perplexity scans the web for answers. If a group of sites creates thousands of pages recommending a specific tool, the AI sees that as authority and repeats the recommendation, even if the tool is poor quality.
How can I tell if a review site is manufactured?
Look for sites that review everything from gardening tools to enterprise software. If they don't have a specific niche and publish hundreds of lists daily, they're likely a content farm.
Is it still safe to use AI for software research?
Yes, but use it as a starting point. Once you have a list of names, go to independent forums or ask for a live demo to see if the tool actually works for your specific business needs.