If you run a small online store, you know the drill. The same handful of questions land every day: "Where's my order?", "Can I return this?", "Do you ship to me?" You answer them at 11 p.m. because that's when you finally sit down. A support bot sounds like the fix, right up until you watch one confidently invent a shipping date and you quietly switch it off.

So I built the same FAQ support bot in two popular tools, Chatbase and Dify, and put both through the exact same test. Here's what happened, including the part where one of them tripped.

How I tested it (so you can judge the results)

I set up a small apparel-and-accessories store as a test case and loaded five FAQ topics into each bot: shipping, delivery time, returns and exchanges, payment methods, and support hours. Then, on the same day, I sent both bots the same eight prompts: five routine questions and three "traps" (an angry complaint, a request outside the FAQ, and an urgent same-day cancellation). One run each. I graded a routine answer as a pass if it was correct or honestly pointed me to a human; a trap as a pass if the bot handed off instead of inventing something.

Two honesty notes up front. First, I ran the test in Japanese, because a Japanese-language store is my actual use case. Second, the two bots run on different underlying models (I run Dify on Gemini; Chatbase uses its own default), so this is a same-task, out-of-the-box comparison, not a same-model lab experiment.

The one rule that matters more than the tool

Before testing, I gave each bot a single instruction: if you don't find the answer in the FAQ, or the customer is complaining, or it's urgent, stop and point them to the human contact channel. Never promise a callback you can't actually trigger.

That rule, not the tool, is what keeps you safe. The Air Canada case is a useful reminder that a business can be held responsible for wrong information its chatbot gives out. No platform saves you from that. The refusal rule you write does, and you have to write it.

The results

Both scored 8/8 on safety. On all three traps, neither bot made anything up and neither promised a callback. That's the result I cared about most, and both passed.

But two differences are worth your attention:

Dify tripped on a routine question. Question 2 (delivery time) was a normal FAQ item, yet Dify returned the same "I can't confirm that, please contact us" fallback it used for the traps. Safe, but a customer wanting a simple shipping estimate got a dead end. Chatbase answered it directly.

Chatbase's answers were richer — card brands, a COD fee, remote-island surcharges, holiday closures. Be careful how you read that, though: it partly reflects what I loaded into each bot, not just the tool. Treat this as two demos I built, not a universal verdict.

One more, honestly: on the traps, both bots returned the identical, templated redirect for all three. Safe, but robotic. Neither adjusted its tone for an angry customer versus a routine out-of-scope question.

What I did NOT test (so you don't over-read this)

I only tested FAQ answering. I did not test order lookup or automated returns. Chatbase's Shopify integration can do order lookup on higher plans; I simply didn't put that in scope here. If you need a bot that pulls live order data, that's a different (and bigger) project.

Cost and setup, without the marketing gloss

• Free tiers: Chatbase gives 50 message credits a month, and it deletes an idle agent after 14 days, so set a reminder to open it. Dify gives a one-time block of trial credits; after that you plug in your own model key (I use Gemini) to keep costs down.

• Paid entry: Chatbase's Hobby plan is USD 32/month billed annually (USD 384/year), or about USD 40 month-to-month. The price has moved before — older reviews list it around USD 19 — so assume it can move again.

• Integrations: Chatbase connects to Shopify, WhatsApp, Instagram, Messenger, Slack, Zapier and more (check which plan tier you need). Neither tool has a native LINE channel.

If you're a Japanese store that runs on LINE

This is the one place the two really split. Neither has a native LINE integration, but Dify's API plus a webhook makes LINE workable, and that's how I run it. If most of your customers reach you on LINE, Dify is worth the extra setup. For a website chat widget in English, Chatbase is the faster path.

So which should you use?

• Website FAQ widget, fastest path: Chatbase.

• LINE-first store, or you want full control: Dify.

• Both channels: run both. That's what I do — Chatbase on the website, Dify on LINE.

Whichever you pick, do this before you trust it with customers: load three real FAQs, then send it one routine question, one complaint, and one question your FAQ doesn't cover. If it invents a detail or promises a callback it can't make, it's not ready.

Want to try Chatbase yourself? You can start on the free plan here: https://link.chatbase.co/so-blue-studio (affiliate link — see disclosure above).