Test answers with a question list
A short, fixed list of questions, asked the same way after every change, shows whether the change helped. Run it in Preview, where it draws on the daily preview allowance rather than credits, and again on the pages where visitors ask.
Build the list
Write 10 to 15 questions your visitors really ask, each with the answer you expect: the exact names of settings or features, the steps, and the page it should cite or link to.
Add two or three questions on other subjects the Assistant already answers well, to catch a change that breaks something unrelated.
Add two or three trick questions about things that don't exist, such as an invented setting name or a feature your product lacks. The right answer is that it isn't sure, or a correction, never confident detail.
Save the list with its wording fixed, so every run asks the same questions.
Run the list
Open the Assistant's Preview.
Click Clear Chat.
Ask one question from the list.
Score the answer: the right names, the right steps, the right citation or link.
Repeat from step 2 for each question.
Ask one question per conversation, because earlier questions shape later answers. Keep the scores from each run to compare before and after.
To compare models or reasoning levels, change Model or Reasoning and run the list again without saving: Preview answers with the unsaved setting. After a Knowledge Base change, wait until indexing has finished; until then Preview answers from the Knowledge Base as it was. See Knowledge Base.
Watch for made-up answers
A model can state invented details with full confidence. For an Assistant that answers about a product, this is the failure that matters most, and the list is what catches it.
An invented name is a hard failure. An Assistant that names a setting, menu item or value that doesn't exist, even once, is missing a fact.
The trick questions are the alarm. A sound Assistant hedges or corrects; one that makes things up elaborates.
Add the missing fact. Put the true fact in a Reasoning Fuel file, where a search will find it. If it still slips through, add that one fact to the Instructions. See Reasoning Fuel and Write an Assistant's Instructions.
Test where visitors ask
Both installs send the page the visitor is reading with each question; Preview sends none. Once the Assistant is in deployment, ask some of the list on the pages where visitors will ask it, and check that each link or citation opens the right page. These answers are billed like any visitor's. See Links and sources in answers.