Managing Evaluators

Managing Evaluators

Start from the AlchemiStudio Console and open the Evaluators page, where you define score rubrics and manage judges.

On the Evaluators page, you can see the list of existing evaluators and create new ones. The page displays evaluator cards with their status, type, and configuration details.

The Evaluators section shows one active evaluator and a New button to create additional evaluators. From here, you can select an evaluator to view or edit its configuration.

The Response Query evaluator card displays its active status, accuracy score type, and Auto LLM judge configuration. Selecting this card opens the detail panel on the right.

From the Evaluators list, select Response Query to view its configuration. The right panel displays the evaluator’s definition, test options, and score history.

The Definition tab displays the evaluator’s complete configuration, including score name, type, evaluator endpoint, rubric, and model settings. This view lets you review how the evaluator judges responses.

With the Definition tab open, examine the judge’s rubric, model configuration, score name, and evaluator type. This panel shows how the evaluator scores responses.

The panel displays the score name, type, evaluator endpoint, rubric text, and model configuration. These fields define how the LLM judge scores responses based on the provided rubric.

The Definition panel displays the evaluator’s score name, type, rubric, and model configuration. These properties are immutable; to modify them, duplicate the evaluator.

The panel now shows the complete evaluator configuration, including the Duplicate to edit button at the bottom. Evaluators cannot be edited directly once created.

The Test button switches to the testing interface, where you can send sample inputs to the configured judge and preview results without affecting production scores.

With the Test tab selected, the panel displays fields for sample input and output. Test results are transient and not persisted as scores.

The panel explains that test results are transient and not saved as scores. You can send one-off samples to the configured judge.

The expanded Test panel shows all available fields, including optional expected output and temperature settings. This interface lets you configure and run a single test against the evaluator’s rubric.

From the Test panel, select the Scores tab to view scoring history and averages. This tab displays persistent score data collected from experiments and runs.

The Scores panel shows the average over recorded scores and a list of recent results. Currently, no scores have been recorded yet for this evaluator.

Click the Response Query evaluator in the left sidebar to return to its detail view. This completes the walkthrough of the evaluator’s Definition, Test, and Scores tabs.