ci)jdZddlmZmZmZddlmZddlmZddlm Z GddejZ y) zModel-based Pairwise Metric.)CallableOptionalUnion)generative_models)_base)metric_prompt_templatec eZdZdZdddedeejefdeee je egeffffdZ e dee je egefffd ZxZS) PairwiseMetricaA Model-based Pairwise Metric. A model-based evaluation metric that compares two generative models' responses side-by-side, and allows users to A/B test their generative models to determine which model is performing better. For more details on when to use pairwise metrics, see [Evaluation methods and metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#pointwise_versus_pairwise). Result Details: * In `EvalResult.summary_metrics`, win rates for both the baseline and candidate model are computed. The win rate is computed as proportion of wins of one model's responses to total attempts as a decimal value between 0 and 1. * In `EvalResult.metrics_table`, a pairwise metric produces two evaluation results per dataset row: * `pairwise_choice`: The choice shows whether the candidate model or the baseline model performs better, or if they are equally good. * `explanation`: The rationale behind each verdict using chain-of-thought reasoning. The explanation helps users scrutinize the judgment and builds appropriate trust in the decisions. See [documentation page](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#understand-results) for more details on understanding the metric results. Usage Examples: ``` baseline_model = GenerativeModel("gemini-1.0-pro") candidate_model = GenerativeModel("gemini-1.5-pro") pairwise_groundedness = PairwiseMetric( metric_prompt_template=MetricPromptTemplateExamples.get_prompt_template( "pairwise_groundedness" ), baseline_model=baseline_model, ) eval_dataset = pd.DataFrame({ "prompt" : [...], }) pairwise_task = EvalTask( dataset=eval_dataset, metrics=[pairwise_groundedness], experiment="my-pairwise-experiment", ) pairwise_result = pairwise_task.evaluate( model=candidate_model, experiment_run_name="gemini-pairwise-eval-run", ) ``` N)baseline_modelmetricrr c6t|||||_y)aInitializes a pairwise evaluation metric. Args: metric: The pairwise evaluation metric name. metric_prompt_template: Pairwise metric prompt template for performing the pairwise model-based evaluation. A freeform string is also accepted. baseline_model: The baseline model for side-by-side comparison. If not specified, `baseline_model_response` column is required in the dataset to perform bring-your-own-response(BYOR) evaluation. )rr N)super__init___baseline_model)selfr rr __class__s R/tmp/pip-target-z3e9_cxr/lib/python/vertexai/evaluation/metrics/pairwise_metric.pyrzPairwiseMetric.__init__Us'* #9   .returnc|jS)N)r)rs rr zPairwiseMetric.baseline_modelps###r)__name__ __module__ __qualname____doc__strrmetric_prompt_template_basePairwiseMetricPromptTemplaterrGenerativeModelrrpropertyr __classcell__)rs@rr r s6B ..!& ' D Dc I! .! #33XseSj5II J .6$  00(C5#:2FF G$$rr N) rtypingrrrvertexairvertexai.evaluation.metricsrrr_ModelBasedMetricr rrr&s/"#,,&- X$U,,X$r