ci:dZddlZddlmZmZmZmZmZddlm Z ddl m Z GddejZGdd eZGd d eZGd d eZGddeZy)z$Base classes for evaluation metrics.N)AnyCallableDictLiteralUnion) constants)metric_prompt_templatec8eZdZdZdefdZdZedefdZy)_Metricz)The abstract class for evaluation metric.metricc||_yN_metric)selfr s H/tmp/pip-target-z3e9_cxr/lib/python/vertexai/evaluation/metrics/_base.py__init__z_Metric.__init__s  c|jSr) metric_namers r__str__z_Metric.__str__"srreturnc|jSrrrs rrz_Metric.metric_name%s ||rN) __name__ __module__ __qualname____doc__strrrpropertyrrrr r s13s Srr c`eZdZdZdedeejejefffd Z xZ S)_ModelBasedMetricaA Model-based Metric. An evaluation metric that evaluates generative AI model responses with another generative model as a judge. This metric can be used to evaluate a single model, or two models side-by-side. For more details on when to use model-based metrics, see [Evaluation methods and metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval). r r cTt||||_t||_y)aInitializes the model-based evaluation metric. Args: metric: Generic model based metric name. metric_prompt_template: A metric prompt template for performing the model-based evaluation. A freeform string is also accepted. r N)superr_raw_metric_prompt_templaterr )rr r __class__s rrz_ModelBasedMetric.__init__5s+" '+A(&)*@&A#r) rrrrrrmetric_prompt_template_basePointwiseMetricPromptTemplatePairwiseMetricPromptTemplater __classcell__r(s@rr#r#*sIBB!& ' E E ' D D  ! BBrr#cLeZdZdZdedeeeefgeeeffffd ZxZ S) CustomMetricaThe custom evaluation metric. A fully-customized CustomMetric that can be used to evaluate a single model by defining a metric function for a computation-based metric. The CustomMetric is computed on the client-side using the user-defined metric function in SDK only, not by the Vertex Gen AI Evaluation Service. Attributes: name: The name of the metric. metric_function: The user-defined evaluation function to compute a metric score. Must use the dataset row dictionary as the metric function input and return per-instance metric result as a dictionary output. The metric score must mapped to the name of the CustomMetric as key. namemetric_functionc@t||||_||_y)z"Initializes the evaluation metric.N)r&rr0r1)rr0r1r(s rrzCustomMetric.__init__[s!  .r) rrrrrrrrrr,r-s@rr/r/KsG  / /" #s(^  cN   / /rr/cVeZdZdZdeej jffd ZxZ S)_AutomaticMetricaAn automatic metric that computes deterministic score based on reference. An lexicon-based evaluation metric that evaluate a generative model's response on the given evaluation task with reference ground truth answers. It is a type of pointwise evaluation metric. For more details on when to use automatic metrics, see [Evaluation methods and metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval). r c&t||y)z|Initializes the automatic evaluation metric. Args: metric: The automatic evaluation metric name. r%N)r&r)rr r(s rrz_AutomaticMetric.__init__us 'r) rrrrrrMetricROUGErr,r-s@rr4r4is+  ( ((../ ( (rr4cveZdZdZdedededeffd ZedefdZedefd Zedefd Z xZ S) _TranslationMetricahA Translation Metric. Evaluates a score for the given instance using an underlying machine learning model. For now, only COMET and MetricX are supported. For more details on how to evaluate translation, see [Evaluation a translation model](https://cloud.google.com/vertex-ai/generative-ai/docs/models/run-evaluation#translation). r0versionsource_languagetarget_languagecP||_||_||_t||y)aInitializes the Translation metric. Args: name: The name of the metric. version: The version to use for evaluation. source_language: The source language of the translation. target_language: The target language of the translation. r%N)_version_source_language_target_languager&r)rr0r:r;r<r(s rrz_TranslationMetric.__init__s,  / / %rrc|jSr)r>rs rr:z_TranslationMetric.versions }}rc|jSr)r?rs rr;z"_TranslationMetric.source_language$$$rc|jSr)r@rs rr<z"_TranslationMetric.target_languagerCr) rrrrrrr r:r;r<r,r-s@rr9r9s &&& &  &*%%%%%%rr9)rabctypingrrrrrvertexai.evaluationrvertexai.evaluation.metricsr r)ABCr r#r/r4r9r!rrrJs^"+ 66) cgg BBB/7/<(w(0+%+%r