ciS?dZddlmZmZmZddlmZddlmZeje Z dZ deee e ffdZGdd ejZGd d eZGd d eZy)zBMetric prompt template classes for model-based metrics evaluation.)DictListOptional)base)prompt_template elementscn|ytjdt|jDS)z?Serializes dictionary to ordered string value without brackets.c30K|]\}}|d|yw)z: N).0keyvalues Y/tmp/pip-target-z3e9_cxr/lib/python/vertexai/evaluation/metrics/metric_prompt_template.py z*serialize_dict_in_order..#sVzsEC55'*Vs)_NEWLINEjoinsorteditems)r s rserialize_dict_in_orderrs, ==VVHNNDT=UV VVceZdZdZddddddeeefdeeefdeedeedeeeefd eed eeefd Ze d efd Z y)_MetricPromptTemplatezBMetric prompt template for generic model-based metrics evaluation.N) instructionevaluation_stepsmetric_definitionfew_shot_examplescriteria rating_rubricinput_variablesrrrrc||_||_||_||_||_||_||_|j|_y)z%Initializes a metric prompt template.N) _input_variables _instruction_metric_definition _criteria_rating_rubric_evaluation_steps_few_shot_examples__str__template)selfrr r!rrrrs r__init__z_MetricPromptTemplate.__init__)sH!0'"3!+!1"3  rreturnc|jS)N)r+r,s r prompt_dataz!_MetricPromptTemplate.prompt_data@s }}r) __name__ __module__ __qualname____doc__rstrrrr-propertyr1r rrrr&sL&*59+/15'sCx.'CH~ ' c ' c] '#4S>2'$C='$DI.'.SrrceZdZdZdddddddeeefdeeefdeeedeedeed eeeefd eeeffd Zd efd Z d eeeffdZ dZ dZ xZ S)PointwiseMetricPromptTemplatezCPointwise metric prompt template for pointwise model-based metrics.Nr!rrrrrr r!rrrrc |sg}tjdtt|dgz}|xs|j }|xs|j }t ||||||||y)aInitializes a pointwise metric prompt template. Args: criteria: The standards and measures used to evaluate the model responses. It is a dictionary of criterion names and criterion definitions. rating_rubric: A dictionary mapping of rating name and rating definition, used to assign ratings or scores based on specific criteria. input_variables: An optional list of input fields to use in the metric prompt template for generating model-based evaluation results. Model "response" column is included by default. If metric_column_mapping is provided, the mapping values of the input fields will be used to retrieve data from the evaluation dataset. instruction: The general instruction to the model that performs the evaluation. If not provided, a default pointwise metric instruction will be used. metric_definition: The optional metric definition. It is a string describing the metric to be evaluated at a high level. If not provided, this field will not be included in the prompt template. evaluation_steps: The optional gudelines of evaluation steps. A dictionary of evaluation step name and evaluation step definition. If not provided, a default pointwise metric evaluation steps will be used. few_shot_examples: The optional list of few-shot examples to be used in the prompt, to provide the model with demonstrations of how to perform the evaluation, and improve the evaluation accuracy. If not provided, this field will not be included in the prompt template. zsThe `input_variables` parameter is empty. Only the `response` column is used for computing this model-based metric.responser!rr rrrrN)_LOGGERinfolistset!get_default_pointwise_instruction&get_default_pointwise_evaluation_stepssuperr- r,rr r!rrrr __class__s rr-z&PointwiseMetricPromptTemplate.__init__HsP O LLI s?j\#ABC!MT%K%K%M   M K K M  +'#/-/  rr.c y)?Returns the default instruction for the metric prompt template.a You are an expert evaluator. Your task is to evaluate the quality of the responses generated by AI models. We will provide you with the user prompt and an AI-generated responses. You should first read the user input carefully for analyzing the task, and then evaluate the quality of the responses based on the Criteria provided in the Evaluation section below. You will assign the response a rating following the Rating Rubric and Evaluation Steps. Give step by step explanations for your rating, and only choose ratings from the Rating Rubric.r r0s rrBz?PointwiseMetricPromptTemplate.get_default_pointwise_instructions   rc dddS)DReturns the default evaluation steps for the metric prompt template.zhAssess the response in aspects of all criteria provided. Provide assessment according to each criterion.zzScore based on the rating rubric. Give a brief rationale to explain your evaluation considering each individual criterion.)Step 1Step 2r r0s rrCzDPointwiseMetricPromptTemplate.get_default_pointwise_evaluation_stepss;R  rcd|jtdg}|jr |jd|jdg|jdt |j ddt |j dg|jr)|jdt |jdg|jr3|jdtj|jdg|jd d g|jD]"}|d k(r |jd |d |dg$|jtddgtj|S)z>:;2>"*4+>+>?@C     ! ! & - -).t/E/EFGrJ   " " & - -,}}T%<%<=>bA  #)) 8:J K #33 N+ & - ->*+(-   #)),  }}788rc<d|jd|jdS)Nz*PointwiseMetricPromptTemplate(prompt_data= , variables=)r1 variablesr0s r__repr__z&PointwiseMetricPromptTemplate.__repr__s,89I9I8JK..) , r)r2r3r4r5rr6rrr-rBrCr*rd __classcell__rFs@rr9r9EsM04%)+/5915> sCx.> CH~ > "$s), > c] > $C=> #4S>2> $DI.> @  3    S#X  89t rr9ceZdZdZdddddddeeefdeeefdeeedeedeed eeeefd eeeffd Zd efd Z d eeeffdZ dZ dZ xZ S)PairwiseMetricPromptTemplatezAPairwise metric prompt template for pairwise model-based metrics.Nr:rr r!rrrrc |sg}tjdtt|ddgz}|xs|j }|xs|j }t ||||||||y)aHInitializes a pairwise metric prompt template. Args: criteria: The standards and measures used to evaluate the model responses. It is a dictionary of criterion names and criterion definitions. rating_rubric: A dictionary mapping of rating name and rating definition, used to assign ratings or scores based on specific criteria. input_variables: An optional list of input fields to use in the metric prompt template for generating model-based evaluation results. Candidate model "response" column and "baseline_model_response" column are included by default. If metric_column_mapping is provided, the mapping values of the input fields will be used to retrieve data from the evaluation dataset. instruction: The general instruction to the model that performs the evaluation. If not provided, a default pairwise metric instruction will be used. metric_definition: The optional metric definition. It is a string describing the metric to be evaluated at a high level. If not provided, this field will not be included in the prompt template. evaluation_steps: The optional gudelines of evaluation steps. A dictionary of evaluation step name and evaluation step definition. If not provided, a default pairwise metric evaluation steps will be used. few_shot_examples: The optional list of few-shot examples to be used in the prompt, to provide the model with demonstrations of how to perform the evaluation, and improve the evaluation accuracy. If not provided, this field will not be included in the prompt template. zThe `input_variables` parameter is empty. Only the `response` and `baseline_model_response` columns are used for computing this model-based metric.r<baseline_model_responser=N)r>r?r@rA get_default_pairwise_instruction%get_default_pairwise_evaluation_stepsrDr-rEs rr-z%PairwiseMetricPromptTemplate.__init__sP O LL,   :/H"II J "LT%J%J%L   L J J L  +'#/-/  rr.c y)rHaYou are an expert evaluator. Your task is to evaluate the quality of the responses generated by two AI models. We will provide you with the user input and a pair of AI-generated responses (Response A and Response B). You should first read the user input carefully for analyzing the task, and then evaluate the quality of the responses based on based on the Criteria provided in the Evaluation section below. You will first judge responses individually, following the Rating Rubric and Evaluation Steps. Then you will give step by step explanations for your judgement, compare results to declare the winner based on the Rating Rubric and Evaluation Steps.r r0s rrkz=PairwiseMetricPromptTemplate.get_default_pairwise_instruction+s  G rcddddddS)rJz-Analyze Response A based on all the Criteria.z-Analyze Response B based on all the Criteria.zcCompare the overall performance of Response A and Response B based on your analyses and assessment.zjOutput your preference of "A", "SAME" or "B" to the pairwise_choice field according to the Rating Rubrics.z9Output your assessment reasoning in the explanation field)rKrLzStep 3zStep 4zStep 5r r0s rrlzBPairwiseMetricPromptTemplate.get_default_pairwise_evaluation_steps;s&FE4JR  rcd|jtdg}|jr |jd|jdg|jdt |j ddt |j dg|jr)|jdt |jdg|jr3|jdtj|jdg|jd d g|jD]!}|d vr|jd |d |dg#|jgdtj|S)z;Serializes the pairwise metric prompt template to a string.rNrOrPrrQrRrSrTz) # User Inputs and AI-generated ResponsesrU)r<rjrVrWrX)z ## AI-generated Responsesz### Response Az{baseline_model_response} z### Response BrYrZr\s rr*z$PairwiseMetricPromptTemplate.__str__Ks   !   & "  " " & - -*../r2  #))*4>>:;2>"*4+>+>?@C     ! ! & - -).t/E/EFGrJ   " " & - -,}}T%<%<=>bA  #)) 9;K L #33 N!HH & - ->*+(-   #))  }}788rc<d|jd|jdS)Nz)PairwiseMetricPromptTemplate(prompt_data=r`rarbr0s rrdz%PairwiseMetricPromptTemplate.__repr__s,78H8H7IJ..) , r)r2r3r4r5rr6rrr-rkrlr*rdrerfs@rrhrhsK04%)+/5915A sCx.A CH~ A "$s), A c] A $C=A #4S>2A $DI.A F #  tCH~ :9x rrhN)r5typingrrrgoogle.cloud.aiplatformrvertexai.evaluationrLoggerr2r>rr6rPromptTemplaterr9rhr rrrvs|"I''( $++h  WhtCH~&>WO::>] $9] @f #8f r