ci dZddlZddlmZmZmZmZmZddlm Z ddl m Z erddlZej GddZej Gdd Zy) zBase classes for evaluation.N)DictListOptionalUnion TYPE_CHECKING)evaluation_service)_baseceZdZUdZded<eeeejfed<e eefed<e jed<e ed<e ed<d ed d fd Zy ) EvaluationRunConfigaEvaluation Run Configurations. Attributes: dataset: The dataset to evaluate. metrics: The list of metric names, or Metric instances to evaluate. metric_column_mapping: An optional dictionary column mapping that overrides the metric prompt template input variable names with mapped the evaluation dataset column names, used during evaluation. For example, if the input_variables of the metric prompt template are ["context", "reference"], the metric_column_mapping can be { "context": "news_context", "reference": "ground_truth", "response": "model_1_response" } if the dataset has columns "news_context", "ground_truth" and "model_1_response". client: The evaluation service client. evaluation_service_qps: The custom QPS limit for the evaluation service. retry_timeout: How long to keep retrying the evaluation requests, in seconds. pd.DataFramedatasetmetricsmetric_column_mappingclientevaluation_service_qps retry_timeout column_namereturnNc|jj|||jjvrIt d|jj||dt |jjdy)zValidates that the column names in the column map are in the dataset. Args: column_name: The column name to validate. Raises: KeyError: If any of the column names are not in the dataset. zRequired column `zQ` not found in the evaluation dataset. The columns in the evaluation dataset are .N)rgetr columnsKeyErrorlist)selfrs @/tmp/pip-target-z3e9_cxr/lib/python/vertexai/evaluation/_base.pyvalidate_dataset_columnz+EvaluationRunConfig.validate_dataset_columnAs  & & * *; D<<'' (//33KMNO++/ 0D0D+E*FaI  ()__name__ __module__ __qualname____doc____annotations__rrstr metrics_base_Metricrgapic_evaluation_servicesEvaluationServiceClientfloatrrrr r "sc, %\1112 33S>) % = ==!!34rr cXeZdZUdZeeefed<dZe ded<dZ e eeefed<y) EvalResultaIEvaluation result. Attributes: summary_metrics: A dictionary of summary evaluation metrics for an evaluation run. metrics_table: A pandas.DataFrame table containing evaluation dataset inputs, predictions, explanations, and metric results per row. metadata: The metadata for the evaluation run. summary_metricsNr metrics_tablemetadata) rr r!r"rr$r)r#r.rr/r*rrr,r,Vs>#u*%%.2M8N+2)-HhtCH~&-rr,)r" dataclassestypingrrrrr#google.cloud.aiplatform_v1.servicesrr'vertexai.evaluation.metricsr r%pandaspd dataclassr r,r*rrr7sf"#== 000f  . . .r