ik.dZddlmZddlmZddlZddlmZmZddlm Z ddl m Z dd l mZdd lmZdd lmZmZdd lmZdd lmZddlmZmZmZmZmZmZmZddl m!Z!m"Z"m#Z#erddl$m%Z%ejLe'Z(dej dej fdZ)dej dej fdZ*eeGddeZ+dedefdZ,dedefdZ-dZ.dZ/eed Gd!d"eZ0eed# Gd$d%eZ1Gd&d'ejdZ3Gd(d)ejdZ4Gd*d+ejdZ5Gd,d-ejdZ6Gd.d/eZ7eGd0d1eZ8Gd2d3ejdZ9Gd4d5e8Z:Gd6d7e8Z;Gd8d9e8Z<Gd:d;e8Z=eGd<d=e8Z>Gd>d?ejdZ?Gd@dAejdZ@GdBdCe8ZAgdDZBy)EzPyTorch OWL-ViT model.) dataclass)AnyN)Tensornn)initialization)ACT2FN)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleis_vision_availablelogging torch_int) OwlViTConfigOwlViTTextConfigOwlViTVisionConfig)center_to_corners_formatlogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr )rs f/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/owlvit/modeling_owlvit.pycontrastive_lossr'4s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r't)r) caption_loss image_losss r& owlvit_lossr.9s,#J/L!*,,.1J : % ,,r(ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ e ed<dZe ed <d eefd Zy) OwlViTOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`OwlViTTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`OwlViTVisionModel`]. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)r6r7Ngetattrto_tuple.0kselfs r& z(OwlViTOutput.to_tuple.._=  LLDGRYZ^`aRbRkRkRm m -0tuplekeysrAs`r&r=zOwlViTOutput.to_tuple^# YY[   r()__name__ __module__ __qualname____doc__r1r# FloatTensor__annotations__r2r3r4r5r6r r7rFrr=r(r&r0r0?s(&*D%  d ")15e''$.504OU&&-4,0K""T)0-1L%##d*148186:3: %* r(r0r+c|jr>|jtjtjfvr|S|j S|jtj tjfvr|S|jSN) is_floating_pointdtyper#float32float64floatint32int64int)r+s r&_upcastr[fs`GG u}}==qL1779LGG U[[99qFquuwFr(boxescft|}|dddf|dddfz |dddf|dddfz zS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrrr)r[)r\s r&box_arear_osB ENE !Q$K%1+ %%1+ad *C DDr(c^t|}t|}tj|dddddf|ddddf}tj|dddddf|ddddf}||z j d}|dddddf|dddddfz}|dddf|z|z }||z } | |fS)Nr^rminr)r_r#maxrbclamp) boxes1boxes2area1area2left_top right_bottom width_heightinterunionious r&box_iouros V E V Eyy4! ,fQUms r&rBz7OwlViTObjectDetectionOutput.to_tuple..rCrDrErHs`r&r=z$OwlViTObjectDetectionOutput.to_tuplerIr()rJrKrLrMr1r#rNrOrzdictrr{r4r5r|r6r r7rFrr=rPr(r&ryrys8&*D%  d ")!Itd{!'+FE   $++/J!!D(/,0K""T)0-1L%##d*1-1L%##d*148186:3: %* r(ryzM Output type of [`OwlViTForObjectDetection.image_guided_detection`]. c0eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ eed <dZeed <d eefd Zy) &OwlViTImageGuidedObjectDetectionOutputa logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWL-ViT represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nrr5query_image_embedstarget_pred_boxesquery_pred_boxesr|r6r7rcHtfdjDS)Nc3dK|]'}|dvr|nt|j)ywr:r;r>s r&rBzBOwlViTImageGuidedObjectDetectionOutput.to_tuple.. rCrDrErHs`r&r=z/OwlViTImageGuidedObjectDetectionOutput.to_tuple rIr()rJrKrLrMrr#rNrOr5rrrr|r6r r7rFrr=rPr(r&rrs8(,FE   $+-1L%##d*137))D0726u((4/615e''$.5-1L%##d*148186:3: %* r(rceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) OwlViTVisionEmbeddingsconfigct||j|_||_|j|_t jtj|j|_ t j|j|j |j|jd|_ |j|jzdz|_|jdz|_t j"|j |j |_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasr^r position_idsr persistent)super__init__ patch_sizer hidden_size embed_dimr Parameterr#randnclass_embeddingConv2d num_channelspatch_embedding image_size num_patches num_positions Embeddingposition_embeddingregister_bufferr$expandrAr __class__s r&rzOwlViTVisionEmbeddings.__init__s  ++ ++!||EKK8J8J,KL!yy++))$$  #--1B1BBqH!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr( embeddingsheightwidthrc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrg?rr^bicubicF)sizemode align_cornersdim)shaperweight unsqueezer#jit is_tracingrrrreshapepermuterr! interpolateviewcat) rArrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r&interpolate_pos_encodingz/OwlViTVisionEmbeddings.interpolate_pos_encoding&sv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4  *11!Q1=BB1b#Nyy/?;CCr( pixel_valuesrch|j\}}}}|j|}|jdjdd}|jj |dd}t j||gd} |r| |j| ||z} | S| |j|jz} | S)Nr^rrr) rrflatten transposerrr#rrrr) rArr batch_size_rr patch_embedsr|rs r&forwardzOwlViTVisionEmbeddings.forwardLs'3'9'9$ Avu++L9 #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr(F) rJrKrLrrr#rrZrrNboolr __classcell__rs@r&rrsmq1q*$D5<<$D$DUX$D]b]i]i$DL E$5$5 QU bgbnbn r(rc eZdZdeffd Z d dej dzdej dzdejdzdejfdZ xZ S) OwlViTTextEmbeddingsrc^t|tj|j|j |_tj|j|j |_|jdtj|jjddy)NrrFr) rrrr vocab_sizertoken_embeddingmax_position_embeddingsrrr#r$rrs r&rzOwlViTTextEmbeddings.__init__Zs !||F,=,=v?Q?QR"$,,v/M/MvOaOa"b  ELL)G)GHOOPWXej  r(N input_idsr inputs_embedsrc||jdn|jd}||jddd|f}||j|}|j|}||z}|S)Nr)rrrr)rArrr seq_lengthposition_embeddingsrs r&rzOwlViTTextEmbeddings.forwardds{ -6,AY__R(}GZGZ[]G^  ,,Q  ^> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r(tensorseq_lenbszc|j|||j|jjddj S)Nrr^)rrrr contiguous)rArrrs r&_shapezOwlViTAttention._shapes7{{3GQQRSUVWbbddr(N hidden_statesattention_maskoutput_attentionsrc |j\}}}|j||jz}|j|j |d|} |j|j |d|} ||j zd|jf} |j|||j| }| j| } | j| } | jd} tj|| jdd} | j||j z|| fk7r/td||j z|| fd| j|{|j|d|| fk7r#td|d|| fd|j| j||j || |z} | j||j z|| } tjj| d} |r?| j||j || }|j||j z|| } nd}tjj!| |j |j" }|j%| j&}tj|| }|j||j z||jfk7r7td ||j ||jfd|j|j||j ||j}|jdd}|j)|||}|j+|}||fS) z#Input shape: Batch x Time x Channelrrr^z$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )rrrrrrrrrr#bmmrrrrr!softmaxrrtorTrr)rArrrkwargsrtgt_lenr query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r&rzOwlViTAttention.forwardsT#0"4"4"6Wi{{=1DJJ> [[]!;RE {{4;;}#=r3G DNN*B > Ct{{<#>CCZP $Z__j1 (|((*5 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<j|j@dz|zt j |jBj|jDdz|zt jF|jH|jjJnTt|tLrDt j|jN|jQ|jR|jTt|tVjXr>t jZ|j\t j^|jt|tVj`rOt j |jd||j\ t jZ|j\y y y ) zInitialize the weightsg{Gz?)meanstdrrr)r'r^N)1rinitializer_factor isinstancerinitnormal_rrrcopy_rr#r$rrrrrrinitializer_rangernum_hidden_layersrrrrrrr r  OwlViTModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dim constant_ logit_scalelogit_scale_init_valueOwlViTForObjectDetectionbox_biascompute_box_biasnum_patches_heightnum_patches_widthrrzeros_rones_r)rAr#factor in_proj_std out_proj_stdfc_stds r& _init_weightsz#OwlViTPreTrainedModel._init_weights$s// f2 3 LL//66Sftm T LL2299RV W JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  6 7 LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d