0idt dZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZmZd d lmZej2eZGddej8Zej<j>dZ dZ!Gddej8Z"d0dejFde$de%dejFfdZ&Gddej8Z'Gddej8Z(Gddej8Z)Gdd ej8Z*d!Z+d"Z,Gd#d$eZ-Gd%d&ej8Z.eGd'd(eZ/eGd)d*e/Z0ed+,Gd-d.e e/Z1gd/Z2y)1zPyTorch ViTDet backbone.N)nn)initialization)ACT2FN) BackboneMixin)GradientCheckpointingLayer)BackboneOutputBaseModelOutput)PreTrainedModel)auto_docstringlogging) VitDetConfigc`eZdZdZfdZdZdejdejfdZxZ S)VitDetEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) to be consumed by a Transformer. cpt||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_ ||_||_||_ |jr?|dz}tjtj d||j |_nd|_tj$|||||_y)Nrr) kernel_sizestride)super__init__pretrain_image_size patch_size num_channels hidden_size isinstance collectionsabcIterable image_size num_patches use_absolute_position_embeddingsr Parametertorchzerosposition_embeddingsConv2d projection) selfconfigrrrrr num_positions __class__s f/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/vitdet/modeling_vitdet.pyrzVitDetEmbeddings.__init__)s !'!;!;V=N=NJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&  2 2'!OM')||EKK=RXRdRd4e'fD $'+D $))L+:^hic|r |ddddf}|jd}ttj|}||z|k7r t dt j js ||k7s||k7r]tjj|jd||djdddd||fdd }|jddddS|jd||dS) a Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token dimension for the original embeddings. Args: abs_pos_embeddings (`torch.Tensor`): Absolute positional embeddings with (1, num_position, num_channels). has_cls_token (`bool`): If true, has 1 embedding in abs_pos_embeddings for cls token. height (`int`): Height of input image tokens. width (`int`): Width of input image tokens. Returns: Absolute positional embeddings after processing with shape (1, height, width, num_channels) Nrz5Absolute position embeddings must be a square number.rrbicubicF)sizemode align_corners) shapeintmathsqrt ValueErrorr#jit is_tracingr functional interpolatereshapepermute)r(abs_pos_embeddings has_cls_tokenheightwidth num_positionr2new_abs_pos_embeddingss r,get_absolute_positionsz'VitDetEmbeddings.get_absolute_positions?s$ !3AqrE!: )//2 499\*+ $;, &TU U 99   !dfn %']]%>%>"**1dD"=EEaAqQe_# &?& "*11!Q1= =%--aC Cr- pixel_valuesreturncz|jd}||jk7rtd|jd|d|j|}|jc|j dddd}||j |jd|jd|jdz}|j dddd}|S) NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rr0rT)r5rr9r'r%r?rF)r(rGr embeddingss r,forwardzVitDetEmbeddings.forwardes#))!, 4,, ,!../yaI __\2  # # /#++Aq!Q7J#d&A&A(($ 0@0@0CZEUEUVWEX'J$++Aq!Q7Jr-) __name__ __module__ __qualname____doc__rrFr#TensorrL __classcell__r+s@r,rr#s0 j,$DLELLU\\r-rcTtdt||zdz }|jd|k7rttjj |j d|jddjddd|d}|j d|jdd}n|}tj|dddft||z dz}tj|dddft||z dz}||z |dz t||z dzz}||jS) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (`int`): Size of query q. k_size (`int`): Size of key k. rel_pos (`torch.Tensor`): Relative position embeddings (num_embeddings, num_channels). Returns: Extracted positional embeddings according to relative positions. r0rrr/linear)r2r3Ng?) r6maxr5rr<r=r>r?r#arangelong)q_sizek_sizerel_pos max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordss r, get_rel_posra{s% q3vv..23L}}Q<'--33 OOAw}}Q/ 4 < bhwkzbhwc,wkc->bhwkN)rar5r>r#einsumview)attnqueries rel_pos_h rel_pos_wrYrZqueries_height queries_width keys_height keys_widthrelative_heightrelative_width batch_size_dimr_qrelative_weights r,!add_decomposed_relative_positionsrts,%+!NM$K!.+yIO  IFN J3 //*nmS ICll#3S/JOll#3S.IO *nm[*U !Q1d* + , !Q4* + , d:~ 5{Z7OP  Kr-c,eZdZdZdfd ZddZxZS)VitDetAttentionz=Multi-head Attention block with relative position embeddings.c t||j}|j}||_||z}|dz|_t j||dz|j|_ t j|||_ |j|_ |jrot jtjd|dzdz ||_t jtjd|dzdz ||_yy)z Args: config (`VitDetConfig`): Model configuration. input_size (`tuple[int]`, *optional*): Input resolution, only required in case relative position embeddings are added. grbiasr0rrN)rrrnum_attention_heads num_headsscalerLinearqkv_biasqkvproj use_relative_position_embeddingsr"r#r$rgrh)r(r) input_sizerqr{head_dimr+s r,rzVitDetAttention.__init__s   .. ")#t^ 99S#'@IIc3' 060W0W-  0 0\\%++a*Q-6G!6KX*VWDN\\%++a*Q-6G!6KX*VWDN 1r-c 0|j\}}}}|j|j|||zd|jdj ddddd}|jd||jz||zdj d\}} } ||j z| jddz} |jr(t| ||j|j||f||f} | jd} | | z}|j||j||d}|j ddddd}|j|||d}|j|}|r>| j||j| jd| jd} || f} | S|f} | S) Nrr/r0rr)rq)r5rr>r{r?unbindr| transposerrtrgrhsoftmaxrdr)r( hidden_stateoutput_attentionsrorBrCrprrfkeysvaluesattention_scoresattention_probsoutputss r,rLzVitDetAttention.forwards'3'9'9$ FE1hh|$,,Z%DNN\^_gghiklnoqrtuv # AzDNN/JFUZN\^ _ f fgh iv#djj0DNN2r4JJ  0 0@ '4>>4>>FTY?]cej\k  +22r2:&/ #((T^^VUTVW #++Aq!Q: #++JrJ yy. -55DNNO,A,A",EG\G\]_G`O$_5G$oGr-NFrMrNrOrPrrLrRrSs@r,rvrvsGX4r-rvinput drop_probtrainingrHc|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)zc Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). rr)r)dtypedevice)r5ndimr#randrrfloor_div)rrr keep_probr5 random_tensoroutputs r, drop_pathrs Cx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr-cxeZdZdZd dedzddffd ZdejdejfdZde fdZ xZ S) VitDetDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrHc0t|||_yr)rrr)r(rr+s r,rzVitDetDropPath.__init__s "r- hidden_statescDt||j|jSr)rrr)r(rs r,rLzVitDetDropPath.forwards FFr-c d|jS)Nzp=)rr(s r, extra_reprzVitDetDropPath.extra_repr"sDNN#$$r-r) rMrNrOrPfloatrr#rQrLstrrrRrSs@r,rrsGb#%$,#$#GU\\GellG%C%r-rc*eZdZdZdfd ZdZxZS)VitDetLayerNormaL A LayerNorm variant, popularized by Transformers, that performs point-wise mean and variance normalization over the channel dimension for inputs that have shape (batch_size, channels, height, width). https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ct|tjt j ||_tjt j||_||_ |f|_ yr) rrrr"r#onesweightr$ryepsnormalized_shape)r(rrr+s r,rzVitDetLayerNorm.__init__-sT ll5::.>#?@ LL-=!>? !1 3r-c|jdd}||z jdjdd}||z tj||jzz }|j ddddf|z|j ddddfz}|S)NrT)keepdimr0)meanpowr#r8rrry)r(xuss r,rLzVitDetLayerNorm.forward4s FF1dF # UKKN  4  0 UejjTXX. . KK4 & *TYYq$}-E Er-)gư>rrSs@r,rr&s 4r-rc(eZdZdZfdZdZxZS)VitDetResBottleneckBlockz The standard bottleneck residual block without the last activation layer. It contains 3 conv layers with kernels 1x1, 3x3, 1x1. ct|tj||dd|_t ||_t|j|_ tj||ddd|_ t ||_ t|j|_ tj||dd|_ t ||_y)ar Args: config (`VitDetConfig`): Model configuration. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. bottleneck_channels (`int`): Number of output channels for the 3x3 "bottleneck" conv layers. rFrxr)paddingryN)rrrr&conv1rnorm1r hidden_actact1conv2norm2act2conv3norm3)r(r) in_channels out_channelsbottleneck_channelsr+s r,rz!VitDetResBottleneckBlock.__init__Bs YY{,?O $%89 6,,- YY24GTU\ab $%89 6,,- YY2L!%P $\2 r-cN|}|jD] }||} ||z}|Sr)children)r(routlayers r,rLz VitDetResBottleneckBlock.forwardZs5]]_ E*C #g r-rrSs@r,rr<s 30r-rcdeZdZdededdffd Zdej dej fdZxZS) VitDetMlp in_featureshidden_featuresrHNct|tj|||_t |j |_tj|||_tj|j|_ yr) rrrr}fc1rractfc2Dropout dropout_probdrop)r(r)rrr+s r,rzVitDetMlp.__init__dsZ 99[/:&++,99_k:JJv223 r-rc|j|}|j|}|j|}|j|}|j|}|Sr)rrrr)r(rs r,rLzVitDetMlp.forwardksH HHQK HHQK IIaL HHQK IIaLr-) rMrNrOr6rr#rQrLrRrSs@r,rrcs84C4#4$4%,,r-rc `|j\}}}}|||zz |z}|||zz |z}tjj|ddd|d|f}||z||z} }|j |||z|| |z||}|j ddddddj j d|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rrrr0rr/)r5rr<padrdr? contiguous) r window_sizerorBrCr pad_height pad_width padded_height padded_widthwindowss r,window_partitionrus/;.@.@+J| 44 CJu{22kAI==$$\Aq!Y:3VWL"(:"5uy7H}dN^N^._ +M+!% /"0" /q1 (,   a .}d>N>NP`cikpbqrM!4>>-#@@ %txx =@Y7Z([[ %--aAq9  " " MM-8M "W,r-)rrFr)rMrNrOrPrrr6boolrr#rQrrLrRrSs@r,rrsMqv!"!49!LO!im! !L#('||' ' u||U\\) *U5<<-@ @ 'r-rc feZdZdeddffd Z d dej dedededee zf d Z xZ S) VitDetEncoderr)rHNc t|||_|j}t j d|j |dDcgc]}|j}}g}t|D]I}|jt|||||jvr |jnd||jvKtj||_d|_ycc}w)Nrcpu)r)rrrF)rrr)num_hidden_layersr#linspaceritemrangeappendrwindow_block_indicesrresidual_block_indicesr ModuleListrgradient_checkpointing)r(r)depthrrlayersir+s r,rzVitDetEncoder.__init__s  ((-2NN1f>S>SUZch,ijq!&&(jju A MM#1!#4676;V;V6V 2 2\]'(F,I,I'I   ]]6* &+#ksCrroutput_hidden_states return_dictc|rdnd}|rdnd}t|jD]'\}}|r||fz}|||} | d}|s|| dfz})|r||fz}|std|||fDSt|||S)Nrrc3&K|] }|| ywrr).0vs r, z(VitDetEncoder.forward..4smq_`_lmslast_hidden_stater attentions) enumeraterrr ) r(rrr r all_hidden_statesall_self_attentionsr  layer_module layer_outputss r,rLzVitDetEncoder.forwards#7BD$5b4(4 POA|#$58H$H!(8IJM)!,M &9]1=M>> from transformers import VitDetConfig, VitDetModel >>> import torch >>> config = VitDetConfig() >>> model = VitDetModel(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 768, 14, 14] ```Nz You have to specify pixel_values)rr r rrr) r)rr use_return_dictr9rKr8r rr) r(rGrr r kwargsembedding_outputencoder_outputssequence_outputs r,rLzVitDetModel.forwardps82C1N-TXT_T_TqTq$8$D $++JjJj &1%v?W?WZ[?[9\]AV//] ^s A7rHc.|jjSrr;rs r,r<z#VitDetBackbone.get_input_embeddingsr=r-NrGr rr c ||n|jj}||n|jj}||n|jj}|j |}|j |d||}|r |j n|d}d} t|j|D]\} } | |jvs| | fz } |s|r | f|ddz} | S| f|ddz} | St| |r |j nd|jS)a Examples: ```python >>> from transformers import VitDetConfig, VitDetBackbone >>> import torch >>> config = VitDetConfig() >>> model = VitDetBackbone(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NT)r rr rrr0) feature_mapsrr) r)r?r rrKr8rzip stage_names out_featuresr r) r(rGr rr r@rArrrKstagerrs r,rLzVitDetBackbone.forwards58&1%r^s &!+9?-,.   H %UryyUp!3!3H&R;bii;~ U\\ e T V[VbVb  %RYY %bii,$ryy$N $2@>M,M`5 BII5 p!+O!+!+HD 'D D N K ]$9K  K \ Er-