ipdZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZmZerdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZmZm Z ddl!m"Z"ddl#m$Z$ddlm%Z%ddl&m'Z'e%jPe)Z*dZ+dZ,dZ-Gddej\Z/Gddej\Z0Gddej\Z1Gddej\Z2Gddej\Z3Gd d!ej\Z4Gd"d#eZ5Gd$d%ej\Z6Gd&d'ej\Z7Gd(d)ej\Z8Gd*d+ej\Z9Gd,d-ej\Z:Gd.d/ej\Z;Gd0d1ej\ZeGd8d9e=Z?ed:5Gd;de=ZAed?5Gd@dAe=ZBedB5GdCdDe=ZCeGdEdFe=ZDeGdGdHe=ZEeGdIdJe=ZFgdKZGy)LzPyTorch FNet model.) dataclass)partialN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)auto_docstringis_scipy_available)linalg)ACT2FN)GradientCheckpointingLayer) BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)logging) FNetConfigc|jd}|d|d|f}|jtj}tjd|||S)z4Applies 2D matrix multiplication to 3D input arrays.rNzbij,jk,ni->bnk)shapetypetorch complex64einsum)xmatrix_dim_onematrix_dim_two seq_lengths b/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/fnet/modeling_fnet.py_two_dim_matmulr)5sNJ#KZK*$<=N uA <<(!^^ LLct|||SN)r))r$r%r&s r(two_dim_matmulr->s 1nn ==r*c|}tt|jddD]#}tjj ||}%|S)z Applies n-dimensional Fast Fourier Transform (FFT) to input array. Args: x: Input n-dimensional array. Returns: n-dimensional Fourier transform of input n-dimensional array. rN)axis)reversedrangendimr!fft)r$outr/s r(fftnr5CsG Cqvvqr*+,iimmCdm+, Jr*c*eZdZdZfdZddZxZS)FNetEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cxt|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j |j |_tj |j"|_|j'dt)j*|jj-dd|j'dt)j.|j0j3t(j4dy) N) padding_idxeps position_idsrF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsLinear projectionDropouthidden_dropout_probdropoutregister_bufferr!arangeexpandzerosr<sizelongselfconfig __class__s r(rDzFNetEmbeddings.__init__Vs: !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>ST))F$6$68J8JKzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r*cX||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} || z} |j|} | | z } |j| } |j| } |j| } | S)Nr>rr@rrBdevice)rYr<hasattrr@rWr!rXrZrarIrMrKrNrQrT) r\ input_idsr@r< inputs_embeds input_shaper'buffered_token_type_ids buffered_token_type_ids_expandedrM embeddingsrKs r(forwardzFNetEmbeddings.forwardjs=  #..*K',,.s3K ^  ,,Q  ^ J"%:: "66|D)) ^^J/ __Z0 \\*- r*)NNNN)__name__ __module__ __qualname____doc__rDri __classcell__r^s@r(r7r7SsQ (!r*r7c*eZdZfdZdZdZxZS)FNetBasicFourierTransformcDt||j|yr,)rCrD_init_fourier_transformr[s r(rDz"FNetBasicFourierTransform.__init__s  $$V,r*c|js+ttjjd|_y|j dkrtr|jdtjtj|jtj|jdtjtj|jtjtt|j |j"|_yt%j&dt|_yt|_y) N)rdimdft_mat_hiddenrA dft_mat_seq)r%r&zpSciPy is needed for DFT matrix calculation and is not found. Using TPU optimized fast fourier transform instead.)use_tpu_fourier_optimizationsrr!r3r5fourier_transformrJr rUtensorr dftrGr"tpu_short_seq_lengthr-rzryrwarning)r\r]s r(rsz1FNetBasicFourierTransform._init_fourier_transforms33%,UYY^^%HD "  + +t 3!#$$$ell6::f>P>P3QY^YhYh&i$$!5<< 6;V;V0W_d_n_n#o*1"43C3CTXTgTg*&**.&%)D "r*c>|j|j}|fSr,)r|real)r\ hidden_statesoutputss r(riz!FNetBasicFourierTransform.forwards" ((7<<zr*)rjrkrlrDrsrirnros@r(rqrqs-*.r*rqc$eZdZfdZdZxZS)FNetBasicOutputct|tj|j|j |_yNr:)rCrDrrNrGrOr[s r(rDzFNetBasicOutput.__init__s, f&8&8f>S>STr*c.|j||z}|Sr,)rNr\r input_tensors r(rizFNetBasicOutput.forwards|m'CD r*rjrkrlrDrirnros@r(rrsUr*rc$eZdZfdZdZxZS)FNetFourierTransformcbt|t||_t ||_yr,)rCrDrqr\routputr[s r(rDzFNetFourierTransform.__init__s& -f5 %f- r*cX|j|}|j|d|}|f}|SNr)r\r)r\r self_outputsfourier_outputrs r(rizFNetFourierTransform.forwards1yy/ \!_mD!#r*rros@r(rrs . r*rcVeZdZfdZdej dej fdZxZS)FNetIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr,) rCrDrrPrGintermediate_sizedense isinstance hidden_actstrrintermediate_act_fnr[s r(rDzFNetIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r*rreturncJ|j|}|j|}|Sr,)rrr\rs r(rizFNetIntermediate.forwards& =1 00? r*rjrkrlrDr!Tensorrirnros@r(rrs#9U\\ellr*rcneZdZfdZdej dej dej fdZxZS) FNetOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rCrDrrPrrGrrNrOrRrSrTr[s r(rDzFNetOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r*rrrcr|j|}|j|}|j||z}|Sr,)rrTrNrs r(rizFNetOutput.forwards7 =1  ]3 }|'CD r*rros@r(rrs1> U\\RWR^R^r*rc*eZdZfdZdZdZxZS) FNetLayerct||j|_d|_t ||_t ||_t||_ yNr) rCrDchunk_size_feed_forward seq_len_dimrfourierr intermediaterrr[s r(rDzFNetLayer.__init__sI '-'E'E$+F3 ,V4 ( r*c|j|}|d}t|j|j|j|}|f}|Sr)rrfeed_forward_chunkrr)r\rself_fourier_outputsr layer_outputrs r(rizFNetLayer.forwardsO#||M:-a00  # #T%A%A4CSCSUc  /r*cL|j|}|j||}|Sr,)rr)r\rintermediate_outputrs r(rzFNetLayer.feed_forward_chunks*"//?{{#6G r*)rjrkrlrDrirrnros@r(rrs) r*rc&eZdZfdZddZxZS) FNetEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rCrDr]r ModuleListr1num_hidden_layersrlayergradient_checkpointing)r\r]_r^s r(rDzFNetEncoder.__init__sN  ]]uVE]E]?^#_!If$5#_` &+#$`sA#c|rdnd}t|jD]\}}|r||fz}||}|d}|r||fz}|std||fDSt||S)Nrc3&K|] }|| ywr,r).0vs r( z&FNetEncoder.forward..sXq!-Xs)last_hidden_stater) enumeratertupler)r\routput_hidden_states return_dictall_hidden_statesi layer_module layer_outputss r(rizFNetEncoder.forward s"6BD(4 -OA|#$58H$H!(7M)!,M  - 1]4D D X]4E$FXX XN_``r*)FTrros@r(rrs, ar*rcVeZdZfdZdej dej fdZxZS) FNetPoolerct|tj|j|j|_tj |_yr,)rCrDrrPrGrTanh activationr[s r(rDzFNetPooler.__init__"s9 YYv1163E3EF '')r*rrc\|dddf}|j|}|j|}|Sr)rr)r\rfirst_token_tensor pooled_outputs r(rizFNetPooler.forward's6+1a40 #56  6 r*rros@r(rr!s#$ U\\ellr*rcVeZdZfdZdej dej fdZxZS)FNetPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) rCrDrrPrGrrrrrtransform_act_fnrNrOr[s r(rDz$FNetPredictionHeadTransform.__init__2s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr*rrcl|j|}|j|}|j|}|Sr,)rrrNrs r(riz#FNetPredictionHeadTransform.forward;s4 =1 --m< }5 r*rros@r(rr1s$UU\\ellr*rc$eZdZfdZdZxZS)FNetLMPredictionHeadct|t||_t j |j |j|_t jtj|j|_ yr,) rCrDr transformrrPrGrFdecoder Parameterr!rXbiasr[s r(rDzFNetLMPredictionHead.__init__CsW 4V<yy!3!3V5F5FG LLV->->!?@ r*cJ|j|}|j|}|Sr,)rrrs r(rizFNetLMPredictionHead.forwardIs$}5  ]3 r*rros@r(rrBsA r*rc$eZdZfdZdZxZS)FNetOnlyMLMHeadcBt|t||_yr,)rCrDr predictionsr[s r(rDzFNetOnlyMLMHead.__init__Ps /7r*c(|j|}|Sr,)r)r\sequence_outputprediction_scoress r(rizFNetOnlyMLMHead.forwardTs ,,_=  r*rros@r(rrOs 8!r*rc$eZdZfdZdZxZS)FNetOnlyNSPHeadclt|tj|jd|_yNru)rCrDrrPrGseq_relationshipr[s r(rDzFNetOnlyNSPHead.__init__[s'  " &*<*r=) rC _init_weightsrr7initcopy_r<r!rVrrWzeros_r@)r\moduler^s r(rz!FNetPreTrainedModel._init_weightswsl f% fn - JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- . .r*) rjrkrlr__annotations__base_model_prefixsupports_gradient_checkpointingrrnros@r(rrqs &*#//r*rz0 Output type of [`FNetForPreTraining`]. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<y)FNetForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsr) rjrkrlrmrr! FloatTensorrrrrrrr*r(rr~sd &*D%  d ")26u((4/68<U..5<59M5**+d29r*rceZdZdZdfd ZdZdZe ddejdzdejdzdejdzd ejdzd e dzd e dzd e e zfd ZxZS) FNetModelz The model can behave as an encoder, following the architecture described in [FNet: Mixing Tokens with Fourier Transforms](https://huggingface.co/papers/2105.03824) by James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon. ct||||_t||_t ||_|r t|nd|_|jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rCrDr]r7rhrencoderrpooler post_init)r\r]add_pooling_layerr^s r(rDzFNetModel.__init__sM   (0"6* ,=j(4  r*c.|jjSr,rhrIr\s r(get_input_embeddingszFNetModel.get_input_embeddingss...r*c&||j_yr,r)r\values r(set_input_embeddingszFNetModel.set_input_embeddingss*/'r*Nrcr@r<rdrrrc |||n|jj}||n|jj}| | td||j }|\} } n&||j dd}|\} } n td|jj r)| dkr$|jj | k7r td| |jn |j} |pt|jdr4|jjddd| f} | j| | } | }n&tj|tj| }|j||||}|j||| }|d }|j |j!|nd}|s ||f|d dzSt#|||j$ S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer>z5You have to specify either input_ids or inputs_embedsrxzThe `tpu_short_seq_length` in FNetConfig should be set equal to the sequence length being passed to the model when using TPU optimizations.r@r`)rcr<r@rd)rrrr)r pooler_outputr)r]ruse_return_dict ValueErrorrYr{rrarbrhr@rWr!rXrZr r rr)r\rcr@r<rdrrkwargsre batch_sizer'rarfrgembedding_outputencoder_outputsrrs r(rizFNetModel.forwards%9$D $++JjJj &1%cd d  "#..*K%0 "J  &',,.s3K%0 "J TU U KK 5 5d" 00J>;  &/%:!! @T@T  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z??%)' +  ,, !5#'  *!,8< 8O O4UY #]3oab6II I)-')77  r*)T)NNNNNN)rjrkrlrmrDrrr r! LongTensorrboolrrrirnros@r(rrs /0.2260426,0#'D ##d*D ((4/D &&- D ((4/ D #Tk D D[D   D D r*rz FNet Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c*eZdZdddZfdZdZdZe ddejdzd ejdzd ejdzd ejdzd ejdzd ejdzde dzde dzde e zfdZ xZS)FNetForPreTrainingcls.predictions.bias&fnet.embeddings.word_embeddings.weightzcls.predictions.decoder.biaszcls.predictions.decoder.weightct||t||_t ||_|j yr,)rCrDrrrclsr r[s r(rDzFNetForPreTraining.__init__ s4  f% '/ r*cB|jjjSr,r$rrrs r(get_output_embeddingsz(FNetForPreTraining.get_output_embeddingsxx##+++r*c||jj_|j|jj_yr,r$rrrr\new_embeddingss r(set_output_embeddingsz(FNetForPreTraining.set_output_embeddings,'5$$2$7$7!r*Nrcr@r<rdlabelsnext_sentence_labelrrrc  ||n|jj}|j||||||} | dd\} } |j| | \} }d}|u|st }|| j d|jj |j d}||j dd|j d}||z}|s| |f| ddz}||f|zS|St|| || jS)aH labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForPreTraining.from_pretrained("google/fnet-base") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Nr@r<rdrrrur>)rrrr) r]rrr$rviewrFrr)r\rcr@r<rdr/r0rrrrrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r(rizFNetForPreTraining.forwards5L&1%rurlogitsr) r]rrr$rr3rFrr)r\rcr@r<rdr/rrrrrrr6r5rs r(rizFNetForMaskedLM.forwardws$&1%:K[b[p[pqqr*NNNNNNN)rjrkrlr9rDr'r-r r!rrrrrirnros@r(r;r;`s)?*R ,8*..2,0-1&*,0#'(r<<$&(r t+(rllT) (r ||d* (r  t# (r#Tk(rD[(r  (r(rr*r;zT FNet Model with a `next sentence prediction (classification)` head on top. ceZdZfdZe d dej dzdej dzdej dzdej dzdej dzdedzd edzd ee zfd Z xZ S) FNetForNextSentencePredictionct||t||_t ||_|j yr,)rCrDrrrr$r r[s r(rDz&FNetForNextSentencePrediction.__init__r=r*Nrcr@r<rdr/rrrc V||n|jj}|j||||||} | d} |j| } d} |2t } | | j dd|j d} |s| f| ddz}| | f|zS|St | | | jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForNextSentencePrediction.from_pretrained("google/fnet-base") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ```Nr2rr>rurA)r]rrr$rr3rr)r\rcr@r<rdr/rrrrrseq_relationship_scoresr7r5rs r(riz%FNetForNextSentencePrediction.forwardsH&1% 1` a classification loss is computed (Cross-Entropy). Nr2r regressionsingle_label_classificationmulti_label_classificationr>rurA)r]rrrTrN problem_typerMrBr!rZintrsqueezerr3rrr)r\rcr@r<rdr/rrrrrrBrr5rs r(riz%FNetForSequenceClassification.forwards$&1%r2rurA) r]rrr3rYrrTrNrrr)r\rcr@r<rdr/rrr num_choicesrrrBreshaped_logitsrr5rs r(rizFNetForMultipleChoice.forwardQsT&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  )) )%'!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E(d?ZaZoZoppr*rC) rjrkrlrDr r!rrrrrirnros@r(rWrWEs*..2,0-1&*,0#'Mq<<$&Mq t+MqllT) Mq ||d* Mq  t# Mq#TkMqD[Mq * *MqMqr*rWceZdZfdZe d dej dzdej dzdej dzdej dzdej dzdedzd edzd ee zfd Z xZ S) FNetForTokenClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr,rLr[s r(rDz#FNetForTokenClassification.__init__si   ++f% zz&"<"<= ))F$6$68I8IJ r*Nrcr@r<rdr/rrrc ||n|jj}|j||||||} | d} |j| } |j | } d} |rurA) r]rrrTrNrr3rMrr)r\rcr@r<rdr/rrrrrrBrr5rs r(riz"FNetForTokenClassification.forwards &1%rv) ignore_indexru)r start_logits end_logitsr) r]rrrdsplitrU contiguouslenrYclamprrr)r\rcr@r<rdrerfrrrrrrBrirjr4 ignored_indexr5 start_lossend_lossrs r(riz FNetForQuestionAnswering.forwards&1%rs! AA&7!9   .6*   H %M>  8RYY8v# #Lbii 299  ryy   *6a"))a8   "))" 299 !bii!&bii& 9299 9 // / /  :{:  :$c #c c L Z ,Z  Z z?r)?r?rD J $7J  J Z Gg$7Gg  GgTYq/YqYqx7d!47d7dtB 2B B J r*