Positional encoding

In deep learning, the positional encoding or position embedding is a component of sequence models, especially Transformers. Input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and the position of each token is encoded as a vector. Positional encoding is used because the standard attention mechanism used in Transformers by default treats all tokens symmetrically over all positions (i.e. permutation-symmetric), thus effectively treating the input as a bag of tokens, rather than a sequence of tokens.

Typology

Absolute vs relative positional encoding:

• In an absolute positional encoding scheme, the position of a token within a context is encoded. That is, it matters whether a token is at position 1 or position 100. This was proposed in 2017, just before Transformer was published.
• In a relative positional encoding scheme, the relative position of two tokens within a context is encoded. That is, it matters not whether a token is at position 1 or position 100, and it only matters whether one token is in front of another token by 1 or 100. This property is also called translational invariance. This was proposed in 2018, just after Transformer was published.

Learned vs designed positional encoding:

• In a learned positional encoding scheme, the parameters of positional encoding are learned like any other parameter in a model, usually by backpropagation.
• In a designed positional encoding scheme, the parameters are not learned, and picked by the researchers. That is, they are hyperparameters.

For example, the original Transformer used designed absolute positional encoding, and the GPT transformer used learned absolute positional encoding. BERT used two levels of positional embeddings, one for the per-token position, and one for per-segment position. Both are absolute and learned. It was reported that learned absolute positional encoding generally performs better than relative positional encoding when there are special tokens that always occur in a specific location, such as the [CLS] token of BERT-like architectures. However, with learned absolute positional encoding, a model cannot process contexts longer than what it is trained for.

Other possibly desirable properties of a positional encoding scheme include:

• Symmetry: In a symmetric positional encoding scheme, the positional encoding of a token $n$ tokens before is the same as that of a token $n$ tokens after. This property only makes sense for non-causal Transformers, since in a causal Transformer, a token cannot pay any attention to its subsequent tokens.
• Monotonicity: In a monotonic positional encoding scheme, the positional encoding increases attention weighting on nearby tokens compared to far-away tokens.

Sinusoidal

A standard transformer architecture, showing on the left an encoder, and on the right a decoder.

For the original transformer (Attention Is All You Need), published in 2017, each token is converted to a token embedding vector $x_{t}$, plus a positional encoding vector $p(t)$.

$x_{t}$ depends only on the identity of the token, not on $t$. That is, the token "the" would have the same $x_{t}$, no matter if it appears in the first or the last position. $p(t)$ depends only on the position of the token, not on the identity of the token. That is, the $t$-th token always has the same $p(t)$, no matter if it is "the" or "a".

Both the encoder and the decoder take $x_{t}+p(t)$ as input. This is shown in the picture.

Both $x_{t}$ and $p(t)$ have the same number of dimensions, since they must be added. Both of them have even dimensions, since that is what the positional encoding vector requires.

The original paper reported two methods of obtaining $p(t)$. One was the learned absolute positional encoding previously used in convolutional seq2seq. The other was the designed, sinusoidal absolute positional encoding.

Let the dimension be $2d$, then the position encoding vector is defined as$$p(t)=(\cos(\omega _{1}t),\sin(\omega _{1}t),\cos(\omega _{2}t),\sin(\omega _{2}t),\dots ,\cos(\omega _{d}t),\sin(\omega _{d}t))$$

Illustration of (absolute) positional encoding with parameters $d=50,b=10000$.

where $\omega _{k}=b^{-{\frac {k-1}{d}}}$.

Here, $b$ is the base frequency, a free parameter that should be significantly larger than the biggest $t$ that would be input into the positional encoding function (normally, this is the context length). The original paper uses $d=256,b=10000$.

More succinctly,$$p(t)=(e^{i\omega _{k}t})_{k\in 1:d},\quad \omega _{k}=b^{-{\frac {k-1}{d}}}$$In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.

Radix interpretation

The positional encoding has been interpreted as implementing a base-$b^{1/d}$ positional number system for representing real numbers:

Position	Frequency	Factor	Tokens per cycle ("wavelength")	Tokens per cycle at $d=50,b=10000$
Lowest	1	$e^{i\omega _{1}t}=e^{it}$	$2\pi$	6.3
Next lowest	$b^{-1/d}$	$e^{i\omega _{2}t}=e^{ib^{-1/d}t}$	$2\pi b^{1/d}$	7.5
Highest	$b^{-(d-1)/d}$	$e^{i\omega _{d}t}=e^{ib^{-(d-1)/d}t}$	$2\pi b^{(d-1)/d}$	52261

The effect of base-$b^{1/d}$ positional number system means the attention mechanism is to apportion its attention by coordinate. For low frequency (long wavelength) coordinates, the encoding implicitly states that these coordinates encode information that varies slowly over a sequence, and vice versa for high frequency (low wavelength) coordinates. Concretely, the first 2 coordinates are treated as encoding features that vary quickly every token, while the last 2 coordinates are treated as encoding features that vary slowly over 10000s of tokens.

Symmetry

This positional encoding method was designed based on previous work on convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position."

A relative position shift is just a linear transformation away:$$p(t+\Delta t)=\mathrm {diag} (p(\Delta t))p(t)$$where $\Delta t\in \mathbb {R}$ is the distance one wishes to shift. More generally, any convolution of relative positions is also just a linear transformation away:$$\sum _{j}c_{j}p(t+\Delta t_{j})=\left(\sum _{j}c_{j}\,\mathrm {diag} (p(\Delta t_{j}))\right)p(t),\quad \forall c_{j}\in \mathbb {R}$$for any constants $c_{j}$. This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors.

Relative position representations

In the original Transformer, positions are encoded by a vector $p(t)$ that is added to the token embedding vector $x_{t}$, and the Transformer only sees their sum $x_{t}+p(t)$. Several subsequent works produced variations of it. Compared to the original positional encoding scheme, they are relative, additive, and partially decouple the position encoding from the token embedding, by injecting positional encoding directly into each attention mechanism block, rather than adding only to the input.

Setup

The attention mechanism block in the original Transformer works as follows. It takes two input sequences $x_{1}^{Q},\dots ,x_{\ell _{\text{query}}}^{Q}$, and $x_{1}^{KV},\dots ,x_{\ell _{\text{key-value}}}^{KV}$, then computes three sequences$$q_{i}=x_{i}^{Q}W^{Q},k_{j}=x_{j}^{KV}W^{K},v_{j}=x_{j}^{KV}W^{V}$$then computes attention logits by taking a dot-product, multiplying by a positive scalar $\lambda$, then adding a mask:$$z_{ij}=\underbrace {\lambda } _{\text{scale}}q_{i}k_{j}^{T}+\underbrace {m_{ij}} _{\text{mask}}$$then computes attention weights by taking a softmax:$$a_{ij}={\frac {e^{z_{ij}}}{\sum _{j}e^{z_{ij}}}}$$then computes output vectors by taking a weighted sum of the value vectors:$$o_{i}=\sum _{j}a_{ij}v_{j}$$This is more succinctly written as $${\begin{aligned}O={\text{softmax}}\left(\lambda QK^{\mathrm {T} }+M\right)V\end{aligned}}$$The scalar $\lambda$ is usually selected to be $1/{\sqrt {d_{K}}}$ where $d_{K}$ is the dimension of the key vectors $k_{1},\dots ,k_{\ell _{\text{key-value}}}$.

Relative position representation

Relative Position Representation (RPR) was proposed in 2018, and two of the coauthors were coauthors of the original Transformer paper. In RPR, three choices were relative, added to the key and value vectors, and learned.

Concretely, the attention logits are calculated as $$z_{ij}=\lambda q_{i}(k_{j}+\underbrace {p^{K}(i-j)} _{\text{key PE}})^{T}+m_{ij}$$and the output vectors are calculated as$$o_{i}=\sum _{j}a_{ij}(v_{j}+\underbrace {p^{V}(i-j)} _{\text{value PE}})$$The key PE vectors and value PE vectors are learned. This is a learned dictionary of embedding vectors for relative positions, the same way as the embedding layer of a Transformer, which is a learned dictionary of embedding vectors for tokens.

The authors additionally clamped the vectors, i.e. for large enough $i-j$, they designed the identification $p^{K}(i-j)=p^{K}(i-j+1)=p^{K}(i-j+2)=\cdots$, and similarly for the value PE vectors, and for small enough $i-j$. This was designed so as to enforce an inductive bias that "relative position information is not useful beyond a certain distance". That is, the positional encoding scheme does not distinguish between "100 tokens apart" and "101 tokens apart". The authors also claimed that this allowed the model to generalize to sequence lengths longer than those seen during training.

deBERTa

The deBERTa transformer was a variant of BERT. It adds relative positional encoding to the query and key vectors.$$z_{ij}=\lambda \left(q_{i}k_{j}+q_{i}(\underbrace {p(i-j)W^{K}} _{\text{key PE}})^{T}+(\underbrace {p(j-i)W^{Q}} _{\text{query PE}})k_{j}^{T}\right)$$It does not include the 4th term $(p(j-i)W^{Q})(p(i-j)W^{K})^{T}$, since this term is a bias term depending purely on position, not token content.

It does not use the masking matrix since it is an all-to-all attention, like the original BERT.

XLNet

XLNet transformer adds relative positional encoding to the query and key vectors, and then identifies some of them. This was based on the previous Transformer-XL, which uses the same formula, except absolute positional encoding instead of relative positional encoding.

Begin with the expression$$z_{ij}=\lambda (q_{i}+\underbrace {p(i-j)W^{Q,PE}} _{\text{query PE}})(k_{j}+\underbrace {p(i-j)W^{K,PE}} _{\text{key PE}})^{T}+m_{ij}$$where $W^{Q,PE},W^{K,PE}$ play the same roles as $W^{Q},W^{K}$, except they map the position encoding vectors, rather than the token encoding vectors. Then expand the brackets to 4 terms, and identify some of the vectors, to obtain$$z_{ij}=\lambda \left(q_{i}k_{j}+q_{i}(p(i-j)W^{K,PE})^{T}+uk_{j}^{T}+v(p(i-j)W^{K,PE})^{T}\right)+m_{ij}$$

ALiBi

ALiBi (Attention with Linear Biases) adds a bias term to encode the relative position.$$z_{ij}=\lambda q_{i}k_{j}^{T}+m_{ij}+s\underbrace {b_{ij}} _{\text{bias}}$$Here, $s\in \mathbb {R}$ is a scalar, and $b_{ij}$ is an entry in the linear bias matrix defined by$$B={\begin{bmatrix}0&-1&-2&-3&\cdots \\-1&0&-1&-2&\cdots \\-2&-1&0&-1&\cdots \\-3&-2&-1&0&\cdots \\\vdots &\vdots &\vdots &\vdots &\ddots \\\end{bmatrix}}$$in other words, $b_{ij}=-|i-j|$. The idea being that the linear bias matrix is a softened mask. Just as $0$ represent full attention paid, and $-\infty$ represents no attention paid, the linear bias matrix linearly decreases the attention logit. Note that in the causally masked Transformer, any attention paid to the future is blanked out by the causal mask matrix$$M_{\text{causal}}={\begin{bmatrix}0&-\infty &-\infty &\dots &-\infty \\0&0&-\infty &\dots &-\infty \\0&0&0&\dots &-\infty \\\vdots &\vdots &\vdots &\ddots &\vdots \\0&0&0&\dots &0\end{bmatrix}}$$so the upper triangular part of $B$ does not matter.

The original paper designed $s$. Each attention head gets a different one. The first attention head gets $s=1/2$, the second attention head gets $s=1/4$, and so on.

ALiBi allows pretraining on short context windows, then fine-tuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).

More generally, $B$ can be a Toeplitz matrix.$$B={\begin{bmatrix}B_{0}&B_{-1}&B_{-2}&\cdots &\cdots &B_{-(n-1)}\\B_{1}&B_{0}&B_{-1}&\ddots &&\vdots \\B_{2}&B_{1}&\ddots &\ddots &\ddots &\vdots \\\vdots &\ddots &\ddots &\ddots &B_{-1}&B_{-2}\\\vdots &&\ddots &B_{1}&B_{0}&B_{-1}\\B_{n-1}&\cdots &\cdots &B_{2}&B_{1}&B_{0}\end{bmatrix}}$$with learned entries. The scalar $s$ can also be learned.

T5

The T5 transformer series used the more general version of ALiBi. Like the original RPR, it also groups the entries $B_{i}$ into "ranges". This enforces an inductive bias that, as positions become further apart, the resolution decreases. It is important to distinguish between 1 and 2, but less important to distinguish 10 and 12, etc. As the original authors state, the ranges "increase in size logarithmically up to an offset of 128 beyond which we assign all relative positions to the same embedding".

In detail, the T5 encoder encodes as follows: $B_{0}=0$, then the entries $B_{1},\dots ,B_{7}$ are distinct, then $B_{8}=\dots =B_{11}$, and so on, until $B_{91}=B_{92}=\dots$ unto infinity. Same on the other direction with $B_{-1},\dots ,B_{-7}$ etc.

RoPE

RoPE (Rotary Positional Encoding) is another scheme for relative positional encoding. It works by rotating the query and key vectors input into an attention mechanism block. It was proposed in 2021 in a Chinese blog, and then verified experimentally by Eleuther AI. Eleuther AI used it in their GPT-NeoX in early 2022, and it subsequently spread in popularity.

Consider a feature vector with even dimensions: $x=(x_{1},x_{2},\dots ,x_{2d-1},x_{2d})\in \mathbb {R} ^{2d}$. Like in the original sinusoidal positional encoding scheme, it can be written as a complex vector $v=(x_{1}+ix_{2},\dots ,x_{2d-1}+ix_{2d})\in \mathbb {C} ^{d}$. Then, to encode it as positioned at $t$, the $t$-th feature vector is represented by rotating its entries:$$\operatorname {RoPE} (((x_{1}+ix_{2}),\dots ,(x_{2d-1}+ix_{2d})),t):=(e^{i\omega _{1}t}(x_{1}+ix_{2}),\dots ,e^{i\omega _{d}t}(x_{2d-1}+ix_{2d}))$$where the frequencies $\omega _{1},\dots ,\omega _{d}\in \mathbb {R}$ are usually designed by hand. A standard design is $\omega _{k}=b^{-{\frac {k-1}{d}}}$ where $b$ is the base frequency hyperparameter, as in the original Transformer.

In the original RoPE paper and Llama, $b=10000$. Larger values of $b$ are often used for models with longer contexts. For example, during the training of Llama 3, the context length was 8192, but its $b=500000$, to "better support longer contexts".

More succinctly, we define $R_{\omega }(t)$ to be the diagonal matrix of $\operatorname {diag} (e^{i\omega _{1}t},e^{i\omega _{2}t},\dots ,e^{i\omega _{d}t})$, so that rotating a vector is the same as multiplying by the matrix on the right, as in $xR_{\omega }(t)$. In real space, it is converted to a block-diagonal matrix, each block being 2×2.

In an attention mechanism, RoPE is implemented as$$z_{ij}=\lambda q_{i}\underbrace {R_{\omega }(i)R_{\omega }(j)^{T}} _{\text{RoPE}}k_{j}^{T}+m_{ij}$$This exploits properties of the rotation matrix: $R_{\omega }(j)^{T}=R_{\omega }(-j)$ and $R_{\omega }(i)R_{\omega }(-j)=R_{\omega }(i-j)$.

Like the original positional encoding, it behaves as a base-$b^{1/d}$ positional number system.

Variants

Several variants begin by an observation about RoPE. In the equation$$z_{ij}=\lambda q_{i}\underbrace {R_{\omega }(i)R_{\omega }(j)^{T}} _{\text{RoPE}}k_{j}^{T}+m_{ij}$$the RoPE part is in the middle, and by the translational invariance of RoPE, is equal to $R_{\omega }(i-j)$. The idea is to modify the $i-j$ term.

• In ReRoPE ("ReLU RoPE"), it is clamped at some maximal length $L$. That is, $R_{\omega }(i-j)$ is replaced by $R_{\omega }(L)$ for all $i-j\geq L$.
• In Leaky ReRoPE ("leaky ReLU RoPE"), $R_{\omega }(i-j)$ is replaced by $R_{\omega }(L+s(i-j-L))$ for all $i-j\geq L$, where $s$ is some scale factor, a constant picked between 0 and 1.
• In Self-Extend, $R_{\omega }(i-j)$ is replaced by $R_{\omega }(L+\operatorname {floor} (s(i-j-L)))$.

These variants have seen use in context extension.

Extrapolatable Position Embedding (xPos) adds decay terms $\zeta _{1},\dots ,\zeta _{d}\in [0,1]$ to the rotation of RoPE, designed to damp out the high-frequency oscillations at long distances.$$\operatorname {xPos} (((x_{1}+ix_{2}),\dots ,(x_{2d-1}+ix_{2d})),t,\omega ,\zeta ):=(\zeta _{1}^{t}e^{i\omega _{1}t}(x_{1}+ix_{2}),\dots ,\zeta _{d}^{t}e^{i\omega _{d}t}(x_{2d-1}+ix_{2d}))$$This, as before, is represented as a diagonal matrix $R_{\omega ,\zeta }(i)=\operatorname {diag} (\zeta _{1}^{t}e^{i\omega _{1}t},\dots ,\zeta _{d}^{t}e^{i\omega _{d}t})$ in complex space, which is equivalent to a 2×2 block-diagonal matrix in real space.$$z_{ij}=\lambda q_{i}\underbrace {R_{\omega ,\zeta }(i)R_{\omega ,\zeta }(j)^{T}} _{\text{xPos}}k_{j}^{T}+m_{ij}$$The original paper suggested $\zeta _{i}={\frac {i/d+\gamma }{1+\gamma }},\gamma =0.4$. xPos is also limited to causal attention masking, since otherwise the attention logits will diverge.

RoPE-2D applies RoPE to tokens that are arranged on a plane. They are used in vision transformers. A vector $x$ representing a token at location $(i_{1},i_{2})$ is split into two halves, the first half rotated by $R_{\omega }(i_{1})$, and the second half rotated by $R_{\omega '}(i_{2})$. Here, $\omega$ and $\omega '$ do not have to be the same sequence of angular frequencies.

Similarly, Transformers that process videos may use RoPE-3D. It can also be used for other purposes. For example, the vision-language model Qwen-VL uses a version of RoPE-3D that they call "M-RoPE" (multimodal RoPE). Each token is given a triple index $(i,j,k)$. A video is divided into individual frames, then patches. The first index $i$ increments every image, frame, or textual token. For a patch, $j,k$ count the horizontal and vertical position. For a textual token, $j=k=i$.

RoPE with Relative distances (RoPER), or value-output RoPE (VO-RoPE), applies RoPE to value vectors.$$o_{i}=\left(\sum _{j}a_{ij}v_{j}R_{\omega }(-j)\right)R_{\omega }(i)$$

NoPE

A worked example of performing a convolution. The convolution has stride 1, zero-padding, with kernel size 3-by-3. The zero-padding allows the convolution to implicitly acquire absolute position of each pixel.

In NoPE (No Positional Encoding), there is no positional encoding. The idea is that a decoder's causal masking lets each token $t$ to see only tokens $1:t$, which is an implicit kind of positional encoding. Relatedly, pure convolutional networks are capable of implicit absolute positional encoding, by exploiting the zero-padding on the edge.

NoPE can represent both absolute and relative positional encodings. Empirically, when trained with gradient descent, a NoPE transformer and a transformer using T5-style positional encoding (ALiBi, learned) produce similar distributions of attention weights for the same input sequences, measured using Jensen–Shannon divergence. It was also shown theoretically and empirically that a trained NoPE transformer represents the first few tokens distinctly in the representational space compared to other positions, which it is capable of distinguishing via causal masking.

NoPE is rarely used for all attention modules of a Transformer, since it would degrade performance significantly. However, NoPE may be used for a fraction of attention modules. This was found in practice to be a worthwhile trade off of performance for computing cost. For example, Llama 4 alternates layers of NoPE and RoPE, with RoPE in 3 layers and NoPE in the 4th. They call it iRoPE ("interleaved RoPE").

Context extension

Since the per-token cost of inference scales with context length, training cost is decreased by using shorter contexts during training. However, this would make the resulting model bad at using longer contexts. Consequently, a common way to train a model is to "train short, test long", i.e. training a model on short contexts, and use it on long contexts.

Context extension faces two types of distributional shift:

• Of positional encoding. A longer context results in positional encoding vectors never seen in short contexts. This tends to make attention logits diverge at long contexts.
• Of attention weights. A longer context results in more entries to spread the attention weights, thus tends to produce flatter attention logits than shorter contexts. Empirically, the entropy of the attention weight distribution grows as $O(\log \ell )$, where $\ell$ is the context length.

Context extension was a concern since the original Transformer paper. In it, the authors reported trying both learned positional encoding and designed positional encoding. They found both to result in similar performance, and decided to use the designed positional encoding, since it "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training".

A slightly more sophisticated method is to train according to a curriculum: Begin by training on a short context length, then train on a longer context length. This can be repeated. For example, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K.

Context lengths can even be mixed during training. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than purely training on long contexts.

In general, context extension using RoPE consists of training a model on one context length $L$ with one set of RoPE angles $\omega _{1},\dots ,\omega _{d}$, then its RoPE angles are modified to $\omega _{1}',\dots ,\omega _{d}'$, and it is trained further on context length $L'=sL$ for some $s>1$.

Some architectures natively support context extension. For example, convolutional seq2seq, and sliding window attention, are both insensitive to context length.

YaRN

YaRN (Yet another RoPE extensioN) is a common way to extend context, used across many models, including DeepSeek, Qwen, Kimi, and GPT-OSS. It was built in steps.

In Positional Interpolation (PI), frequencies are stretched equally: $\omega _{k}'=\omega _{k}/s$. This was found to degrade performance beyond $s=8$, as the model loses the high-frequency resolution that tells adjacent tokens apart.

In Adjusted Base Frequency (ABF) (other names include "NTK-aware", "NTK-RoPE", "RoPE-ABF"), the base is increased: $\omega _{k}'=(b')^{-{\frac {k-1}{d}}}$, where $b'=bs^{\frac {d}{d-1}}$ is designed so that$$\omega _{1}'=\omega _{1},\quad \omega _{d}'=\omega _{d}/s$$Equivalently, $\omega _{k}'=\omega _{k}/s^{\frac {k-1}{d-1}}$. The effect is that the highest frequency is not stretched, the lowest frequency is stretched by $s$, and those in-between are interpolated.

In "NTK-by-parts", high frequencies are not stretched, low frequencies are stretched by $s$, and those in-between are interpolated:$$\omega _{k}'=\gamma _{k}\omega _{k}+(1-\gamma _{k})\omega _{k}/s$$where

• $\gamma _{k}=1$ if the feature rotates by more than $\beta$ times over the length of a context: $\omega _{k}L>2\pi \beta$. Equivalently, the wavelength is short enough: $\lambda _{k}<L/\beta$.
• $\gamma _{k}=0$ if the feature rotates by less than $\alpha$ times over the length of a context: $\omega _{k}L<2\pi \alpha$. Equivalently, the wavelength is long enough: $\lambda _{k}>L/\alpha$.
• $\gamma _{k}$ is linearly interpolated otherwise: $\gamma _{k}={\frac {L/\lambda _{k}-\alpha }{\beta -\alpha }}$.

YaRN uses NTK-by-parts with frequency scales $\alpha =1,\beta =32$, meaning that wavelengths shorter than $L/32$ are not stretched, wavelengths longer than $L$ are fully stretched by $s$, and those in-between are stretched by an in-between value.

YaRN also scales up the logits in the attention mechanism. A longer context results in more entries to spread the attention weights, thus tends to produce flatter attention weights than shorter contexts. Scaling up the logits un-flattens them. The authors experimented with context-extending models from the Llama and Llama 2 series, and found empirically that $\lambda =(1+0.1\ln s)^{2}/{\sqrt {d}}$ worked well.

Dynamic scaling

Because extending the context length involves a degradation of performance, during deployment, a model is used with its original context length $L$, and when the context exceeds $L$, switch to the extended context length $L'$. This was found to cause a sudden drop of performance at the switching point. Consequently, dynamic scaling ("dynamic NTK") was used to allow graceful degradation.

The algorithm:

• When context length $\ell$ is lower than the short context length $L$, use the positional encoding of $L$.
• When context length $\ell$ is longer than the short context length $L$, use the positional encoding obtained by extending the context by a factor of $\ell /L$.

Dynamic scaling can be used even with no extra training on extended context $L'$. It was first proposed in a post in the subreddit r/LocalLLaMA.

See also

• Transformer (deep learning)
• Context window
• KV cache
• Recurrent neural network
• Large language models
• Training of large language models