Repository files navigation

This repository provides a comprehensive evaluation framework for positional encoding methods in transformer-based time series models, along with implementations and benchmarking results.

Our work is available on arXiv:Positional Encoding in Transformer-Based Time Series Models: A Survey

We present a systematic analysis of positional encoding methods evaluated on two transformer architectures:

1. Multivariate Time Series Transformer Framework (TST)
2. Time Series Transformer with Patch Embedding

We implement and evaluate eight positional encoding methods:

Legend:

• Abs=Absolute, Rel=Relative, Hyb=Hybrid
• Add=Additive, Att=Attention, Comb=Combined
• F=Fixed, L=Learnable, M=Mixed
• L: sequence length, d: embedding dimension, h: attention heads, K: kernel size, l: layers

• Python 3.10
• PyTorch 2.4.1+cu121
• NumPy
• Scikit-learn
• CUDA 12.2

# Clone the repositorygit clone https://github.com/imics-lab/positional-encoding-benchmark.gitcd positional-encoding-benchmark# Create virtual environmentpython -m venv venvsource venv/bin/activate# Linux/Mac# or.\venv\Scripts\activate# Windows# Install dependenciespip install -r requirements.txt# Run benchmark with default configpython examples/run_benchmark.py# Or with custom configpython examples/run_benchmark.py --config path/to/custom_config.yaml

fromencodings.positional_encodingsimportPE_Namefrommodels.transformerimportTimeSeriesTransformer# Use in transformermodel=TimeSeriesTransformer(input_timesteps=SEQ_LENGTH,# Sequence lengthin_channels=INPUT_CHANNELS,# Number of input channelspatch_size=PATCH_SIZE,# Patch size for embeddingembedding_dim=EMBED_DIM.# Embedding dimensionnum_transformer_layers=NUM_LAYERS,# Number of transformer layers (4, 8, etc.)num_heads=N_HEADS,# Number of attention headsnum_layers=NUM_LAYERS,# Number of transformer layersdim_feedforward=DIM_FF,# Feedforward dimensiondropout=DROPOUT,# Dropout rate (0.1, 0.2, etc.)num_classes=NUM_CLASSES# Number of output classespos_encoding='PE_Name',# Positional encoding type    )# Forward passx=torch.randn(BATCH_SIZE,SEQ_LENGTH,INPUT_CHANNELS)# (batch, sequence, features)output=model(x)

Our experimental evaluation encompasses ten distinct positional encoding methods tested across eleven diverse time series datasets using two transformer architectures.

• Long sequences (>100 steps): 5-6% improvement with advanced methods
• Medium sequences (50-100 steps): 3-4% improvement
• Short sequences (<50 steps): 2-3% improvement

• TST: More distinct performance gaps
• Patch Embedding: More balanced performance among top methods

• SPE: 1.727 (TST), 2.090 (patch embed)
• TUPE: 1.909 (TST), 2.272 (patch embed)
• T-PE: 2.636 (TST), 2.363 (patch embed)

• TUPE achieves highest average accuracy
• SPE shows strong performance
• Both methods demonstrate effectiveness in capturing long-range dependencies

• SPE exhibits superior performance
• TUPE maintains competitive accuracy
• Relative encoding methods show improved local pattern recognition

Training time measurements on Melbourne Pedestrian dataset (100 epochs):

ConvSPE emerges as the efficiency frontier leader, achieving highest accuracy (75.3%) with reasonable computational overhead (2.11×).

• Short sequences (L ≤ 50): Learnable PE or tAPE (minimal gains don't justify computational overhead)
• Medium sequences (50 < L ≤ 100): SPE or eRPE (3-4% accuracy improvements)
• Long sequences (L > 100): TUPE for complex patterns, SPE for regular data, ConvSPE for linear complexity

• Biomedical signals: TUPE > SPE > T-PE (physiological complexity handling)
• Environmental sensors: SPE > eRPE (regular sampling patterns)
• High-dimensional data (d > 5): Advanced methods consistently outperform simple approaches

• Limited resources: Sinusoidal PE, tAPE (O(Ld) complexity)
• Balanced scenarios: SPE, TUPE (optimal accuracy-efficiency trade-off)
• Performance-critical: TUPE, SPE regardless of computational cost

• Time Series Transformers: Prioritize content-position separation methods (TUPE) and relative positioning (eRPE, SPE)
• Patch Embedding Transformers: Multi-scale approaches (T-PE, ConvSPE) handle hierarchical processing more effectively

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

@article{irani2025positional,title={Positional Encoding in Transformer-Based Time Series Models: A Survey},author={Irani, Habib and Metsis, Vangelis},journal={arXiv preprint arXiv:2502.12370},year={2025}}