Bidirectional Sign Language Translation Using Deep Learning Transformers

Overview

Bidirectional sign language translation refers to the ability to translate between sign language and spoken/written language in both directions: (1) Sign-to-Text/Speech (sign language recognition and translation into spoken language) and (2) Text/Speech-to-Sign (sign language production from spoken language). Deep learning transformer architectures have revolutionized this field, offering powerful sequence-to-sequence modeling capabilities that can handle the complex spatio-temporal nature of sign languages.

1. The Sign-to-Text Direction (Sign Language Recognition & Translation)

Transformer-Based Architectures

The seminal work by Camgoz et al. introduced Sign Language Transformers, a novel architecture that jointly learns continuous sign language recognition (CSLR) and translation (SLT) in an end-to-end manner [2]. Their approach uses a transformer encoder (Sign Language Recognition Transformer or SLRT) trained with Connectionist Temporal Classification (CTC) loss to recognize sign gloss sequences, combined with an autoregressive transformer decoder (Sign Language Translation Transformer or SLTT) that generates spoken language sentences. This joint multi-task learning formulation achieved state-of-the-art results on the PHOENIX14T dataset, doubling previous translation performance (reaching 21.32 BLEU-4 on the test set), and importantly surpassing text-to-text gloss-based translation — demonstrating that direct video-to-text translation can outperform approaches that rely on intermediate gloss representations [2].

Keypoint-Based Approaches

Modern sign language translation systems increasingly use keypoint-based representations extracted via tools like MediaPipe or OpenPose, rather than raw RGB frames. This reduces computational complexity and improves robustness to background variations.

Maia et al. developed a transformer-based system using MediaPipe Holistic to extract body keypoints, which are then processed through convolutional keypoint embeddings and transformer blocks for Sign2Gloss translation, followed by fine-tuning BART for Gloss2Text translation [1]. Their Sign2Gloss2Text pipeline achieved a BLEU-4 score of 21.30 on the PHOENIX14T dataset using only 2D keypoints, demonstrating that strong performance is achievable without pixel-level video data [1].

Kim et al. proposed a robust keypoint normalization method for Korean sign language translation using transformers, normalizing keypoint positions based on the length of the neck-shoulder bone [3]. This normalization proved critical — normalization based on neck or shoulder lengths significantly outperformed both no normalization and mean/std normalization, achieving a BLEU-4 score of 0.76 compared to 0.72 with mean/std normalization [3].

Multi-Stream Attention Networks

Guan et al. introduced the Multi-Stream Keypoint Attention Network (MSKA-SLT), which decouples keypoint sequences into four parallel streams: left hand, right hand, face, and whole body [6]. Each stream uses pure attention modules without manually designed graph topologies, and features are fused using self-distillation techniques. On the PHOENIX14T dataset, this approach achieved state-of-the-art results with a BLEU-4 score of 29.03, substantially improving upon previous methods while using significantly fewer FLOPs (861G vs. 9970G for TwoStream) [6].

Real-Time Considerations

Jintanachanaiwat et al. demonstrated that LSTM models combined with MediaPipe Holistic can achieve 86% real-time accuracy for Thai sign language translation, though they noted significant accuracy drops when moving from controlled test datasets to real-world scenarios with diverse signers [10].

2. The Text-to-Sign Direction (Sign Language Production)

Neural Machine Translation + Motion Graphs

Stoll et al. introduced Text2Sign, the first approach to continuous sign video generation that does not use a classical graphical avatar [5]. Their pipeline consists of two stages:

Text2Pose: An attention-based NMT network (encoder-decoder with GRUs) translates spoken language sentences into sign gloss sequences. These gloss probabilities are then used to solve a Motion Graph (MG) constructed from skeletal pose data, generating continuous sign pose sequences.
Pose2Video: A pose-conditioned GAN generates photorealistic sign language video frames from the pose sequences. The system supports both multi-signer generation (using an encoder-decoder with 1D latent space) and high-definition (HD) sign generation (using a fully convolutional architecture with feature matching loss) [5].

The NMT component achieved a BLEU-4 score of 16.34/15.26 on dev/test sets for text-to-gloss translation on PHOENIX14T [5].

Avatar-Based Approaches

Traditional sign language production has relied on 3D avatar animation, driven either by motion capture data or parametric descriptions of signs (e.g., HamNoSys, SiGML). Ferreira et al. conducted a systematic review of spoken-to-sign translation systems and identified four dominant technological approaches: ASR-NLP pipelines, avatar-based rendering systems, VR/AR learning tools, and sensor-based gesture recognition platforms [4]. Key challenges identified include the representation of non-manual signals (facial expressions, head movements, body posture), multimodal grammar processing, and scarcity of high-quality datasets [4].

Challenges in Production

Wolfe highlighted several fundamental challenges in sign language production [8]:

Differences in modality: Sign languages use multiple concurrent articulators (hands, face, head, torso) simultaneously, unlike the sequential nature of speech.
Absence of a standard written form: This prevents the direct application of text-based translation technologies.
The "uncanny valley": Avatars that are not sufficiently realistic can be unsettling or difficult to understand.

Bragg et al. emphasized that avatar generation faces challenges including realistic transitions between signs, modeling of modulations (e.g., changing movement quality to convey adverbs), and the difficulty of finding "model holes" — gaps in the generation model that produce unnatural output [9].

3. Integrated Bidirectional Systems

End-to-End Bidirectional Frameworks

Fernandes et al. presented a complete bidirectional sign language translation system that handles both directions [7]. For sign-to-speech, their system uses a CNN trained on a custom dataset of 1,200 images per gesture, achieving 99.98% accuracy for alphabet recognition. The recognized text is then converted to speech via an Android application. For speech-to-sign, the system takes speech input, converts it to text using speech recognition libraries, and then generates a video sequence of corresponding sign language gestures. When a recognized phrase has a dedicated gesture (e.g., "THANK YOU"), the system displays that specific gesture rather than spelling it out letter by letter [7].

Current Limitations

Despite progress, bidirectional sign language translation faces significant obstacles:

Data scarcity: Sign language datasets are orders of magnitude smaller than speech datasets. While speech recognition relies on corpora of millions of words, sign language datasets typically contain fewer than 100,000 signs [4,9].
Modality gap: Sign languages are visual-spatial and multimodal, using simultaneous articulation across multiple channels. Current sequential models struggle to capture this parallelism [8,9].
Non-manual signals: Facial expressions, eye gaze, head movements, and body posture convey critical grammatical and affective information but are often neglected in current systems [4,9].
Generalization: Most systems struggle to generalize to unseen signers, different dialects, and real-world conditions [9,10].
Lack of standardization: No widely accepted annotation system exists for sign languages, hindering data sharing and model development [8,9].

4. Key Datasets and Benchmarks

The RWTH-PHOENIX-Weather 2014T (PHOENIX14T) dataset remains the primary benchmark for sign language translation research [1,2,5]. It contains 8,257 sequences of German Sign Language (DGS) weather broadcasts performed by 9 signers, with parallel gloss annotations and German spoken language translations (gloss vocabulary: 1,066; text vocabulary: 2,887) [2]. Other important datasets include How2Sign (American Sign Language, ~60 hours, gloss-free) [1], CSL-Daily (Chinese Sign Language) [6], and the KETI Sign Language Dataset (Korean) [3].

Evaluation metrics include BLEU-4 (n-gram precision for translation quality), ROUGE (recall-oriented metrics), METEOR (semantic similarity), and Word Error Rate (WER) (for recognition tasks) [1,2].

5. Future Directions

Several research directions are critical for advancing bidirectional sign language translation:

Large Language Models (LLMs) integration into the translation pipeline to improve contextual understanding and fluency [1].
Gloss-free approaches that reduce dependency on costly gloss annotations, enabling scaling to more languages and domains [1,6].
Improved non-manual signal modeling through deeper multimodal integration of facial expressions, head movements, and gaze [2,4].
Deaf community involvement in all stages of development to ensure culturally appropriate and practically useful systems [9].
Unsupervised and self-supervised learning methods to leverage unlabeled sign language video data [9].
End-to-end differentiable pipelines for sign language production that can be trained jointly rather than in separate stages [5].

References

[1]W. F. Maia, A. M. Lopes, and S. A. David, "Automatic sign language to text translation using MediaPipe and transformer architectures," Neurocomputing, vol. 642, p. 130421, Aug. 2025
DOI: 10.1016/j.neucom.2025.130421 [2]N. Cihan Camgoz, O. Koller, S. Hadfield, and R. Bowden, "Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation," in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Jun. 2020, pp. 10020–10030
DOI: 10.1109/cvpr42600.2020.01004 [3]S. Kim, C. J. Kim, H.-M. Park, Y. Jeong, J. Y. Jang, and H. Jung, "Robust Keypoint Normalization Method for Korean Sign Language Translation using Transformer," in 2020 International Conference on Information and Communication Technology Convergence (ICTC), IEEE, Oct. 2020
DOI: 10.1109/ictc49870.2020.9289551 [4]W. Ferreira, P. Tanajura, J. P. Simas, Y. T. S. Ferreira, M. Soussa, and I. Winkler, "Automatic Spoken-To-Sign Language Translation: A Review of Key Technologies, Limitations, and Interdisciplinary Opportunities," in 2025 27th Symposium on Virtual and Augmented Reality (SVR), IEEE, Sep. 2025, pp. 423–432
DOI: 10.1109/svr67689.2025.00062 [5]S. Stoll, N. C. Camgoz, S. Hadfield, and R. Bowden, "Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks," International Journal of Computer Vision, vol. 128, no. 4, pp. 891–908, Jan. 2020
DOI: 10.1007/s11263-019-01281-2 [6]M. Guan, Y. Wang, G. Ma, J. Liu, and M. Sun, "MSKA: Multi-stream keypoint attention network for sign language recognition and translation," Pattern Recognition, vol. 165, p. 111602, Sep. 2025
DOI: 10.1016/j.patcog.2025.111602 [7]L. Fernandes, P. Dalvi, A. Junnarkar, and M. Bansode, "Convolutional Neural Network based Bidirectional Sign Language Translation System," in 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), IEEE, Aug. 2020, pp. 769–775
DOI: 10.1109/icssit48917.2020.9214272 [8]R. Wolfe, "Special issue: Sign language translation and avatar technology," Machine Translation, vol. 35, no. 3, pp. 301–304, Jun. 2021
DOI: 10.1007/s10590-021-09270-4 [9]D. Bragg et al., "Sign Language Recognition, Generation, and Translation," in Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ACM, Oct. 2019, pp. 16–31
DOI: 10.1145/3308561.3353774 [10]W. Jintanachaiwat et al., "Using LSTM to translate Thai sign language to text in real time," Discover Artificial Intelligence, vol. 4, no. 1, Feb. 2024
DOI: 10.1007/s44163-024-00113-8

Echoes Of Day

Bu Blogda Ara