A small demo app showcasing how the texts are encoded into tokens
and how they are decoded back to characters by AI Tokenizer
This is inspired by Steve Kinney's Python: AI & Hugging Face Workshop on Frontend Masters
Built using Hugging Face Transformers.js CDN/library and Models to tokenize, encode and decode text using various AI models
- Steps to use the app
- Choose your preferred AI Model from the dropdown
- Enter text in the first text area to tokenize & encode the text
- Enter encoded IDs in the second text area to decode them
Choose a Model
Enter text to tokenize & encode
Converts text by adding special tokens and converting them to IDs respective to the model chosen.
Decode Tokens
Converts encoded IDs back to text.
Model used
Model's max text length
Tokenized
Encoded
Decoded