From aafcd3d70d85eaacc63aabed194c48b9d0d9a70b Mon Sep 17 00:00:00 2001
From: Ali-Nawaz-devt
4. Why should I avoid transformers.LlamaTokenizer for loading this model?
-Certain versions of Hugging Face Transformers' LlamaTokenizer fail to load custom SentencePiece Unigram models correctly, silently substituting an empty or 3-token fallback vocabulary without throwing an exception. Always use sentencepiece.SentencePieceProcessor directly or via our verified SPMTokenizerAdapter.
+Certain versions of Hugging Face Transformers' LlamaTokenizer fail to load custom SentencePiece Unigram models correctly, silently substituting an empty or 3-token fallback vocabulary without throwing an exception. Use the SindhiTokenizer SDK or sentencepiece.SentencePieceProcessor directly.
+5. Does pip install the tokenizer model too?
+No. The PyPI wheel installs the SDK and its runtime dependencies. The first call to SindhiTokenizer.from_pretrained() downloads the selected model from Hugging Face and caches it locally. Use local_files_only=True to require a cached model without network access.
+6. Which vocabulary sizes can I load?
+The released model sizes are 32000, 64000 (the recommended default), 100000, and 150000. Pass one of these values to SindhiTokenizer.from_pretrained(vocab_size=...).
Sindhi Tokenizer Documentation
Sindhi Tokenizer Documentation
silent deletion defects observed in current general-purpose multilingual LLMs and existing monolingual
models.
+ The Python SDK is published on PyPI as sindhi-tokenizer. Install it with pip
+ and use the SindhiTokenizer API to load a released model, tokenize text, encode IDs or pieces,
+ decode text, and process batches. The SDK supports Python 3.9–3.12; model files are downloaded separately
+ from Hugging Face and cached locally.
+
The flagship CustomSindhi_64k model achieves an average subword fertility of 1.147 tokens per word and 88.6% single-token word coverage across four evaluation @@ -173,7 +179,7 @@
- The tokenizers can be used either through the official Google SentencePiece Python library or via Hugging
- Face's transformers library.
+ The recommended way to use the released tokenizer models is the sindhi-tokenizer Python SDK
+ on PyPI. It supports Python 3.9–3.12 and installs the two runtime dependencies it needs: SentencePiece and
+ the Hugging Face Hub client. You do not need to clone this repository to use the SDK.
# Install required runtime dependencies
-pip install sentencepiece transformers tokenizers
+ python -m pip install --upgrade sindhi-tokenizer
+ PyPI package: sindhi-tokenizer. The SDK wheel
+ contains the Python package, not the model files. The first call to from_pretrained()
+ downloads the selected model from Hugging Face; later calls reuse the Hugging Face Hub cache. To pin the
+ release documented here, install sindhi-tokenizer==1.1.0.
+
- Loading via SentencePiece directly is the most robust and performant interface: + A virtual environment is recommended to keep dependencies separate from other projects:
+py -3.12 -m venv .venv
+.\.venv\Scripts\Activate.ps1
+python -m pip install --upgrade pip
+python -m pip install sindhi-tokenizer
+ python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install sindhi-tokenizer
+ import sentencepiece as spm
+ from sindhi_tokenizer import SindhiTokenizer
+
+# Defaults to the recommended 64,000-piece model.
+tokenizer = SindhiTokenizer.from_pretrained()
+text = "سنڌ جي تاريخ تمام قديم آهي"
+
+pieces = tokenizer.tokenize(text)
+token_ids = tokenizer.encode(text)
+
+print("Pieces:", pieces)
+print("Token IDs:", token_ids)
+print("Vocabulary size:", tokenizer.vocab_size) # 64000
+
+decoded = tokenizer.decode(token_ids)
+assert decoded == text
+print("Decoded:", decoded)
+
+ Four vocabulary sizes are available: 32000, 64000, 100000, and
+ 150000. Pass the size when loading a different model:
+
from sindhi_tokenizer import SindhiTokenizer
-text = "سنڌ جي تاريخ تمام قديم ۽ وسيع ثقافتي ورثي تي مشتمل آهي"
+tokenizer_100k = SindhiTokenizer.from_pretrained(vocab_size=100000)
+pieces = tokenizer_100k.encode("سنڌي ٻولي", out_type="pieces")
+print(pieces)
+
+ encode() returns integer IDs by default. Use out_type="pieces" for readable
+ SentencePiece strings. The SDK also provides batch methods and can load a local SentencePiece
+ .model file.
+
from sindhi_tokenizer import SindhiTokenizer
+
+tokenizer = SindhiTokenizer.from_pretrained()
+texts = ["سنڌ جي تاريخ", "هي سنڌي ٽوڪنائزر آهي"]
+batch_ids = tokenizer.encode_batch(texts)
+decoded_texts = tokenizer.decode_batch(batch_ids)
+assert decoded_texts == texts
+
+pieces = tokenizer.encode(texts[0], out_type="pieces")
+ids = tokenizer.encode(texts[0])
+assert tokenizer.decode(pieces) == texts[0]
+assert tokenizer.id_to_piece(ids[0]) == pieces[0]
+assert tokenizer.piece_to_id(pieces[0]) == ids[0]
+
+# Replace this example path with a local model file.
+local_tokenizer = SindhiTokenizer.from_file("path/to/sindhi_custom_64000.model")
+
+ If a model has already been cached, local_files_only=True prevents an online lookup:
+
from sindhi_tokenizer import SindhiTokenizer
-# Round-trip decode example
-decoded = sp.decode_pieces(pieces)
-assert decoded == text, "This example should round-trip exactly"
+tokenizer = SindhiTokenizer.from_pretrained(local_files_only=True)
- You can load the published tokenizer directly from the Hugging Face Hub using
- PreTrainedTokenizerFast:
+ Run this short check after installation. The first model load requires internet access to Hugging Face;
+ later loads use the local Hub cache.
from transformers import PreTrainedTokenizerFast
+ from sindhi_tokenizer import SindhiTokenizer
-# Load from Hugging Face Model Hub
-tokenizer = PreTrainedTokenizerFast.from_pretrained("alinawazmahar/sindhi-tokenizer")
+tokenizer = SindhiTokenizer.from_pretrained()
+text = "سنڌ جي تاريخ تمام قديم آهي"
+ids = tokenizer.encode(text)
+assert tokenizer.decode(ids) == text
+print("SDK works; vocabulary:", tokenizer.vocab_size)
+ | API | Purpose |
|---|---|
SindhiTokenizer.from_pretrained(...) | Load a Hub model. Options include model_id, vocab_size, revision, cache_dir, and local_files_only. |
SindhiTokenizer.from_file(path) | Load a local SentencePiece .model file. |
tokenizer.tokenize(text) | Return readable SentencePiece pieces. |
tokenizer.encode(text, out_type="ids") | Return integer IDs; set out_type="pieces" for piece strings. |
tokenizer.decode(tokens) | Decode a sequence of IDs or piece strings back to text. |
encode_batch(texts) / decode_batch(batch) | Encode or decode multiple texts in order. |
id_to_piece(id) / piece_to_id(piece) | Convert between a token ID and its piece string. |
tokenizer.vocab_size | Read the loaded model's vocabulary size. |
transformers.LlamaTokenizer directly on the raw .model file.
- LlamaTokenizer assumes a hard-coded 32k BPE structure and silently corrupts custom Unigram
- vocabularies (often reducing them to 3 tokens). Always load via PreTrainedTokenizerFast or use
- the provided SPMTokenizerAdapter class in evaluation/metrics.py.
+ .model files directly to
+ transformers.LlamaTokenizer; some versions silently produce a fallback vocabulary. Use
+ SindhiTokenizer from this SDK or sentencepiece.SentencePieceProcessor directly.
+ The SDK code is MIT licensed. The model files have separate licensing and attribution terms; review
+ Data Sources and Licensing and the
+ Hugging Face model card before
+ redistributing model files.
- The project repository is structured cleanly with separated concerns for training, evaluation, data - preparation, documentation, and model artifacts: + The repository separates the installable inference SDK, model evaluation, training code, research materials, + and website source. Large datasets and downloaded model files are not included in the PyPI wheel.
git clone https://github.com/alinawazcode/sindhi-tokenizer.git
cd sindhi-tokenizer
-pip install -r requirements.txt
+python -m pip install -e ".[dev,training]"
+ This source install is for development and training. For ordinary tokenizer use, install the published
+ package with python -m pip install sindhi-tokenizer as shown above. Training requires a
+ prepared text corpus, which is not bundled in this repository; see the
+ training guide and data source
+ notes.
+
# Train 64k flagship model on processed corpus
-python training/train.py --vocab_size 64000 --model_type unigram --output_dir tokenizers/custom_unigram_64k/
+ # One sentence per line; the model files are written to models/
+python training/train_tokenizer.py --corpus corpus.txt --out-dir models --vocab-sizes 64000
# Run comparative evaluation across 18 benchmark rows and 4 evaluation registers
-python evaluation/benchmark.py --output results/macro_benchmark_results.json
+ # The text file should contain one sentence per line.
+python evaluation/metrics.py --model models/sindhi_custom_64000.model --text data/test_sentences.txt
+ This command evaluates one model against one text file; it does not recreate the full benchmark matrix. + For the benchmark materials and results, see the repository notebooks and + benchmark results guide. +
@@ -958,9 +1059,10 @@.model and Hugging Face tokenizer.json files can be used
- directly as the base tokenizer or merged with existing multilingual vocabularies using vocabulary
- expansion techniques (e.g., via transformers.PreTrainedTokenizerFast).
+ The SDK exposes SentencePiece IDs and pieces for downstream preprocessing and model workflows. For
+ Hugging Face Transformers or vocabulary expansion, use a tokenizer configuration supported by your
+ target architecture and validate that tokenizer IDs match the model embedding table. The SDK itself
+ is not a transformers.PreTrainedTokenizer subclass.
The repository code is licensed under MIT. The tokenizer models are currently labeled CC BY-SA 4.0 on the project listing, but compatibility with all training-source terms remains under review. See - Data Sources and Licensing before relying on the model-file license. + Data Sources and Licensing before relying on the model-file license.