python3-pytorch-tokenizers - Fedora Packages

C++ implementations for various tokenizers (sentencepiece, tiktoken etc). Useful for other PyTorch repos such as torchchat, ExecuTorch to build LLM runners using ExecuTorch stack or AOT Inductor stack.

Releases Overview

Release	Stable	Testing
Fedora Rawhide	1.4.1-4.fc46	-
Fedora 45	1.4.1-4.fc45	-

File a new bug report »

Package Info

Upstream: https://github.com/meta-pytorch/tokenizers
License(s): BSD-3-Clause AND Apache-2.0 AND MIT
Maintainer: pbrobinson

Related Packages

python3-pytorch-tokenizers-devel

You can contact the maintainers of this package via email at python-pytorch-tokenizers dash maintainers at fedoraproject dot org.

python3-pytorch-tokenizers Subpackage of python-pytorch-tokenizers

Releases Overview