python3-tokenizers Subpackage of python-tokenizers
Implementation of today's most used tokenizers
Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Bindings over the rust-tokenizers implementation.
Releases Overview
| Release | Stable | Testing |
|---|---|---|
| Fedora Rawhide | 0.22.2-1.fc45 | - |
| Fedora 44 | 0.22.2-1.fc44 | - |
File a new bug report »
Package Info
Package Info
- Upstream: https://github.com/huggingface/tokenizers
- License(s): Unicode-DFS-2016 AND Apache-2.0 AND (Apache-2.0 AND MIT) AND (Apache-2.0 OR BSL-1.0) AND (Apache-2.0 OR MIT OR Zlib) AND BSD-2-Clause AND MIT AND (Unlicense OR MIT)
- Maintainer: xanderlent
You can contact the maintainers of this package via email at
python-tokenizers dash maintainers at fedoraproject dot org.
Sources on Pagure