rust-tokenizers+parity-aware-bpe-devel Subpackage of rust-tokenizers
Implementation of today's most used tokenizers in Rust
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility. This package contains library source intended for building other packages which use the "parity-aware-bpe" feature of the "tokenizers" crate.
Releases Overview
| Release | Stable | Testing |
|---|---|---|
| Fedora Rawhide | 0.23.2-1.fc46 | - |
| Fedora 45 | - | 0.23.2-1.fc45 |
File a new bug report »
Package Info
Related Packages
Package Info
- Upstream: https://crates.io/crates/tokenizers
- License(s): Apache-2.0
- Maintainer: xanderlent
Related Packages
- rust-tokenizers+default-devel
- rust-tokenizers+esaxx_fast-devel
- rust-tokenizers+fancy-regex-devel
- rust-tokenizers+hf-hub-devel
- rust-tokenizers+http-devel
- View More »
You can contact the maintainers of this package via email at
rust-tokenizers dash maintainers at fedoraproject dot org.
Sources on Pagure