BpeTokenizer 1.0.2

.NET 7.0 This package targets .NET 7.0. The package is compatible with this framework or higher.
Suggested Alternatives

BpeTokenizer 1.0.5

Additional Details

Heavy refactoring of the internal types. Breaking changes from previous versions. This deprecates the prior versions for that reason.

There is a newer version of this package available.
See the version list below for details. 
dotnet add package BpeTokenizer --version 1.0.2                

                    
                

            
NuGet\Install-Package BpeTokenizer -Version 1.0.2
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="BpeTokenizer" Version="1.0.2" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
paket add BpeTokenizer --version 1.0.2                

                    
                

            
#r "nuget: BpeTokenizer, 1.0.2"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
// Install BpeTokenizer as a Cake Addin
#addin nuget:?package=BpeTokenizer&version=1.0.2

// Install BpeTokenizer as a Cake Tool
#tool nuget:?package=BpeTokenizer&version=1.0.2

BpeTokenizer

NuGet Last Commit GitHub Issues Used by Contributors License

BpeTokenizer is a C# implementation of tiktoken written by OpenAI. It is a byte pair encoding tokenizer that can be used to tokenize text into subword units.

As a BpeTokenizer derived from tiktoken, it can be used as a token counter. Useful to ensure that when streaming tokens from the OpenAI API for GPT Chat Completions, you could keep track of the cost related to the software calling the API.

This library is built for x64 architectures.

Product Compatible and additional computed target framework versions.
.NET net7.0 is compatible.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on BpeTokenizer:

Package Downloads
BpeChatAI

Package Description

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
1.0.5 225 7/21/2023
1.0.4 175 7/15/2023 1.0.4 is deprecated.
1.0.3 208 7/15/2023 1.0.3 is deprecated.
1.0.2 216 7/14/2023 1.0.2 is deprecated.
1.0.1 181 7/13/2023 1.0.1 is deprecated.
1.0.0 166 7/13/2023 1.0.0 is deprecated.

Altered how Decode method handles errors. Previously, "strict" did not behave as intended. Converted method to use an enum for its parameter vs a string. Added tags and documentation comments.