AICentral 0.17.0-pullrequest0123-0015

This is a prerelease version of AICentral.
There is a newer version of this package available.
See the version list below for details.
dotnet add package AICentral --version 0.17.0-pullrequest0123-0015
                    
NuGet\Install-Package AICentral -Version 0.17.0-pullrequest0123-0015
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="AICentral" Version="0.17.0-pullrequest0123-0015" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="AICentral" Version="0.17.0-pullrequest0123-0015" />
                    
Directory.Packages.props
<PackageReference Include="AICentral" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add AICentral --version 0.17.0-pullrequest0123-0015
                    
#r "nuget: AICentral, 0.17.0-pullrequest0123-0015"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#addin nuget:?package=AICentral&version=0.17.0-pullrequest0123-0015&prerelease
                    
Install AICentral as a Cake Addin
#tool nuget:?package=AICentral&version=0.17.0-pullrequest0123-0015&prerelease
                    
Install AICentral as a Cake Tool

AI Central

AI Central gives you control over your AI services.

  • Lightweight out-the-box token logging using Open Telemetry
  • Intelligent Routing
  • Custom consumer OAuth2 authorisation
  • Fallback AI service
  • Round Robin AI services
  • Lowest Latency AI service
  • Circuit breakers, and backoff-retry over downstream AI services
  • Request based and Token based rate limiting
  • Prompt and usage logging
    • Works for streaming endpoints as-well as non streaming
  • Additional Open Telemetry Metrics and Traces

Configuration

See docs on Github for more details.

Minimal

This sample produces a AI-Central proxy that

  • Listens on a hostname of your choosing
  • Proxies directly through to a back-end Open AI server
  • Can be accessed using standard SDKs
  • Outputs open-telemetry metrics to capture usage information
{
  "AICentral": {
    "Endpoints": [
      {
        "Type": "AzureOpenAIEndpoint",
        "Name": "openai-1",
        "Properties": {
          "LanguageEndpoint": "https://<my-ai>.openai.azure.com",
          "AuthenticationType": "ApiKey",
          "ApiKey": "<key>"
        }
      }
    ],
    "EndpointSelectors": [
      {
        "Type": "SingleEndpoint",
        "Name": "default",
        "Properties": {
          "Endpoint": "openai-1"
        }
      }
    ],
    "AuthProviders": [
      {
        "Type": "ApiKey",
        "Name": "apikey",
        "Properties": {
          "Clients": [
            {
              "ClientName": "Consumer-1",
              "Key1": "<random-key>",
              "Key2": "<random-key>"
            }
          ]
        }
      }
    ],
    "Pipelines": [
      {
        "Name": "OpenAIPipeline",
        "Host": "mypipeline.mydomain.com",
        "EndpointSelector": "default",
        "AuthProvider": "apikey",
        "OpenTelemetryConfig": {
          "AddClientNameTag": true,
          "Transmit": true
        }
      }
    ]
  }
}

Full example

This pipeline will:

  • Present an Azure Open AI, and an Open AI downstream as a single upstream endpoint
    • maps the incoming deployment Name "GPT35Turbo0613" to the downstream Azure Open AI deployment "MyGptModel"
    • maps incoming Azure Open AI deployments to Open AI models
  • Present it as an Azure Open AI style endpoint
  • Protect the front-end by requiring an AAD token issued for your own AAD application
  • Put a local Asp.Net core rate-limiting policy over the endpoint
  • Emit Open Telemetry to be picked up by your OTel collector
  • Add rich logging to Azure monitor
    • Logs quota, client caller information (IP and identity name), and in this case the Prompt but not the response.
  • Publish the client-name as a tag in Open Telemetry
{
  "AICentral": {
    "Endpoints": [
      {
        "Type": "AzureOpenAIEndpoint",
        "Name": "openai-priority",
        "Properties": {
          "LanguageEndpoint": "https://<my-ai>.openai.azure.com",
          "AuthenticationType": "Entra|EntraPassThrough|ApiKey",
          "MaxConcurrency": 10,
          "ModelMappings": {
            "Gpt35Turbo0613": "MyGptModel"
          }
        }
      },
      {
        "Type": "OpenAIEndpoint",
        "Name": "openai-fallback",
        "Properties": {
          "LanguageEndpoint": "https://api.openai.com",
          "ModelMappings": {
            "Gpt35Turbo0613": "gpt-3.5-turbo",
            "Ada002Embedding": "text-embedding-ada-002"
          },
          "ApiKey": "<my-api-key>",
          "Organization": "<optional-organisation>"
        }
      }
    ],
    "AuthProviders": [
      {
        "Type": "Entra",
        "Name": "simple-aad",
        "Properties": {
          "Entra": {
            "ClientId": "<my-client-id>",
            "TenantId": "<my-tenant-id>",
            "Instance": "https://login.microsoftonline.com/",
            "Audience": "<custom-audience>"
          }
        }
      }
    ],
    "EndpointSelectors": [
      {
        "Type": "Prioritised",
        "Name": "my-endpoint-selector",
        "Properties": {
          "PriorityEndpoints": ["openai-1"],
          "FallbackEndpoints": ["openai-fallback"]
        }
      }
    ],
    "GenericSteps": [
      {
        "Type": "AspNetCoreFixedWindowRateLimiting",
        "Name": "token-rate-limiter",
        "Properties": {
          "LimitType": "PerConsumer|PerAICentralEndpoint",
          "MetricType": "Tokens",
          "Options": {
            "Window": "00:01:00",
            "PermitLimit": 1000
          }
        }
      },
      {
        "Type": "AspNetCoreFixedWindowRateLimiting",
        "Name": "window-rate-limiter",
        "Properties": {
          "LimitType": "PerConsumer|PerAICentralEndpoint",
          "MetricType": "Requests",
          "Options": {
            "Window": "00:00:10",
            "PermitLimit": 100
          }
        }
      },
      {
        "Type": "AzureMonitorLogger",
        "Name": "azure-monitor-logger",
        "Properties": {
          "WorkspaceId": "<workspace-id>",
          "Key": "<key>",
          "LogPrompt": true,
          "LogResponse": false,
          "LogClient": true
        }
      },
      {
        "Type": "BulkHead",
        "Name": "bulk-head",
        "Properties": {
          "MaxConcurrency": 20
        }
      }
    ],
    "Pipelines": [
      {
        "Name": "MyPipeline",
        "Host": "prioritypipeline.mydomain.com",
        "EndpointSelector": "my-endpoint-selector",
        "AuthProvider": "simple-aad",
        "Steps": [
          "window-rate-limiter",
          "bulk-head",
          "azure-monitor-logger"
        ],
        "OpenTelemetryConfig": {
          "AddClientNameTag": true,
          "Transmit": true
        }
      }
    ]
  }
}

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 was computed.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.21.4 906 2/21/2025
0.21.4-pullrequest0155-0003 66 2/21/2025
0.21.3 78 2/21/2025
0.21.3-pullrequest0155-0002 65 1/30/2025
0.21.1 602 1/14/2025
0.21.1-pullrequest0153-0003 49 1/14/2025
0.21.0 371 12/9/2024
0.21.0-pullrequest0152-0005 61 12/9/2024
0.21.0-pullrequest0152-0004 61 12/9/2024
0.21.0-pullrequest0152-0003 59 12/9/2024
0.21.0-pullrequest0152-0002 59 12/9/2024
0.20.7 231 11/26/2024
0.20.7-pullrequest0151-0002 63 11/26/2024
0.20.6 369 11/1/2024
0.20.6-pullrequest0150-0003 58 11/1/2024
0.20.5 265 9/30/2024
0.20.5-pullrequest0148-0004 71 9/30/2024
0.20.5-pullrequest0148-0003 67 9/30/2024
0.20.5-pullrequest0148-0002 87 9/30/2024
0.20.3 119 9/5/2024
0.20.3-pullrequest0146-0005 60 9/5/2024
0.20.3-pullrequest0146-0004 63 9/5/2024
0.20.2 85 9/4/2024
0.20.2-pullrequest0145-0004 53 9/4/2024
0.20.0 109 8/17/2024
0.20.0-pullrequest0143-0013 77 8/17/2024
0.20.0-pullrequest0143-0012 76 8/17/2024
0.20.0-pullrequest0143-0011 77 8/17/2024
0.20.0-pullrequest0143-0010 75 8/17/2024
0.20.0-pullrequest0143-0009 74 8/17/2024
0.19.6-pullrequest0143-0008 83 8/17/2024
0.19.5 78 8/7/2024
0.19.5-pullrequest0141-0002 56 8/7/2024
0.19.4 81 8/1/2024
0.19.4-pullrequest0140-0003 58 8/1/2024
0.19.4-pullrequest0139-0001 57 7/29/2024
0.19.3 76 7/29/2024
0.19.2 81 7/29/2024
0.19.2-pullrequest0138-0004 58 7/29/2024
0.19.1 75 7/29/2024
0.19.1-pullrequest0137-0003 59 7/29/2024
0.19.0 70 7/29/2024
0.19.0-pullrequest0136-0014 56 7/29/2024
0.19.0-pullrequest0136-0013 56 7/29/2024
0.19.0-pullrequest0136-0012 146 7/29/2024
0.19.0-pullrequest0136-0011 58 7/29/2024
0.19.0-pullrequest0136-0010 60 7/29/2024
0.19.0-pullrequest0136-0009 71 7/27/2024
0.19.0-pullrequest0136-0008 63 7/27/2024
0.19.0-pullrequest0136-0007 64 7/27/2024
0.18.6 92 7/15/2024
0.18.6-pullrequest0135-0002 59 7/15/2024
0.18.5 78 7/15/2024
0.18.5-pullrequest0134-0003 56 7/15/2024
0.18.4 80 7/15/2024
0.18.4-pullrequest0133-0009 61 7/15/2024
0.18.4-pullrequest0133-0008 57 7/15/2024
0.18.4-pullrequest0133-0007 61 7/15/2024
0.18.3 86 7/12/2024
0.18.3-pullrequest0132-0005 60 7/12/2024
0.18.2 79 7/12/2024
0.18.2-pullrequest0131-0003 68 7/11/2024
0.18.2-pullrequest0130-0003 61 7/11/2024
0.18.1 79 7/10/2024
0.18.1-pullrequest0129-0007 58 7/10/2024
0.18.0 124 7/10/2024
0.18.0-pullrequest0125-0014 60 7/10/2024
0.18.0-pullrequest0125-0013 67 7/10/2024
0.18.0-pullrequest0125-0012 55 7/10/2024
0.18.0-pullrequest0125-0011 58 7/10/2024
0.18.0-pullrequest0125-0010 61 7/10/2024
0.18.0-pullrequest0125-0009 71 7/10/2024
0.17.2-pullrequest0125-0008 62 7/10/2024
0.17.2-pullrequest0125-0007 66 7/10/2024
0.17.2-pullrequest0125-0006 62 7/10/2024
0.17.2-pullrequest0125-0005 64 7/10/2024
0.17.2-pullrequest0125-0004 66 7/10/2024
0.17.2-pullrequest0125-0003 63 7/9/2024
0.17.1 80 7/9/2024
0.17.1-pullrequest0124-0005 64 7/9/2024
0.17.0 85 7/8/2024
0.17.0-pullrequest0123-0024 52 7/8/2024
0.17.0-pullrequest0123-0023 53 7/8/2024
0.17.0-pullrequest0123-0021 68 7/8/2024
0.17.0-pullrequest0123-0019 59 7/8/2024
0.17.0-pullrequest0123-0018 51 7/8/2024
0.17.0-pullrequest0123-0017 63 7/8/2024
0.17.0-pullrequest0123-0015 59 7/8/2024
0.17.0-pullrequest0123-0014 55 7/8/2024
0.17.0-pullrequest0123-0013 60 7/8/2024
0.17.0-pullrequest0123-0012 54 7/8/2024
0.17.0-pullrequest0123-0011 62 7/8/2024
0.17.0-pullrequest0123-0009 64 7/6/2024
0.17.0-pullrequest0123-0008 58 7/6/2024
0.17.0-pullrequest0123-0007 98 7/6/2024
0.16.23 83 6/12/2024
0.16.23-pullrequest0118-0005 67 6/12/2024
0.16.23-pullrequest0118-0004 59 6/12/2024
0.16.23-pullrequest0118-0003 60 6/12/2024
0.16.23-pullrequest0118-0002 59 6/12/2024
0.16.21 140 5/29/2024
0.16.21-pullrequest0114-0002 65 5/29/2024
0.16.20 86 5/28/2024
0.16.20-pullrequest0112-0002 59 5/28/2024
0.16.19 91 5/13/2024
0.16.19-pullrequest0110-0004 74 5/13/2024
0.16.19-pullrequest0110-0003 63 5/13/2024
0.16.17 107 4/22/2024
0.16.17-pullrequest0108-0005 74 4/19/2024
0.16.17-pullrequest0108-0004 67 4/19/2024
0.16.17-pullrequest0108-0003 66 4/19/2024
0.16.16 99 4/16/2024
0.16.16-pullrequest0107-0003 80 4/16/2024
0.16.15 96 4/16/2024
0.16.15-pullrequest0106-0004 68 4/16/2024
0.16.15-pullrequest0106-0003 65 4/16/2024
0.16.15-pullrequest0106-0002 76 4/16/2024
0.16.14 157 4/16/2024
0.16.13 109 3/23/2024
0.16.13-pullrequest0103-0003 84 3/23/2024
0.16.12 93 3/23/2024
0.16.12-pullrequest0102-0006 74 3/22/2024
0.16.11 94 3/21/2024
0.16.11-pullrequest0101-0004 66 3/21/2024
0.16.11-pullrequest0101-0003 73 3/21/2024
0.16.11-pullrequest0101-0002 65 3/21/2024
0.16.9 100 3/16/2024
0.16.9-pullrequest0098-0008 69 3/16/2024
0.16.9-pullrequest0098-0007 72 3/16/2024
0.16.8 120 3/15/2024
0.16.8-pullrequest0097-0002 69 3/15/2024
0.16.7 96 3/10/2024
0.16.7-pullrequest0095-0002 79 3/10/2024
0.16.6 177 3/9/2024
0.16.6-pullrequest0094-0003 70 3/9/2024
0.16.6-pullrequest0094-0002 68 3/9/2024
0.16.2 105 3/5/2024
0.16.2-pullrequest0089-0009 76 3/5/2024
0.16.1 103 3/3/2024
0.16.1-pullrequest0088-0003 72 3/2/2024
0.16.1-pullrequest0088-0002 62 3/2/2024
0.16.0 147 2/28/2024
0.16.0-pullrequest0087-0010 64 2/28/2024
0.16.0-pullrequest0087-0009 68 2/28/2024
0.16.0-pullrequest0087-0007 78 2/28/2024
0.16.0-pullrequest0087-0006 68 2/28/2024
0.16.0-pullrequest0087-0005 64 2/28/2024
0.16.0-pullrequest0087-0004 75 2/28/2024
0.16.0-pullrequest0087-0003 67 2/28/2024
0.15.2 92 2/26/2024
0.15.2-pullrequest0086-0007 69 2/26/2024
0.15.2-pullrequest0086-0006 59 2/26/2024
0.15.1 100 2/23/2024
0.15.1-pullrequest0085-0006 69 2/23/2024
0.15.0 100 2/22/2024
0.15.0-pullrequest0084-0004 72 2/22/2024
0.15.0-pullrequest0084-0003 100 2/22/2024
0.14.1-pullrequest0084-0002 83 2/22/2024
0.14.0 111 2/21/2024
0.14.0-pullrequest0083-0011 93 2/8/2024
0.14.0-pullrequest0083-0010 74 2/8/2024
0.14.0-pullrequest0083-0009 75 2/8/2024
0.13.7 119 2/8/2024
0.13.7-pullrequest0082-0002 82 2/8/2024
0.13.6-pullrequest0081-0006 74 2/8/2024
0.13.5 125 2/7/2024
0.13.5-pullrequest0080-0002 75 2/7/2024
0.13.4 92 2/7/2024
0.13.4-pullrequest0079-0003 68 2/7/2024
0.13.1 102 2/1/2024
0.13.1-pullrequest0076-0003 76 2/1/2024
0.13.1-pullrequest0076-0002 76 2/1/2024
0.13.0 118 2/1/2024
0.13.0-pullrequest0075-0005 73 2/1/2024
0.13.0-pullrequest0075-0004 67 2/1/2024
0.12.2-pullrequest0075-0003 68 2/1/2024
0.12.1 94 2/1/2024
0.12.1-pullrequest0074-0002 71 2/1/2024
0.12.0 92 2/1/2024
0.12.0-pullrequest0073-0002 68 2/1/2024
0.11.0 93 1/31/2024
0.11.0-pullrequest0072-0011 72 1/31/2024
0.11.0-pullrequest0072-0010 70 1/31/2024
0.11.0-pullrequest0072-0008 69 1/31/2024
0.10.0 96 1/31/2024
0.10.0-pullrequest0070-0005 76 1/29/2024
0.10.0-pullrequest0070-0004 71 1/29/2024
0.9.1-pullrequest0069-0001 71 1/29/2024
0.9.0 90 1/29/2024
0.9.0-pullrequest0069-0002 68 1/29/2024
0.8.7-pullrequest0068-0001 71 1/29/2024
0.8.6 94 1/29/2024
0.8.4 112 1/19/2024
0.8.4-pullrequest0066-0009 72 1/19/2024
0.8.4-pullrequest0066-0008 63 1/19/2024
0.8.4-pullrequest0066-0007 70 1/19/2024
0.8.4-pullrequest0066-0005 66 1/19/2024
0.8.4-pullrequest0066-0004 74 1/19/2024
0.8.4-pullrequest0066-0003 68 1/19/2024
0.8.3 89 1/19/2024
0.8.3-pullrequest0065-0003 70 1/19/2024
0.8.3-pullrequest0065-0002 69 1/19/2024
0.8.1 92 1/19/2024
0.8.1-pullrequest0063-0013 68 1/19/2024
0.8.1-pullrequest0063-0011 72 1/19/2024
0.7.16-pullrequest0063-0010 77 1/19/2024
0.7.15 99 1/18/2024
0.7.15-pullrequest0062-0002 73 1/18/2024
0.7.12 106 1/16/2024
0.7.12-pullrequest0059-0006 76 1/16/2024
0.7.10 99 1/15/2024
0.7.10-pullrequest0059-0007 72 1/16/2024
0.7.10-pullrequest0058-0004 66 1/15/2024
0.7.10-pullrequest0057-0002 72 1/15/2024
0.7.9 99 1/15/2024
0.7.9-pullrequest0057-0003 67 1/15/2024
0.7.7 95 1/15/2024
0.7.7-pullrequest0056-0003 82 1/15/2024
0.7.4 98 1/15/2024
0.7.4-pullrequest0055-0012 75 1/15/2024
0.7.4-pullrequest0055-0011 75 1/15/2024
0.7.4-pullrequest0055-0010 80 1/15/2024
0.7.4-pullrequest0055-0009 74 1/15/2024
0.7.4-pullrequest0055-0008 73 1/15/2024
0.7.4-pullrequest0055-0007 77 1/15/2024
0.7.4-pullrequest0054-0005 76 1/15/2024
0.7.4-pullrequest0053-0004 88 1/15/2024
0.7.4-pullrequest0052-0002 69 1/15/2024
0.7.2 94 1/15/2024
0.7.2-pullrequest0051-0013 77 1/15/2024
0.7.2-pullrequest0050-0010 67 1/15/2024
0.7.1 105 1/13/2024
0.7.0 102 1/13/2024
0.7.0-pullrequest0046-0079 87 1/13/2024
0.6.12 122 1/3/2024
0.6.12-pullrequest0045-0018 73 1/13/2024
0.6.12-pullrequest0042-0010 76 1/12/2024
0.6.12-pullrequest0041-0007 75 1/12/2024
0.6.12-pullrequest0039-0002 81 1/3/2024
0.6.10 117 12/22/2023
0.6.10-pullrequest0038-0007 99 1/2/2024
0.6.10-pullrequest0038-0006 82 1/2/2024
0.6.10-pullrequest0038-0005 83 1/2/2024
0.6.8 112 12/21/2023
0.6.6 107 12/18/2023
0.6.6-pullrequest0035-0007 86 12/21/2023
0.6.6-pullrequest0035-0006 87 12/21/2023
0.6.6-pullrequest0035-0005 81 12/21/2023
0.6.3 106 12/18/2023
0.6.3-pullrequest0034-0018 85 12/18/2023
0.6.1 100 12/15/2023
0.6.1-pullrequest0029-0008 89 12/15/2023
0.6.1-pullrequest0028-0006 91 12/15/2023
0.6.1-pullrequest0027-0004 69 12/15/2023
0.5.3 109 12/11/2023
0.5.3-pullrequest0025-0049 89 12/14/2023
0.5.3-pullrequest0025-0047 86 12/14/2023
0.5.3-pullrequest0025-0043 81 12/13/2023
0.5.3-pullrequest0022-0038 82 12/13/2023
0.5.1 96 12/7/2023
0.5.0 102 12/6/2023
0.4.3 108 12/4/2023
0.4.2 99 12/4/2023
0.4.1 109 12/4/2023
0.3.10 117 11/22/2023
0.3.10-pullrequest0022-0008 83 11/27/2023
0.3.10-pullrequest0021-0005 86 11/22/2023
0.3.9 95 11/20/2023
0.3.8 105 11/9/2023
0.3.8-pullrequest0020-0005 81 11/9/2023
0.3.6 96 11/9/2023
0.3.5 103 11/9/2023
0.3.4 98 11/8/2023
0.3.3 100 11/8/2023
0.3.1 102 11/8/2023
0.3.1-pullrequest0018-0004 89 11/8/2023
0.2.1 136 11/2/2023
0.2.1-pullrequest0017-0050 98 11/8/2023
0.2.1-pullrequest0016-0048 100 11/8/2023
0.2.1-pullrequest0015-0045 89 11/8/2023
0.2.1-pullrequest0014-0035 99 11/7/2023
0.2.1-pullrequest0013-0033 90 11/7/2023
0.2.1-pullrequest0012-0031 95 11/7/2023
0.2.1-pullrequest0011-0024 93 11/6/2023
0.2.1-pullrequest0010-0015 106 11/2/2023
0.2.1-pullrequest0009-0011 96 11/2/2023
0.2.1-pullrequest0008-0009 97 11/2/2023
0.2.1-pullrequest0007-0007 104 11/2/2023
0.2.1-pullrequest0006-0005 99 11/2/2023
0.2.1-pullrequest0005-0001 90 11/2/2023
0.1.0 124 10/27/2023
0.1.0-pullrequest0004-0024 99 11/1/2023
0.1.0-pullrequest0003-0022 100 11/1/2023
0.1.0-pullrequest0002-0019 98 11/1/2023
0.1.0-pullrequest0001-0017 100 11/1/2023
0.1.0-pullrequest0001-0010 101 10/30/2023

releasenotes.md