Walter 2024.11.14.1710

WALTER

Introducing the WALTER Framework: Workable Algorithms for Location-aware Transmission, Encryption Response. Designed for modern developers, WALTER is a groundbreaking suite of NuGet packages crafted for excellence in .NET Standard 2.0, 2.1, Core 3.1, and .NET 6, 7, 8, as well as C++ environments. Emphasizing 100% AoT support and reflection-free operations, this framework is the epitome of performance and stability.

Whether you're tackling networking, encryption, or secure communication, WALTER offers unparalleled efficiency and precision in processing, making it an essential tool for developers who prioritize speed and memory management in their applications.

About this Nuget Package

This NuGet package contains helpful extension methods for any .NET project targeting anything from the Web to MAUI, Native Windows, Linux, or Mac. Even though this NuGet package is not intended for direct use, it does come with some handy features. You can download the cmpiled help file as well as see code samples in github

Delphi Inside

The WALTER framework's integration with the Walter.Native.Wrapper NuGet package is a result of our intentional effort to leverage Delphi for native development excellence. This decision enables WALTER to provide developers with access to robust wrapper classes, facilitating direct utilization of native functionalities across various platforms. The capabilities we've built into Delphi are not by accident but the outcome of dedicated engineering to ensure future-proof encryption, top-tier performance, and security.

Cross-Platform by Design

This strategic integration means that when you use WALTER, you're also harnessing the power of native cross-platform binaries provided by Walter.Native.Wrapper. As a result, developers can enjoy the full spectrum of native functionalities without compromising on performance or cross-platform compatibility.

Seamless Inclusion in Deployments

The package automatically includes the appropriate native binary for each platform, ensuring your applications benefit from native performance enhancements and capabilities directly within .NET projects. This integration is tailored to each platform's needs:

• For Windows, the walternative.dll is included.
• In Android APK packages, WalterNative.so is bundled.
• For iOS, applications leverage WalterNative.dylib.
• On Linux, the framework utilizes libWalterNative.xso, a decision made to prevent the highjacking of .so files by the Android build process. It's important to note that the file name libWalterNative.xso is carefully chosen and should not be altered to libWalterNative.so, as the framework expects to link libWalterNative.xso from disk.

This seamless inclusion simplifies the development process, allowing you to focus on building robust applications while WALTER takes care of the underlying native integration.

Ahead-Of-Time (AOT) Compilation Compliant

The NuGet package can be used with projects that are trimming and use AoT. However, Newtonsoft Json is not AoT compliant as it uses reflection; therefore, Newtonsoft extension methods are excluded from AoT support.

Json Extensions

We offer helpful extension methods for when you need to process Large JSON files and can't load the whole file in memory as well as exception free processing of json files, strings and streams using the IsValidJson<T>() extension method

The JsonStreamReader<T> class in the Walter namespace is a powerful utility for deserializing JSON data from streams in .NET. This class provides a seamless way to read JSON content instance by instance, handling exceptions gracefully, and is compliant with various .NET versions including .NET Standard 2.0, 2.1, Core 3.1, and .NET 6, 7, 8. It's designed to work efficiently.

Usage

Cross Platform compatible SecureString

The SecureString class offers a robust solution for handling confidential strings and byte arrays securely within memory. Designed to safeguard sensitive information against reverse engineering and memory dumps, SecureString is ideal for a wide range of platforms, including Android, iOS, Unix, and Linux. By extending functionality from a native wrapper, it capitalizes on platform-specific memory protection mechanisms, without relying on the underlying platform features to secure memory.

Usage Example

This example demonstrates how to use SecureString to securely handle an API key required for web requests to a third-party service. The API key is loaded from a disk file, ensuring it's protected both in memory and when persisted.

// Load the API key from a file on disk securely
var apiKeyFilePath = Path.Combine("path", "to", "apiKey.txt");
var apiKeyFile = new FileInfo(apiKeyFilePath);
var apiKey = SecureString.LoadFromDisk(SecureMemoryProtection.Machine, apiKeyFile);

// Use the API key in a web request
// Note: Actual usage might depend on the specific mechanism provided by SecureString to convert it back to a usable format securely.
// This is a simplified illustration.
using (var client = new HttpClient())
{
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", apiKey.GetProtectedString());
    var response = await client.GetAsync("https://api.example.com/data");
    Console.WriteLine(response.Content.ReadAsStringAsync().Result);
}


The class is used for reading JSON data from a stream, with error handling mechanisms to capture any exceptions during the deserialization process.
### Process a file row by row so you do not have to have the whole file in memory while processing it.
```c#
using var stream = File.OpenRead("AppData\\LinkedIn.json");
var sr = new Walter.JsonStreamReader<TCPIPNetwork>(stream,TCPIPNetworkJsonContext.Default.TCPIPNetwork);
int i=0;
var result= new List<TCPIPNetwork>();
await foreach (TCPIPNetwork item in sr.ReadAsync())
{
    i++;
    result.Add(item)
    if(sr.Errors.Count>0)
    {
        _logger.LogError(sr.Errors[^1]
                        ,"File line {line}, failed so entry {i} is skipped as it failed due to {error}"
                        ,sr.Errors[^1].LineNumber,i+1,sr.Errors[^1].Message);
        
    }

}

The Importance of Secure<T> for Data Protection

In the digital age, safeguarding sensitive data is paramount. As applications extend across various platforms—mobile devices (iOS, Android), and desktop operating systems (Windows, Linux)—the need for robust data protection mechanisms grows. The Secure<T> class stands out as a vital tool for developers, designed to protect sensitive information from cyber threats, unauthorized access, and exploitation.

Cross-Platform Compatibility

Secure<T> offers a unified approach to data security, ensuring compatibility across different environments. This universality makes it an ideal solution for applications running on a variety of platforms, providing a consistent method to secure sensitive data in memory.

Protection Against Common Vulnerabilities

Sensitive data, when stored in plain text or insufficiently protected, is vulnerable to reverse engineering and memory dumps. These techniques, frequently employed by attackers, can compromise data integrity and confidentiality. Secure<T> is engineered to mitigate these risks by leveraging advanced encryption and memory protection mechanisms.

Ideal for Sensitive Data

Storing confidential information—user credentials, personal details, financial records, or proprietary business data—demands stringent security measures. Secure<T> excels in this regard, enabling developers to encrypt and securely manage such data. It ensures that sensitive information is not only protected on disk but also kept secure when loaded into memory.

Read-Only Memory Protection

Secure<T> uniquely offers a mechanism to maintain a read-only version of sensitive data in memory. This approach minimizes the risk of unintended modifications or leaks, further enhancing the security posture of applications that handle critical data.

Use Case: Mobile and Desktop Applications

Whether developing for mobile devices or desktops, Secure<T> provides a scalable and effective solution to protect sensitive data. It is particularly beneficial for applications that require the highest level of security, such as financial apps, health care systems, and personal data managers.

In conclusion, the Secure<T> class is a cornerstone of modern data protection strategies, ensuring sensitive information remains secure across all platforms and use cases. Its ability to safeguard data against common vulnerabilities, coupled with cross-platform compatibility and read-only memory protection, makes it an indispensable tool for developers committed to data security.

Process data from a webservice

The sample shows fetching a JSON stream from a web service streaming endpoint and processing the data row by row:

using System.Net.Http;
using System.IO;
using System.Threading.Tasks;

// ... other necessary namespaces

public async Task ProcessJsonDataFromWebServiceAsync()
{
    using var httpClient = new HttpClient();
    try
    {
        var response = await httpClient.GetStreamAsync("https://example.com/api/datastream");
        
        var sr = new Walter.JsonStreamReader<TCPIPNetwork>(response, TCPIPNetworkJsonContext.Default.TCPIPNetwork);
        int i = 0;
        await foreach (TCPIPNetwork item in sr.ReadAsync())
        {
            i++;
            // Process each item
            // ...

            if (sr.Errors.Count > 0)
            {
                _logger.LogError(sr.Errors[i - 1],
                                 "File line {line}, failed so entry {i} is skipped as it failed due to {error}",
                                 sr.Errors[i - 1].LineNumber, i, sr.Errors[i - 1].Message);
            }
        }
    }
    catch (HttpRequestException e)
    {
        _logger.LogError("Error fetching data from web service: {Message}", e.Message);
    }
}

Read a file in one time and process all entries skip those that would fail

using var stream = File.OpenRead("TestData\\LinkedIn.json");
var sr = new Walter.JsonStreamReader<List<TCPIPNetwork>>(stream,TCPIPNetworkListJsonContext.Default.ListTCPIPNetwork);
var list = sr.Read();

foreach (var item in list)
{
 ...
}

String extension methods

This NuGet package includes a range of useful extension methods, enhancing functionality for JSON processing, conversions between hex and bytes, and handling SQL Server varbinary types. For additional examples and comprehensive documentation, you can visit the GitHub repository.

ToSqlBinaryString() integration of Encrypted Data into T-SQL Command

The encryptedBytes.ToSqlBinaryString() method is utilized to convert the byte array (which is the result of the encryption process) into a SQL Server-friendly varbinary format. This format is compatible with T-SQL syntax and is necessary for correctly storing the encrypted data in the database.

The resulting string, prefixed with 0x, represents the hexadecimal representation of the encrypted data. This string is directly embedded into the T-SQL command, allowing the encrypted data to be used in database operations such as INSERT or UPDATE statements. In this way, the encrypted data can be stored securely in the database while maintaining the ability to perform standard SQL operations on it.

var encryptedDataString = encryptedBytes.ToSqlBinaryString();
var tsql = @$"
    DECLARE @EncryptedData VARBINARY(64) = {encryptedDataString};
    ...
    INSERT INTO MyTable (EncryptedColumn) VALUES (@EncryptedData);
";

Secure encryption and decryption

Sample Test Cases for Understanding the Deterministic Encryption

Deterministic encryption is a method where identical plain text values are always encrypted into identical cipher text. This approach is particularly useful in scenarios where you need to:

• Store Sensitive Data on Third-party Servers: Especially relevant under GDPR compliance when using cloud services where you don't have control over the underlying hardware or database encryption mechanisms.
• Enable Grouping and Searching: Deterministic encryption allows for the encrypted data to be searchable and groupable, which is essential for performing database operations without decrypting the data, thus maintaining security and privacy.

Benefits and Considerations

• Data Privacy and Security: Ensures that sensitive data, such as personal information or proprietary corporate data, is stored securely, even in environments not directly controlled by your organization.
• Searchability and Operational Efficiency: Unlike other forms of encryption, deterministic encryption allows for efficient database operations such as indexing, searching, and grouping on encrypted data.
• Compliance with Regulations: Meets the requirements of regulations like GDPR, which mandate the protection of personal data, especially when processed or stored on external or cloud-based systems.

When to Use

• GDPR Compliance in Cloud-Based Storage: Ideal for scenarios requiring GDPR compliance while using cloud-based databases.
• Maintaining Operational Capabilities: When the ability to search or group data directly in the database is necessary for operational efficiency.

Important Considerations

• Not a One-Size-Fits-All Solution: Deterministic encryption is a tool among many in the security engineer's toolkit and should be used judiciously. It is not always the preferred method of encryption but can be effective for specific use cases.
• Secure Key Management: The security of deterministic encryption heavily relies on how the encryption keys are managed. It is crucial to ensure these keys are stored and handled securely, separate from the data they encrypt.

Deterministic encryption provides a balance between operational functionality and data security, making it a valuable option for specific use cases in cloud-based applications and services, particularly where GDPR compliance is a concern.

// Sample to demonstrate GDPR-compliant encryption of sensitive data using deterministic encryption
// for storage in a third-party hosted SQL server.

// Define the company name to be encrypted.
string companyName = "Undefined Corp";

// Create an instance of the symmetric encryption service with a secure password and salt.
// Note: In a production environment, securely manage the password and salt, avoiding hardcoded values.
var encryptionService = new Walter.Cypher.DeterministicEncryption(
    password: "My $ectet Pa$w0rd",
    salt: "123456789+*ç%&/"
);

// Encrypt the company name into a byte array.
byte[] encryptedBytes = encryptionService.Encrypt(companyName.ToBytes());

// Prepare the T-SQL command for data insertion, using the encrypted company name.
var tsql = @$"
DECLARE @UndefinedCorp VARBINARY(64) = {encryptedBytes.ToSqlBinaryString()};
DECLARE @checksum int = CHECKSUM(@UndefinedCorp);

// Check for the existence of the company and insert if not present.
if not exists(select * from [dbo].[Companies] where [CompanyName] = @UndefinedCorp and [cs_CompanyName] = @checksum)
BEGIN
    INSERT [dbo].[Companies] ([CompanyName],[cs_CompanyName],[TrueUpDays],[AutoInvoice],[ApplicableLicenseExcempt])
    Values(@UndefinedCorp, @checksum, -1, 0, 1);
END
";

// Execute the T-SQL command to store the encrypted data.
using var con = new SqlConnection(config.GetConnectionString("Billing"));
using var cmd = con.CreateCommand();
cmd.CommandText = tsql;
cmd.CommandType = System.Data.CommandType.Text;
con.Open();
cmd.ExecuteNonQuery();

Sample Test Cases for Understanding the Symmetric Encryption

We've included sample test cases in our codebase to demonstrate the functionality of the Symmetric Encryption process. These samples are crafted not just to test the code, but also to serve as practical examples for those looking to understand the encryption and decryption mechanisms in depth.

Why Test Cases?

• Hands-On Learning: By setting breakpoints and stepping through these tests, you can gain a hands-on understanding of how the encryption and decryption process works.
• Debugging and Inspection: It's an excellent opportunity to inspect the flow of data, observe how the encryption algorithm behaves, and understand how different components interact.
• Real-World Examples: These tests are more than theoretical scenarios; they represent real-world use cases, helping you relate the functionality to practical applications.

What's in the Sample?

• Encryption Consistency Test: Encrypt_WithSamePassword_ShouldGenerateDifferentCiphertexts ensures that the encryption process is secure and generates different ciphertexts for the same plaintext.
• Cross-Instance Compatibility Test: EncryptAndDecrypt_WithDifferentInstances_ShouldBeCompatible confirms that the encrypted data by one instance can be decrypted by another, ensuring consistency across different instances.

How to Use the Sample

Understood, let's adjust the instructions to focus on using the NuGet package directly without the need for cloning a repository. Here's the revised section for your README.md:

How to Use the Sample

To effectively use and understand the Symmetric Encryption examples in the 'walter' NuGet package, follow these steps:

1. Install the 'walter' NuGet Package: Start by adding the 'walter' package to your C# project. This package is essential as it contains the components you'll need for the encryption examples.
2. Navigate to your Tests: Copy and past the test cases in the project's test project.
3. Set Breakpoints: Place breakpoints at critical points in the tests.
4. Debug and Step Through: Run the tests in debug mode and step through the code to observe how the encryption process is executed and validated.

We encourage you to explore these tests to deepen your understanding of symmetric encryption in a .NET environment.

[TestClass]
public class SymmetricEncryptionTests
{
    // This test verifies that the same text encrypted with the same password generates different byte arrays.
    // This is important to ensure that the encryption algorithm uses a unique initialization vector (IV) for each encryption,
    // which enhances security by producing different ciphertexts for the same plaintext.
    [TestMethod]
    public void Encrypt_WithSamePassword_ShouldGenerateDifferentCiphertexts()
    {
        var secretText = "Hello World";
        var encryptionInstance1 = new SymmetricEncryption("TestPassword");
        var encryptionInstance2 = new SymmetricEncryption("TestPassword");

        byte[] encryptedBytes1 = encryptionInstance1.Encrypt(Encoding.UTF8.GetBytes(secretText));
        byte[] encryptedBytes2 = encryptionInstance2.Encrypt(Encoding.UTF8.GetBytes(secretText));

        string ciphertext1 = Encoding.UTF8.GetString(encryptedBytes1);
        string ciphertext2 = Encoding.UTF8.GetString(encryptedBytes2);

        Assert.AreNotEqual(ciphertext1, ciphertext2, "Encrypted bytes should be different for the same input text.");

        string decryptedText1 = Encoding.UTF8.GetString(encryptionInstance1.Decrypt(encryptedBytes1));
        string decryptedText2 = Encoding.UTF8.GetString(encryptionInstance2.Decrypt(encryptedBytes2));

        Assert.AreEqual(decryptedText1, decryptedText2, "Decrypted texts should match the original secret text.");
    }

    // This test ensures that text encrypted by one instance of the SymmetricEncryption class
    // can be decrypted by another instance using the same password. This is crucial for verifying
    // that the encryption and decryption processes are compatible and consistent across different instances.
    [TestMethod]
    public void EncryptAndDecrypt_WithDifferentInstances_ShouldBeCompatible()
    {
        var secretText = "Hello World";
        var encryptionInstanceClient = new SymmetricEncryption("TestPassword");
        var encryptionInstanceServer = new SymmetricEncryption("TestPassword");

        string ciphertext = encryptionInstanceClient.EncryptString(secretText);
        string decryptedText = encryptionInstanceServer.DecryptString(ciphertext);

        Assert.AreEqual(secretText, decryptedText, "Decrypted text should match the original secret text.");
    }
}

Extension methods

There are several extension methods that are usfull like ToBytes() and ToSqlBinaryString() as shown in the bellow code sample.

In the sample bellow we show how you could pre-populat database defaults in a GDPR compliant way where you can use the framweork to generate predefined standard values.

var corp = "Undefined Corp";
var cypher = new Walter.Cypher.SymmetricEncryption(password: "My $ectet Pa$w0rd"
                                             , padding: System.Security.Cryptography.PaddingMode.PKCS7
                                             );

byte[] bytes = cypher.Encrypt(corp.ToBytes());
var tsql = @$"
    declare @UndefinedCorp Varbinary(64) = {bytes.ToSqlBinaryString()};
    declare @checksum int = CHECKSUM(@UndefinedCorp);

    if not exists(select * from  [dbo].[Companies] where [CompanyName] =@UndefinedCorp and [cs_CompanyName]= @checksum)
    BEGIN
        INSERT [dbo].[Companies] ([CompanyName],[cs_CompanyName],[TrueUpDays],[AutoInvoice])
        Values(@UndefinedCorp,@checksum,-1,0);
    END
";


using var con = new SqlConnection(config.GetConnectionString("Billing"));
using var cmd = con.CreateCommand();
cmd.CommandText = tsql;
cmd.CommandType = System.Data.CommandType.Text;
con.Open();
cmd.ExecuteNonQuery();

AoT Exceptions Extension for .NET

The ExceptionsExtension class is a powerful utility for .NET developers, enhancing exception handling with additional diagnostic information. This extension provides methods to extract class names, method names, file names, and approximate line numbers from exceptions especially usefull in AoT.

Features

• ClassName: Retrieves the class name where the exception originated.
• MethodName: Obtains the method name that generated the exception.
• FileName: Gets the filename of the class that generated the exception.
• CodeLineNumber: Provides the actual or approximate line number where the exception was thrown.

Usage

Here is a examples of how to use the ExceptionsExtension method:

try
{
    _ = File.Open("A:\\doesNotExist.txt", FileMode.Open);
}
catch (Exception e)
{
    //if the binary is AoT compiled the line number is this line
    _logger.LogError(e, "{Class}.{method} (line {line}) failed with a {exception}:{message}",e.ClassName(), e.MethodName(), e.CodeLineNumber(),e.GetType().Name,e.Message);
}

ThrottleExtensions Class

The ThrottleExtensions class provides extension methods for managing throttled HTTP requests at the application level. This is especially useful when interacting with APIs that impose rate limits on the number of requests within a specific time window. With ThrottleExtensions, you can ensure that requests sent to the same domain or endpoint do not exceed the allowed rate by delaying requests when necessary.

The throttling mechanism applies globally across the application, meaning that requests from different parts of your application, even if they're being made from different threads, are synchronized to respect the same rate limits. This helps prevent overloading an API with too many requests, avoiding issues like 429 Too Many Requests errors.

In this example, we demonstrate how to use ThrottleExtensions to throttle requests based on both domain and API key. Two separate API keys are used for the same domain, and the requests are throttled independently for each key.

Example: Throttling Requests by Domain

using var client = new HttpClient();
string url = "https://api.example.com/data";

// Send a GET request with a rate limit of 5 requests per 10 seconds
var response = await client.GetAsync(url, 5, TimeSpan.FromSeconds(10));

if (response.IsSuccessStatusCode)
{
    var content = await response.Content.ReadAsStringAsync();
    Console.WriteLine(content);
}
else
{
    Console.WriteLine($"Request failed with status code {response.StatusCode}");
}

Example: Throttling Requests by Domain and API Key

using System;
using System.Net.Http;
using System.Threading.Tasks;

public class ApiRequestExample
{
    private static readonly HttpClient _client = new HttpClient();

    public static async Task Main(string[] args)
    {
        string domain = "api.example.com";
        string apiKey1 = "API_KEY_1";
        string apiKey2 = "API_KEY_2";

        string url = $"https://{domain}/data";

        // Send requests with different API keys concurrently, but throttle based on both domain and API key
        Task requestWithApiKey1 = SendThrottledRequest(url, domain, apiKey1, 5, TimeSpan.FromSeconds(10));
        Task requestWithApiKey2 = SendThrottledRequest(url, domain, apiKey2, 5, TimeSpan.FromSeconds(10));

        await Task.WhenAll(requestWithApiKey1, requestWithApiKey2);
    }

    private static async Task SendThrottledRequest(string url, string domain, string apiKey, int requests, TimeSpan per)
    {
        // Combine the domain and API key to create a unique throttling key
        string throttlingKey = $"{domain}:{apiKey}";

        // Throttle requests based on the domain and API key combination
        await throttlingKey.Throttle(requests, per);

        // Make the actual HTTP request with the API key
        using var requestMessage = new HttpRequestMessage(HttpMethod.Get, url);
        requestMessage.Headers.Add("Authorization", $"Bearer {apiKey}");

        var response = await _client.SendAsync(requestMessage);

        if (response.IsSuccessStatusCode)
        {
            string content = await response.Content.ReadAsStringAsync();
            Console.WriteLine($"Response with API key {apiKey}: {content}");
        }
        else
        {
            Console.WriteLine($"Request with API key {apiKey} failed: {response.StatusCode}");
        }
    }
}

Visit www.asp-waf.com for more information.