ExternalSort 1.0.0-beta001

.NET 8.0 This package targets .NET 8.0. The package is compatible with this framework or higher.
This is a prerelease version of ExternalSort.
There is a newer version of this package available.
See the version list below for details. 
dotnet add package ExternalSort --version 1.0.0-beta001
                    


                    

                
NuGet\Install-Package ExternalSort -Version 1.0.0-beta001
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="ExternalSort" Version="1.0.0-beta001" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="ExternalSort" Version="1.0.0-beta001" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="ExternalSort" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add ExternalSort --version 1.0.0-beta001
                    


                    

                
#r "nuget: ExternalSort, 1.0.0-beta001"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#addin nuget:?package=ExternalSort&version=1.0.0-beta001&prerelease
                    


                            Install ExternalSort as a Cake Addin
                        

                    

                
#tool nuget:?package=ExternalSort&version=1.0.0-beta001&prerelease
                    


                            Install ExternalSort as a Cake Tool

External Sort

Sorts data (IAsyncEnumerable<T>) that is larger than the amount of RAM available.

basic example

  IAsyncEnumerable<User> myUsersToSort = ...
  var sortedUsers = myUsersToSort
            .OrderByExternal(u => u.Email);

Behaviour

  • Unstable sort - the rows that 'tie' will be returned in an undefined way
  • O(n log n) - best case performance when the data fits in RAM. The more RAM you give it, the faster it will sort. More performance details here https://en.wikipedia.org/wiki/External_sorting
  • No disk activity when data fits in RAM

Options / Features

Sort Descending

  IAsyncEnumerable<User> myUsersToSort = ...
  var sortedUsers = myUsersToSort
            .OrderByDescendingExternal(u => u.Email);

Sort on multiple properties

  IAsyncEnumerable<User> myUsersToSort = ...
  var sortedUsers = myUsersToSort
            .OrderByExternal(u => u.Email)
            .ThenBy(u => u.FirstName)
            .ThenByDescending(u => u.UserId);

Control the amount of RAM used

  IAsyncEnumerable<User> myUsersToSort = ...
  var sortedUsers = myUsersToSort
            .OrderByExternal(u => u.Email)
            .OptimiseFor(calculateBytesInRam: u => u.CalculateBytesInRam(), mbLimit: 1_000, openFilesLimit: 10);
  • calculateBytesInRam - Calculates how much ram the row occupies in RAM. Not calculated for every row, but enough to get an average. Default = 300 bytes
  • mbLimit - The rough limit how much RAM to use to sort the data. In practice this is not exact, but it will never be unlimited. Default = 200 MB
  • openFilesLimit - The number of files open at one time. A larger number can reduce the number of files needed to sort, but then each file has a smaller buffer and can become too "chatty" with the disk. Buffer size for one temp file = mbLimit / openFileLimit. Default = 10

Internally, ExternalSort uses Parquet temp files. See https://github.com/aloneguid/parquet-dotnet for class serialisation options.

Thanks to https://josef.codes/sorting-really-large-files-with-c-sharp/ for the inspiration.

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 was computed.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
2.0.0-beta02 2,986 7/4/2024
2.0.0-beta01 105 7/4/2024
1.2.0 159 5/15/2024
1.2.0-beta001 833 4/15/2024
1.1.0 148 4/14/2024
1.1.0-beta003 188 4/11/2024
1.1.0-beta002 98 4/11/2024
1.1.0-beta001 125 4/10/2024
1.0.0 130 4/9/2024
1.0.0-beta005 158 4/8/2024
1.0.0-beta004 95 4/8/2024
1.0.0-beta003 131 4/8/2024
1.0.0-beta002 114 4/7/2024
1.0.0-beta001 108 4/4/2024