Spidey 5.0.72

There is a newer version of this package available.
See the version list below for details.
dotnet add package Spidey --version 5.0.72                
NuGet\Install-Package Spidey -Version 5.0.72                
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Spidey" Version="5.0.72" />                
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Spidey --version 5.0.72                
#r "nuget: Spidey, 5.0.72"                
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Spidey as a Cake Addin
#addin nuget:?package=Spidey&version=5.0.72

// Install Spidey as a Cake Tool
#tool nuget:?package=Spidey&version=5.0.72                

Spidey

.NET Publish

Library to help with crawling web content. Compatible with .Net Core and .Net Framework.

Setting up the Library

Spidey relies on Canister in order to hook itself up. In order for this to work, you must do the following at startup:

new ServiceCollection().AddCanisterModules();

The AddCanisterModules function is an extension method that registers it with the IoC container. When this is done, Spidey is ready to use.

Basic Usage

Spidey really boils down to using one class called Crawler:

ServiceCollection.AddSingleton(new Options
			{
			    ItemFound = FoundFile=>{}                                            //The callback method used when a new page is discovered.
			    Allow = new List<string> { "http://mywebsite", "http://mywebsite2" },    //Regexes of what sites/pages are allowed to be crawled.
			    FollowOnly = new List<string> { "..." },                                 //Regexes of pages to only follow links that are found on them.
			    Ignore = new List<string> { "..." },                                     //Regexes that the system will ignore when they are encountered.
			    StartLocations = new List<string> { "http://mywebsite", "http://mywebsite2" },    //Starting URLs for the crawler.
			    UrlReplacements = new Dictionary<string,string> {...}                    //When the system hits one of the keys in the dictionary, it will replace it with the value.
			});
			

Note that it is recommended that you actually register the Options object in your ServiceCollection and resolve the Crawler object from the service provider but it is not required. You can simply new up an instance of Crawler if you want. Anyway, the Options class has a number of properties, some of which are not displayed above such as NetworkCredentials, UseDefaultCredentials, and Proxy. The callback method is what will be called by the system once a link's info has been received and looks like this:

void CallbackMethod(ResultFile obj) { ... }

The library will handle parsing of links found within the page, downloading the content, etc. for the most part. At this point all you have to do is call the StartCrawl method:

MyCrawler.StartCrawl();

Customization

Note that it's possible to customize the crawler's various parts. The system is divided into the following sections:

  1. Content Parser (IContentParser) - This parses the resulting data and converts it to the ResultFile object.
  2. Engine (IEngine) - This downloads the content from the server.
  3. Link Discoverer (ILinkDiscoverer) - Takes the content from the engine and looks for links to other resources.
  4. Processor (IProcessor) - Takes the parsed content and hands it off to your code. The default one simply calls the method provided in the options.
  5. Scheduler (IScheduler) - Handles handing out work to the various workers.
  6. Pipeline (IPipeline) - Manages the various parts of the process by feeding the content to the next bit of the process.

These subsystems all implement interfaces found in the Spidey.Engines.Interfaces namespace. In order to replace the default in any of these systems all you need to do is create a class that implements the interface that you want to replace. After that the system will automatically pick it up if resolved from the service provider. If you, instead, new up a Crawler object then you will need to compose the Pipeline object.

Installation

The library is available via Nuget with the package name "Spidey". To install it run the following command in the Package Manager Console:

Install-Package Spidey

FAQ

  1. Is it possible to run the crawler using multiple nodes?

The default scheduler assumes that you are only running the crawler from one location and doesn't talk to other instances of the application. But it is possible to replace the scheduler with one that will talk via some mechanism like a database to coordinate work between instances and is recommended for more complex setups.

Build Process

In order to build the library you will require the following:

  1. Visual Studio 2019

Other than that, just clone the project and you should be able to load the solution and build without too much effort.

Product Compatible and additional computed target framework versions.
.NET net6.0 is compatible.  net6.0-android was computed.  net6.0-ios was computed.  net6.0-maccatalyst was computed.  net6.0-macos was computed.  net6.0-tvos was computed.  net6.0-windows was computed.  net7.0 was computed.  net7.0-android was computed.  net7.0-ios was computed.  net7.0-maccatalyst was computed.  net7.0-macos was computed.  net7.0-tvos was computed.  net7.0-windows was computed.  net8.0 was computed.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
5.0.131 82 11/12/2024
5.0.130 74 11/11/2024
5.0.129 76 11/6/2024
5.0.128 78 11/5/2024
5.0.127 70 11/4/2024
5.0.126 77 10/31/2024
5.0.125 66 10/30/2024
5.0.124 75 10/29/2024
5.0.123 87 10/11/2024
5.0.122 81 10/10/2024
5.0.121 73 10/9/2024
5.0.120 85 10/2/2024
5.0.119 87 10/1/2024
5.0.118 84 9/24/2024
5.0.117 96 9/17/2024
5.0.116 135 9/10/2024
5.0.115 94 9/3/2024
5.0.114 95 8/30/2024
5.0.113 102 8/27/2024
5.0.112 100 8/26/2024
5.0.111 121 8/23/2024
5.0.110 112 8/21/2024
5.0.109 106 8/20/2024
5.0.108 110 8/16/2024
5.0.107 110 8/15/2024
5.0.106 92 8/5/2024
5.0.105 72 8/2/2024
5.0.104 86 8/1/2024
5.0.103 85 7/26/2024
5.0.102 102 7/11/2024
5.0.101 101 7/2/2024
5.0.100 106 6/27/2024
5.0.99 84 6/26/2024
5.0.98 112 6/19/2024
5.0.97 102 6/18/2024
5.0.96 111 6/17/2024
5.0.95 109 6/14/2024
5.0.94 96 6/13/2024
5.0.93 95 6/12/2024
5.0.92 108 5/31/2024
5.0.91 98 5/30/2024
5.0.90 98 5/17/2024
5.0.89 103 5/16/2024
5.0.88 125 5/8/2024
5.0.87 123 5/7/2024
5.0.86 126 5/6/2024
5.0.85 85 5/3/2024
5.0.84 93 5/2/2024
5.0.83 104 5/1/2024
5.0.82 125 4/30/2024
5.0.81 109 4/16/2024
5.0.80 108 4/12/2024
5.0.79 107 4/11/2024
5.0.78 129 4/1/2024
5.0.77 118 3/29/2024
5.0.76 120 3/18/2024
5.0.75 113 3/15/2024
5.0.74 117 3/14/2024
5.0.73 118 3/11/2024
5.0.72 109 3/8/2024
5.0.71 113 3/7/2024
5.0.70 133 3/6/2024
5.0.69 122 3/5/2024
5.0.68 123 3/4/2024
5.0.67 120 2/29/2024
5.0.66 110 2/28/2024
5.0.65 120 2/26/2024
5.0.64 119 2/23/2024
5.0.63 124 2/22/2024
5.0.62 126 2/21/2024
5.0.61 116 2/16/2024
5.0.60 115 2/15/2024
5.0.59 122 2/12/2024
5.0.58 117 2/8/2024
5.0.57 110 2/7/2024
5.0.56 111 2/6/2024
5.0.55 103 2/1/2024
5.0.54 116 1/31/2024
5.0.53 112 1/30/2024
5.0.52 108 1/24/2024
5.0.51 111 1/23/2024
5.0.50 122 1/12/2024
5.0.49 122 1/11/2024
5.0.48 124 12/26/2023
5.0.47 117 12/22/2023
5.0.46 103 12/18/2023
5.0.45 98 12/15/2023
5.0.44 94 12/14/2023
5.0.43 110 12/13/2023
5.0.42 105 12/12/2023
5.0.41 151 11/24/2023
5.0.40 129 11/21/2023
5.0.39 113 11/20/2023
5.0.38 121 11/17/2023
5.0.37 113 11/16/2023
5.0.36 120 11/14/2023
5.0.35 121 11/8/2023
5.0.34 105 11/7/2023
5.0.33 124 11/6/2023
5.0.32 120 11/1/2023
5.0.31 114 10/31/2023
5.0.30 130 10/30/2023
5.0.29 125 10/26/2023
5.0.28 150 10/12/2023
5.0.27 149 10/5/2023
5.0.26 138 9/26/2023
5.0.25 115 9/20/2023
5.0.24 108 9/19/2023
5.0.23 147 9/18/2023
5.0.22 123 9/14/2023
5.0.21 118 9/13/2023
5.0.20 130 9/11/2023
5.0.19 125 9/7/2023
5.0.18 126 9/6/2023
5.0.17 122 9/5/2023
5.0.16 140 9/4/2023
5.0.15 128 9/1/2023
5.0.14 136 8/31/2023
5.0.13 150 8/30/2023
5.0.12 149 8/29/2023
5.0.11 132 8/25/2023
5.0.10 142 8/23/2023
5.0.9 131 8/18/2023
5.0.8 155 8/10/2023
5.0.7 154 8/8/2023
5.0.6 161 8/8/2023
5.0.5 175 8/7/2023
5.0.4 160 8/3/2023
5.0.3 175 7/26/2023
5.0.2 154 7/20/2023
5.0.1 164 7/14/2023
5.0.0 309 12/12/2022
4.0.5 542 6/10/2022
4.0.2 500 1/11/2022
4.0.1 456 7/19/2021
3.0.9 507 1/7/2021
3.0.7 602 9/13/2020
3.0.6 564 6/26/2020
3.0.5 542 6/26/2020
3.0.3 554 3/25/2020
3.0.2 603 3/1/2020
3.0.1 622 1/1/2020
3.0.0 593 12/23/2019
2.0.15 631 11/22/2019
2.0.14 584 11/22/2019
2.0.13 565 11/22/2019
2.0.12 564 11/21/2019
2.0.11 557 11/21/2019
2.0.10 572 11/21/2019
2.0.9 557 11/21/2019
2.0.8 730 3/3/2019
2.0.7 661 3/3/2019
2.0.6 679 3/3/2019
2.0.5 662 3/3/2019
2.0.4 712 2/7/2019
2.0.3 1,019 6/1/2018
2.0.2 1,005 5/22/2018
2.0.1 1,126 1/2/2018
1.0.12 996 11/2/2017
1.0.11 993 10/30/2017
1.0.10 992 10/26/2017
1.0.9 995 10/26/2017
1.0.8 1,022 10/26/2017
1.0.7 979 10/25/2017
1.0.6 964 10/25/2017
1.0.5 999 10/24/2017
1.0.4 946 10/24/2017
1.0.3 934 10/19/2017
1.0.2 1,069 10/18/2017
1.0.1 990 9/29/2017