WebReaper 3.0.4

WebReaper

Please star this project if you find it useful!

Overview

Declarative high performance web scraper in C#. Easily crawl any web site and parse the data, save structed result to a file, DB, etc.

Install

dotnet add WebReaper

Requirements

.NET 6

Tech stack:

<p align="left"> <img src="https://img.shields.io/badge/C%23-239120?style=for-the-badge&logo=c-sharp&logoColor=white" /> <img src="https://img.shields.io/badge/Azure_Functions-0062AD?style=for-the-badge&logo=azure-functions&logoColor=white" /> <img src="https://img.shields.io/badge/MongoDB-4EA94B?style=for-the-badge&logo=mongodb&logoColor=white" /> <img src="https://img.shields.io/badge/redis-%23DD0031.svg?&style=for-the-badge&logo=redis&logoColor=white" /> <img src="https://img.shields.io/badge/-Puppeteer-green?style=for-the-badge&logo=appveyor" /> </p>

📋 Example:

var webReaper = new Scraper("reddit")
    .WithStartUrl("https://www.reddit.com/r/dotnet/")
    .FollowLinks("a.SQnoC3ObvgnGjWt90zD9Z._2INHSNB8V5eaWp4P0rY_mE")
    .Parse(new Schema
    {
        new("title", "._eYtD2XCVieq6emjKBH3m"),
        new("text", "._3xX726aBn29LDbsDtzr_6E._1Ap4F5maDtT1E1YuCiaO0r.D3IL3FD0RFy_mkKLPwL4")
    })
    .WriteToJsonFile("output.json")
    .WithLogger(new ColorConsoleLogger())
    .Run(10);

Console.ReadLine();

Features:

• ⚡ It's extremly fast due to parallelism and asynchrony
• 🗒 Declarative parsing with a structured scheme
• 💾 Saving data to any sinks such as JSON or CSV file, MongoDB, CosmosDB, Redis, etc.
• 🌎 Distributed crawling support: run your web scraper on ony cloud VMs, serverless functions, on-prem servers, etc.
• 🐙 Crowling and parsing Single Page Applications with Puppeteer
• 🖥 Proxy support
• 🌀 Automatic reties

Usage examples

• Data mining
• Gathering data for machine learning
• Online price change monitoring and price comparison
• News aggregation
• Product review scraping (to watch the competition)
• Gathering real estate listings
• Tracking online presence and reputation
• Web mashup and web data integration
• MAP compliance
• Lead generation

API overview

SPA parsing example

Parsing single page applications is super simple, just specify PageType.Dynamic. In this case Puppeteer will be used to load the pages.

var webReaper = new Scraper("reddit")
    .WithStartUrl("https://www.reddit.com/r/dotnet/", PageType.Dynamic)
    .FollowLinks("a.SQnoC3ObvgnGjWt90zD9Z._2INHSNB8V5eaWp4P0rY_mE", PageType.Dynamic)
    .Parse(new Schema
    {
        new("title", "._eYtD2XCVieq6emjKBH3m"),
        new("text", "._3xX726aBn29LDbsDtzr_6E._1Ap4F5maDtT1E1YuCiaO0r.D3IL3FD0RFy_mkKLPwL4")
    })
    .WriteToJsonFile("output.json")
    .WithLogger(new ColorConsoleLogger())
    .Run(10);

Console.ReadLine();

Additionaly, you can run any JavaScript on dynamic pages as they are loaded with headless browser. In order to do that you need to pass the third parameter:

var webReaper = new Scraper("reddit")
    .WithStartUrl("https://www.reddit.com/r/dotnet/", PageType.Dynamic, "window.scrollTo(0, document.body.scrollHeight);")
    .FollowLinks("a.SQnoC3ObvgnGjWt90zD9Z._2INHSNB8V5eaWp4P0rY_mE", PageType.Dynamic, "window.scrollTo(0, document.body.scrollHeight);")
    .Parse(new Schema
    {
        new("title", "._eYtD2XCVieq6emjKBH3m"),
        new("text", "._3xX726aBn29LDbsDtzr_6E._1Ap4F5maDtT1E1YuCiaO0r.D3IL3FD0RFy_mkKLPwL4")
    })
    .WriteToJsonFile("output.json")
    .WithLogger(new ColorConsoleLogger())
    .Run(10);

It can be helpful if the required content is loaded only after some user interactions such as clicks, scrolls, etc.

Authorization

If you need to pass authorization before parsing the web site, you can call Authorize method on Scraper that has to return CookieContainer with all cookies required for authorization. You are responsible for performing the login operation with your credentials, the Scraper only uses the cookies that you provide.

scraper = new Scraper()
    .WithLogger(logger)
    .WithStartUrl("https://rutracker.org/forum/index.php?c=33")
    .Authorize(() =>
    {
        var container = new CookieContainer();
        container.Add(new Cookie("AuthToken", "123");
        return container;
    })

Distributed web scraping with Serverless approach

In the Examples folder you can find the project called WebReaper.AzureFuncs. It demonstrates the use of WebReaper with Azure Functions. It consists of two serverless functions:

StartScrapting

First of all, this function uses ScraperConfigBuilder to build the scraper configuration e. g.:

Secondly, this function writes the first web scraping job with startUrl to the Azure Service Bus queue:

WebReaperSpider

This Azure function is triggered by messages sent to the Azure Service Bus queue. Messages represent web scraping job.

Firstly, this function builds the spider that is going to execute the job from the queue.

Secondly, it executes the job by loading the page, parsing content, saving to the database, etc.

Finally, it iterates through these new jobs and sends them the the Job queue.

Extensibility

Adding a new sink to persist your data

Out of the box there are 4 sinks you can send your parsed data to: ConsoleSink, CsvFileSink, JsonFileSink, CosmosSink (Azure Cosmos database).

You can easly add your own by implementing the IScraperSink interface:

public interface IScraperSink
{
    public Task EmitAsync(JObject scrapedData);
}

Here is an example of the Console sink:

public class ConsoleSink : IScraperSink
{
    public Task EmitAsync(JObject scrapedData)
    {
        Console.WriteLine($"{scrapedData.ToString()}");
        return Task.CompletedTask;
    }
}

The scrapedData parameter is JSON object that contains scraped data that you specified in your schema.

Adding your sink to the Scraper is simple, just call AddSink method on the Scraper:

scraper = new Scraper()
    .AddSink(new ConsoleSink());
    .WithStartUrl("https://rutracker.org/forum/index.php?c=33")
    .FollowLinks("#cf-33 .forumlink>a")
    .FollowLinks(".forumlink>a")
    .FollowLinks("a.torTopic", ".pg")
    .Parse(new Schema {
        new("name", "#topic-title"),
    });

For other ways to extend your functionality see the next section.

Intrefaces

Main entities

• Job - a record that represends a job for the spider
• LinkPathSelector - represents a selector for links to be crawled

Repository structure

Coming soon:

• Nuget package
• Azure functions for the distributed crawling
• Parsing lists
• Loading pages with headless browser and flexible SPA page manipulations (clicks, scrolls, etc)
• Proxy support
• Add flexible conditions for ignoring or allowing certain pages
• Breadth first traversal with priprity channels
• Save auth cookies to redis
• Rest API example for web scraping
• Sitemap crawling support
• Ports to NodeJS and Go

Features under consideration

• Saving logs to Seq
• Add LogTo method with Console and File support
• Site API support
• CRON for scheduling
• Request auto throttling
• Add bloom filter for revisiting same urls
• Improve architecture and refactor
• Subscribe to logs with lambda expression

See the LICENSE file for license rights and limitations (GNU GPLv3).