PommaLabs.Thumbnailer.Client 5.0.0

Thumbnailer

Web service which exposes endpoints to generate file thumbnails and to optimize media files.

Thumbnailer heavily relies on many open source command line tools. Just to name a few:

• ffmpeg, used to generate video thumbnails and to optimize MP4 files.
• GraphicsMagick, used to generate image thumbnails.
• LibreOffice, used to convert Office files to images.
• PhantomJS, used to convert HTML files to images.
• pngquant, used to optimize PNG images and file thumbnails.

Web service runs on ASP.NET Core and this repository contains a .NET Standard client which can be used to interact with it.

Table of Contents

• Install
  • Tags
  • Configuration
  • Logging
  • Database
  • Persistence
    • Read-only
  • Templates
    • Fallback image
• Usage
  • Synchronous API (v1)
  • Asynchronous API (v2)
• Maintainers
• Contributing
  • Editing
  • Restoring dependencies
  • Starting development server
  • Running tests
  • Building Docker image
• License

Install

Thumbnailer web service is provided as a Docker image hosted on Docker Hub.

You can quickly start a local test instance with following command:

docker run -it --rm -e Security__AllowAnonymousAccess=true -p 8080:8080 pommalabs/thumbnailer:latest

Local test instance will be listening on port 8080 and it will accept unauthenticated requests. Please check the Configuration section to find further information about how the web service can be properly configured.

A .NET client, PommaLabs.Thumbnailer.Client, is available as a NuGet package:

dotnet add package PommaLabs.Thumbnailer.Client

Tags

Following tags are available:

• latest, based on pommalabs/dotnet:5-aspnet, contains the code found in main branch.
• azure-appsvc, based on pommalabs/dotnet:5-azure-appsvc, contains the code found in main branch. This tag is customized in order to support SSH connection exposed through Azure App Service.
• preview, based on pommalabs/dotnet:5-aspnet, contains the code found in preview branch.

Each tag is rebuilt every week.

Configuration

Docker image can be configured using the following environment variables:

Security__ApiKeysJson variable is a JSON array of API key objects; each object can have three properties:

• name, required, descriptive name.
• value, required, which will be used to perform authentication.
• expiresAt, optional, expiration timestamp. If not specified, API key will not expire.

An example value for Security__ApiKeysJson is the following one:

[{"name":"Expiring API key","value":"123","expiresAt":"2018-12-10T13:49:51.141Z"},{"name":"Not expiring API key","value":"456"}]

Please note that setting both Security__ApiKeysJson and single array item configurations (e.g. Security__ApiKeys__0__Name) is not supported and its behavior cannot be relied upon.

Logging

Web service writes log messages to the console.

If an Azure Application Insights connection string is specified using the ApplicationInsights__ConnectionString configuration, logs and telemetry data are also sent to that service.

Database

Setup scripts for supported systems are available:

• MySQL and MariaDB
• PostgreSQL
• SQL Server

You can customize those scripts according to your needs, but please remember that if you change schema name, you should update Database__SchemaName configuration property accordingly.

Persistence

When an external database is configured, it is also used by the web service to store cache entries using the KVLite library.

Cache entries store the SHA384 of input files and the parameters used; that data is linked to the location of the output file in Thumbnailer temporary directory (/tmp/thumbnailer). Temporary files stay in that directory for at least as long as it has been specified with Database__CacheLifetime configuration.

Moreover, distributed cache is also used to track job information for API v2. Job IDs are encrypted using local keys (/tmp/thumbnailer/data-protection-keys).

So, you might want to persist the temporary directory root, /tmp/thumbnailer, in order to be able to handle the following use cases:

• Web service receives a lot of requests and temporary files directory can grow quite a lot. In that case, a separate volume with proper size can be configured.
• Web service works in load balancing and it is configured to use an external DB. In that case, sharing the temporary files directory is required in order to let all instances be able to read cached outputs. Moreover, sharing the same encryption keys is necessary to be able to decrypt on one node data generated on another node.

However, persisting that directory is not mandatory at all. Web service works fine without having it persisted, especially when only one node is used.

Read-only

A special note on persistence applies when Docker container is run in read-only mode. In that scenario, following directories need to be mounted on writeable storage in order for web service to properly work:

• /home/app, where LibreOffice writes configuration files at runtime.
• /tmp, used by web service itself (/tmp/thumbnailer) and by .NET runtime.

Templates

Web service stores its templates in a dedicated directory, /opt/app/Templates. Currently, that directory contains only one template, fallback.svg, which is used to generate fallback images when thumbnail generation fails.

If templates need to be customized, then templates directory can be mounted to a local directory using a bind mount, for example:

docker run -it --rm \
  -e Security__AllowAnonymousAccess=true -p 8080:8080 \
  -v /my/templates:/opt/app/Templates:ro \
  pommalabs/thumbnailer:latest

Source templates can be found on GitLab.

When templates are changed, web service should be restarted, because templates are loaded at startup.

Fallback image

Template for fallback image, stored in fallback.svg, is handled as a Liquid template using dotLiquid library. Liquid syntax can be used to apply filters and conditional logic.

Fallback image template receives a file object containing the following attributes:

• content_type, containing input file MIME type.
• extension, file extension deduced by Thumbnailer, which might differ from the one received as input. File extension includes the leading dot character.
• size, file size in bytes.
• humanized_size, file size converted into a readable string (e.g. "2.4 MB").

Usage

Please check OpenAPI docs on your instance (just visit the /swagger page) or head to the demo instance Swagger page. Each endpoint has been documented using the OpenAPI specification.

Media optimization supports following content types:

• image/gif
• image/jpeg
• image/png
• image/svg+xml
• image/webp
• video/mp4

Thumbnail generation supports quite a lot of content types. Anyway, you can use the test files as a quick reference to find out what the web service really supports. Inside each "format" directory you will find a copy of the latest thumbnails, that is, for each file you will find the most recent copy of its thumbnail.

However, not every content type for which a thumbnail could be "theoretically" generated is supported. Some notable formats which are not supported are the following:

• PSD files represent a security issue since GraphicsMagick support is not complete. As a matter of fact, their support in GraphicsMagick has been disabled.

For files whose thumbnail cannot be generated either because there is none, e.g. for archive files, or because it is not supported by the web service, a fallback flag can be set when requesting thumbnail generation. When enabled, a fallback image is generated when thumbnail generation fails or when it cannot be started at all. See Templates section to find out how to customize fallback images.

Synchronous API (v1)

Synchronous API is exposed through v1 endpoints and, as its name suggests, it allows to send synchronous HTTP requests for thumbnail generation or media generation, where the output file is provided as a response for each request.

This approach is quite simple but it is not recommended, because each HTTP request should have a very short timeout and that fact does not work well with the processing of large or complex files. Anyway, this kind of API was the first one to be implemented by Thumbnailer and it will be maintained in the future. There are use cases where a simple API is enough and it is preferred over a more complex one.

Synchronous requests timeout is controlled by Security__ProcessTimeout setting, as described in the Configuration section.

With a Thumbnailer instance running, as described in the Install section, you can generate the thumbnail of a sample image with the following request:

GET /api/v1/thumbnail?fileUri=https://via.placeholder.com/2048&widthPx=256&heightPx=256&shavePx=0&fill=true&smartCrop=false&responseType=Binary HTTP/1.1
Host: localhost:8080

That endpoint will generate a response containing the requested thumbnail.

Please inspect Swagger documentation of v1 endpoints in order to find further information about exposed operations. For example, input files can also be POSTed using a multipart/form-data request or with a JSON payload.

Asynchronous API (v2)

Asynchronous API is exposed through v2 endpoints and it allows to start processing jobs which can be monitored with a dedicated API. When a job is finished, its output will be available for download.

Asynchronous API is more complex to work with, because it requires at least three HTTP calls in order to obtain the operation output. However, the asynchronous approach is required when potentially large or complex files need to be processed.

Asynchronous jobs timeout is controlled by Security__AsyncProcessTimeout setting, as described in the Configuration section.

With a Thumbnailer instance running, as described in the Install section, you can start the generation of the thumbnail of a sample image with the following request:

GET /api/v2/thumbnail?fileUri=https://via.placeholder.com/2048&widthPx=256&heightPx=256&shavePx=0&fill=true&smartCrop=false HTTP/1.1
Host: localhost:8080

That endpoint will respond with a JSON payload containing the job ID:

{
    "failureReason": null,
    "jobId": "NEW_JOB_ID",
    "status": "Queued"
}

That job ID can be used to query the job status:

GET /api/v2/jobs/NEW_JOB_ID HTTP/1.1
Host: localhost:8080

When job status becomes Processed, as in sample below:

{
    "failureReason": null,
    "jobId": "NEW_JOB_ID",
    "status": "Processed"
}

Then job result is ready to be downloaded from following endpoint:

GET /api/v2/jobs/NEW_JOB_ID/download?responseType=Binary HTTP/1.1
Host: localhost:8080

That endpoint will generate a response containing the requested thumbnail.

Please inspect Swagger documentation of v2 endpoints in order to find further information about exposed operations. For example, input files can also be POSTed using a multipart/form-data request or with a JSON payload.

Maintainers

@pomma89.

Contributing

MRs accepted.

Small note: If editing the README, please conform to the standard-readme specification.

Editing

Visual Studio Code, with Remote Containers extension, is the recommended way to work on this project.

A development container has been configured with all required tools.

Visual Studio Community is also supported and an updated solution file, thumbnailer.sln, has been provided.

Restoring dependencies

When opening the development container, dependencies should be automatically restored.

Anyway, dependencies can be restored with following command:

dotnet restore

Starting development server

A local development server listening on port 8080 can be started with following command:

dotnet run --project ./src/PommaLabs.Thumbnailer/PommaLabs.Thumbnailer.csproj

Running tests

Tests can be run with following command:

dotnet test

Tests can also be run with following command, which collects coverage information:

./build.sh --target run-tests

Tests for a specific file format can be using the following command, where FORMAT should be the name of one of directories holding test files, written in upper case:

export FORMAT="PNG"
dotnet test --filter "Name~${FORMAT}_"

Tests can also be run again a web service instance. Following command should be adapted according to instance base URI and optional API key:

export THUMBNAILER_BASE_URI="http://localhost:8080"
export THUMBNAILER_API_KEY="123QWE"
dotnet test

Building Docker image

Docker image can be built with following command:

docker build . -f ./src/PommaLabs.Thumbnailer/Dockerfile.latest -t $DOCKER_TAG

Please replace $DOCKER_TAG with a valid tag (e.g. thumbnailer).

License

MIT © 2019-2022 Alessio Parma