Deckard 0.0.31

Deckard: A Highly Scalable Cyclic Priority Queue

Deckard is a priority queue system inspired by projects such as Google Cloud PubSub, Nats, Kafka, and others. Its main distinction lies in its ability to associate a priority score with each message and have a queue that can be optionally cyclic. This means that messages can be delivered again after a user-managed time, allowing throughput configuration and many other features, all built using GoLang and gRPC to be as scalable as possible.

Briefly:

• An application inserts a message to be queued and its configuration (TTL, metadata, payload, priority score, etc).
  • The message will be prioritized with a default timestamp-based algorithm if the provided score is 0 (the default value).
• A worker application pull messages from Deckard at regular intervals and performs any processing.
  • The worker can also send a score filter (max score and min score) in the pull request. This enables theoughput controlling with score algorithms and many other features.
  • When it finishes processing a message, the application must notify with the processing result.
  • When notifying, the application may also provide a lock time, to lock the message for a certain duration of time before being requeued and delivered again. Locking mechanism have a 1-second precision.
  • It is also possible to notify a message without lock but changing its priority score to be delivered again.
• When the message's TTL is reached, it stops being delivered;
  • For some use cases the TTL can be set as infinite.
  • An application can also remove the message when notifying.

Motivation

Deckard was born from an initiative to simplify and unify applications called Frontiers which were part of STILINGUE's orchestration system for data gathering.

Several different implementations were made, but they proved to have limitations such as:

• Debugging
• Genericity
• Scalability
• Auditability
• Observability
• Prioritization
• Developer friendliness

Deckard was created to solve these problems and to be used in any application that needs to queue data and prioritize it.

The main objectives of the project are:

• Generic: any data and application should be able to use Deckard to queue messages;
• Observable: it should be easy to visualize what happened in a request and easy to investigate problems with audit;
• Developer friendly: easy to understand and pleasant to use;
• Scalable: it should be possible to support thousands of requests per second and millions of messages;

What Deckard is not?

It is important to note that Deckard is not a conventional messaging/queue system. If your use case does not involve priority, cyclic queuing, or a locking mechanism, it is recommended to consider alternatives such as GCP PubSub, Kafka, RabbitMQ, Azure Service Bus, Amazon SQS, or other messaging systems.

Project Status

Deckard has been used in a production environment since 2019, handling billions of messages and thousands of requests per second. Since its initial internal release, Deckard has undergone significant improvements and enhancements.

The performance and reliability of Deckard are directly dependent on the storage and cache services. Therefore, it is crucial to configure MongoDB and Redis correctly to ensure optimal performance. Redis is responsible for storing all priority queues and has a significant impact on Deckard's performance, while MongoDB usage is simpler and has a lower impact on performance.

Currently, we have projects that utilize MongoDB in both a virtual machine environment and a Kubernetes environment. However, it's worth noting that we have had more extensive usage of MongoDB in a virtual machine (VM) environment compared to Kubernetes. When deploying Deckard, it is important to configure MongoDB properly, following MongoDB's recommendations for production environments (documented here: MongoDB Production Notes).

While the project is being released with a 0.0.x version due to necessary code modifications for open-sourcing, it has been thoroughly tested and proven to be reliable in production scenarios.

Please refer to our issues section for more details. Your feedback and suggestions are highly appreciated and can be shared in our discussions.

Getting Started

To quickly get started with Deckard, please consult our getting started guide, which provides step-by-step instructions. Additionally, we have provided client documentation for various programming languages to assist you in integrating Deckard into your projects.

You can also access our documentation on Deckard's components here. More details on the project structure can be found here.

Running Deckard

Here's a quick guide on how to run Deckard. You should check the getting started guide for more details.

On Linux you can run it from sources with:

make run

You can run it with Docker:

docker run --rm -p 8081:8081 blipai/deckard

By default for Docker and Linux it will use a memory storage and a memory cache engine.

To change the default configuration see the configuration section.

You can also run it in a Kubernetes cluster using Helm:

It will deploy a MongoDB for storage and a Redis for cache.

Check the chart values.yaml to see all available configurations.

helm repo add deckard https://takenet.github.io/deckard/
helm install deckard deckard/deckard

You may also download the latest release from the releases page and execute it.

Configuration

The current implementation delegates the configuration to environment variables.

All available environment variables are listed below:

Overall Configuration

gRPC Server Configuration

Values should be specified using time units such as 1s for seconds, 1m for minutes, 1h for hours.

The default values depends on the server implementation. For more information check these links:

• Keepalive configuration specification
• grpc-go/internal/transport/defaults.go

Cache Configuration

Storage Configuration

Housekeeper Configuration

Audit Configuration

Metrics Configuration

TLS Configuration

To learn more about gRPC TLS configuration please refer to gRPC Auth.

Contributing

We are always looking for new contributors to help us improve Deckard.

If you want to contribute to Deckard, please read our contributing guide which includes how to build, run and test Deckard and a complete description of our project structure.

License

Deckard is licensed under the MIT License.

Acknowledgments

We would like to thank the following people for their initial contributions building Deckard's first version:

• Lucas Soares: @lucasoares
• Gustavo Paiva: @paivagustavo
• Cézar Augusto: @cezar-tech
• Júnior Rhis: @juniorrhis