Python
Kafka-based Job Queue for Python

KQ: Kafka-based Job Queue for Python

KQ (Kafka Queue) is a light-weight Python library which provides a simple API to queue and process jobs asynchronously in the background. It is backed by Apache Kafka and designed primarily for ease of use.

Requirements

• Apache Kafka 0.9+
• Python 2.7+ 3.4+ or 3.5+

Getting Started

First, ensure that your Kafka instance is up and running:

# This command is just an example
~$ ./kafka-server-start.sh -daemon server.properties

Let's say you want to run the following function asynchronously:

import time

def my_func(foo, bar, baz=None):
    """This is a blocking function."""
    time.sleep(10)
    return foo, bar, baz

Start a KQ worker:

~$ kq worker --verbose
[INFO] Starting Worker(topic=foobar) ...

Enqueue the function call as a job:

# Import the blocking function
from my_module import my_func

# Initialize a queue
from kq import Queue
q = Queue()

# Enqueue the function call
q.enqueue(my_func, 1, 2, baz=3)

Sit back and watch the worker process the job in the background:

~$ kq worker --verbose
[INFO] Starting Worker(topic=default) ...
[INFO] Processing Record(topic=default, partition=5, offset=3) ...
[INFO] Running Job 1b92xle0: my_module.my_func(1, 2, baz=3) ...
[INFO] Job 1b92xle0 returned: (1, 2, 3)

Check out the full documentation for more details!

Installation

To install a stable version from PyPI (recommended):

~$ pip install kq

To install the latest version directly from GitHub:

~$ pip install -e git+git@github.com:joowani/kq.git@master#egg=kq

You may need to use sudo depending on your environment setup.

Credits

This project was inspired by RQ and built on top of kafka-python.