joowani / kq
- среда, 2 ноября 2016 г. в 03:13:27
Python
Kafka-based Job Queue for Python
KQ (Kafka Queue) is a light-weight Python library which provides a simple API to queue and process jobs asynchronously in the background. It is backed by Apache Kafka and designed primarily for ease of use.
First, ensure that your Kafka instance is up and running:
# This command is just an example
~$ ./kafka-server-start.sh -daemon server.properties
Let's say you want to run the following function asynchronously:
import time
def my_func(foo, bar, baz=None):
"""This is a blocking function."""
time.sleep(10)
return foo, bar, baz
Start a KQ worker:
~$ kq worker --verbose
[INFO] Starting Worker(topic=foobar) ...
Enqueue the function call as a job:
# Import the blocking function
from my_module import my_func
# Initialize a queue
from kq import Queue
q = Queue()
# Enqueue the function call
q.enqueue(my_func, 1, 2, baz=3)
Sit back and watch the worker process the job in the background:
~$ kq worker --verbose
[INFO] Starting Worker(topic=default) ...
[INFO] Processing Record(topic=default, partition=5, offset=3) ...
[INFO] Running Job 1b92xle0: my_module.my_func(1, 2, baz=3) ...
[INFO] Job 1b92xle0 returned: (1, 2, 3)
Check out the full documentation for more details!
To install a stable version from PyPI (recommended):
~$ pip install kq
To install the latest version directly from GitHub:
~$ pip install -e git+git@github.com:joowani/kq.git@master#egg=kq
You may need to use sudo
depending on your environment setup.
This project was inspired by RQ and built on top of kafka-python.