This is an AI updated revision of the original post
A stock update endpoint takes 5-14 seconds, blocking customer requests and locking up workers. The obvious fix is a background task queue – but which broker and Python client should you use? This post benchmarks Apache Kafka, Amazon SQS, and Redis across reliability, throughput, and developer experience to find the best fit.
The Problem
A single API serves both customer-facing requests and business-to-business stock updates. The stock update queries take 5–14 seconds, tying up workers and degrading customer response times. Clients wait up to 14 seconds for results. Scaling horizontally doesn’t help because every worker is equally likely to draw the expensive update request.
The Proposed Solution
Offload stock updates to a separate background process via a message queue:
- API receives a stock update request
- Request is validated and pushed to a queue
- API immediately returns
200 OKor202 Accepted - A background consumer picks up the message and runs the query
This decouples the slow update path from the fast customer-facing path so neither affects the other.
Queue Brokers
The following brokers are available in the current infrastructure:
- Apache Kafka – Distributed event streaming platform
- Amazon SQS – Fully managed message queuing service
- Redis – In-memory data structure store, used here as a message broker
- PostgreSQL – Relational database (listed for completeness)
For a full directory of task queue options, see taskqueues.com.
Python Client Libraries
- Celery – Mature distributed task queue supporting Redis, SQS, RabbitMQ, and more
- RQ (Redis Queue) – Lightweight Python library backed by Redis
- Dramatiq – Background task processing with RabbitMQ and Redis brokers
- Huey – Lightweight task queue supporting Redis, SQLite, and file-system storage
- Faust – Stream processing library for Kafka
- kafka-python – Low-level Python client for Apache Kafka
Methodology
Tests compare the following broker + library combinations:
| Broker | Library | Tested On |
|---|---|---|
| Amazon SQS | Celery | AWS |
| Apache Kafka | kafka-python | AWS (MSK) |
| Redis | RQ | Local only |
All tests run on the same hardware with a single process and thread for both producer and consumer. The producer makes database queries to build the payload – this is the most expensive part of the pipeline.
Three criteria are evaluated:
- Reliability – Are messages ever lost?
- Performance – How fast can messages be produced and consumed?
- Simplicity – Joy of developing and operating
Reliability Test
Send 100 messages (each containing 100 stock records) to the queue with the consumer stopped. Wait 5 minutes. Start the consumer, stop it halfway, then restart. Verify all messages are consumed. Record total time to send.
Performance Test
Send 10,000 messages (each containing 100 stock records) with the consumer running at default settings. Record total completion time for both producer and consumer.
Results
Amazon SQS + Celery
Status: Not viable for this payload size.
Amazon SQS enforces a 256 KB message limit:
botocore.exceptions.ClientError: An error occurred (InvalidParameterValue)
when calling the SendMessage operation: One or more parameters are invalid.
Reason: Message must be shorter than 262144 bytes (256 KB)The approximate size of a 100-record payload:
| Serialization | Size |
|---|---|
json.dumps |
172 KB |
pickle |
44 KB |
A batch of 10,000 records is approximately 1.72 MB – far exceeding the SQS limit. Even per-message JSON payloads at 172 KB are under the limit, but Celery adds serialization overhead that pushes it close. More importantly, Celery cannot accept raw bytes as a task argument, ruling out compact Protocol Buffers serialization. Splitting the payload into smaller chunks and reassembling adds unwanted complexity – the team wanted to respond to the client and handle the work in one go.
The maximum is roughly 800 records per SQS message given the combined payload and Celery overhead.
Reliability (downsized messages, 100 messages)
Time to produce messages: 4.69 s
Time to consume messages: 10.18 sPerformance (10,000 messages)
Time to produce messages: 485.48 s
Time to consume messages: 520.18 sMonitoring
SQS provides per-queue metrics in the AWS Console including message count, approximate age, and in-flight message count. The spike in "messages not visible" above reflects the 5-minute reliability test delay. Note: the same queue was shared across multiple task types.
Apache Kafka + Kafka-Python (JSON)
Default max message size: 1 MB
Kafka accepts both JSON and binary messages – so sending serialized Protocol Buffers bytes works without issue. A single message containing 10,000 stock records as JSON is 1,724,201 bytes (~1.72 MB), which exceeds Kafka’s default limit:
kafka.errors.MessageSizeTooLargeError: [Error 10] MessageSizeTooLargeError:
The message is 1724201 bytes when serialized which is larger than the maximum
request size you have configured with the max_request_size configurationDropping to batches of 5,000 records (~850 KB per message) fits within the 1 MB limit.
Reliability (100 messages)
Time to produce messages: 3.52 s
Time to consume messages: 9.23 sPerformance (10,000 messages)
Time to produce messages: 303.30 s
Time to consume messages: 400.54 sMonitoring
Kafka on AWS is provisioned as MSK (Managed Streaming for Kafka). Topic-level metrics are not surfaced in the console – only cluster-level metrics are visible. However, the Kafka producer exposes detailed metrics via producer.metrics():
{
"kafka-metrics-count": { "count": 72.0 },
"producer-metrics": {
"batch-size-avg": 17341.93,
"batch-size-max": 17364.0,
"byte-rate": 317140.18,
"bufferpool-wait-ratio": 0.0,
"compression-rate-avg": 1.0,
"connection-close-rate": 0.0,
"connection-count": 3.0,
"connection-creation-rate": 0.0,
"incoming-byte-rate": 1183.47,
"io-ratio": 0.0117,
"io-time-ns-avg": 196607.57,
"io-wait-ratio": 0.8313,
"io-wait-time-ns-avg": 13976968.09,
"metadata-age": 29.27,
"network-io-rate": 36.82,
"outgoing-byte-rate": 320075.00,
"produce-throttle-time-avg": 0.0,
"produce-throttle-time-max": 0.0,
"record-error-rate": 0.0,
"record-queue-time-avg": 0.0004,
"record-queue-time-max": 0.0055,
"record-retry-rate": 0.0,
"record-send-rate": 18.29,
"record-size-avg": 17280.93,
"record-size-max": 17303.0,
"records-per-request-avg": 1.0,
"request-latency-avg": 1.87,
"request-latency-max": 226.86,
"request-rate": 18.40,
"request-size-avg": 17399.04,
"request-size-max": 17437.0,
"requests-in-flight": 0.0,
"response-rate": 18.40,
"select-rate": 59.48
},
"producer-topic-metrics.test-stephen": {
"byte-rate": 317135.83,
"compression-rate": 1.0,
"record-error-rate": 0.0,
"record-retry-rate": 0.0,
"record-send-rate": 18.29
}
}These metrics help diagnose producer-side bottlenecks – batch sizes, request latency, error rates, and throughput per topic.
Apache Kafka + Kafka-Python (Protocol Buffers)
Switching from JSON to Protocol Buffers binary serialization reduces payload size dramatically:
| Format | 10,000 records | 5,000 records |
|---|---|---|
| JSON | ~1.72 MB | ~850 KB |
| Protobuf | ~490 KB | ~245 KB |
Binary serialization is roughly 3× more compact, meaning 3× less data transferred over the network.
Reliability (100 messages)
Time to produce messages: 2.91 s
Time to consume messages: 10.66 sPerformance (10,000 messages)
Time to produce messages: 262.09 s
Time to consume messages: 530.83 sProducing is faster with Protobuf (262 s vs 303 s for JSON) – expected given the smaller payload. Consumption took longer (531 s vs 401 s), likely due to deserialization overhead on the consumer side.
Redis + RQ
Caveat: These tests run on a local Redis instance, not the AWS ElastiCache cluster. Connecting to ElastiCache produced MOVED redirect errors, which suggest Redis cluster mode is not fully compatible with RQ’s pipeline usage:
redis.exceptions.ResponseError: Command # 1 (SADD rq:queues rq:queue:test)
of pipeline caused error: MOVED 8713 xxx.euw1.cache.amazonaws.com:6379RQ may require a single-node Redis instance.
On the local instance, RQ’s worker model follows a fetch → fork → execute loop. The initial reliability test showed a dramatic 240-second consumption time because the worker imports its libraries after forking. See the RQ performance notes for details.
Reliability (100 messages) – Before Fix
Time to produce messages: 2.54 s
Time to consume messages: 240.00 sReliability (100 messages) – After Fix
Optimizing the worker startup (pre-importing libraries):
Time to produce messages: 2.40 s
Time to consume messages: 9.13 sThese are local-only results. Performance on a networked Redis instance will differ.
Performance (10,000 messages)
Not separately tested – the reliability test revealed the bottleneck, and the fix resolved it. RQ’s throughput characteristics are well-documented and should scale linearly with the optimizations applied.
Monitoring
RQ ships with a CLI monitoring tool:
rq info
rq info --interval 1Monitoring is a first-class RQ feature with queue length, worker counts, and per-job timing available out of the box.
Results Comparison
| Broker + Library | Viable? | Produce 100 | Consume 100 | Produce 10k | Consume 10k | Notes |
|---|---|---|---|---|---|---|
| SQS + Celery | ❌ | 4.69 s | 10.18 s | 485.48 s | 520.18 s | Payload exceeds 256 KB limit; Celery rejects raw bytes |
| Kafka + kafka-python (JSON) | ⚠️ | 3.52 s | 9.23 s | 303.30 s | 400.54 s | Needs 5k-record batches to stay under 1 MB limit |
| Kafka + kafka-python (Protobuf) | ✅ | 2.91 s | 10.66 s | 262.09 s | 530.83 s | 3× smaller payload; fastest producer |
| Redis + RQ (local only) | ⚠️ | 2.40 s | 9.13 s | – | – | Fastest, but cluster compatibility unverified |
Conclusion & Recommendation
Apache Kafka + kafka-python with Protocol Buffers is the best fit for this use case:
- Payload fits comfortably – Protobuf serialization keeps messages at ~490 KB for 10,000 records, well under Kafka’s 1 MB default limit.
- Fastest production speed – 262 s to produce 10,000 messages, 13% faster than the JSON variant and 46% faster than SQS + Celery.
- Binary support – Kafka accepts raw bytes natively, unlike Celery which blocks Protobuf serialization.
- Production-ready – Kafka is battle-tested for high-throughput, reliable message processing with strong durability guarantees.
Redis + RQ is the runner-up if the cluster compatibility issue can be resolved. Its local performance is excellent (sub-10-second consumption for 100 messages), and the monitoring tooling is polished. However, the MOVED redirect error on ElastiCache clusters is a significant blocker without dedicated single-node Redis instances.
Amazon SQS + Celery is not viable for this workload. The 256 KB message limit forces payload splitting, and Celery’s inability to accept raw bytes eliminates Protobuf as a workaround.
For teams already running Kafka or willing to provision it, the combination of kafka-python and Protocol Buffers delivers the best balance of throughput, reliability, and developer experience for large-payload background task processing in Python.
