Retrieve S3-Offloaded SQS Payloads
Configure Tarmac to detect and retrieve large AWS SQS message payloads offloaded to Amazon S3.
SQS caps message bodies at 1 MiB. Systems that move larger payloads put the real body in S3 and send a pointer through the queue. The Amazon SQS Extended Client does this, and so do plenty of homegrown variants. Tarmac spots those pointers and lets you retrieve the actual payload instead of working with an opaque pointer blob.
What Tarmac detects
Tarmac checks every received message for three kinds of pointer:
- Extended Client pointers. The pointer shape the SQS Extended Client libraries produce.
- SNS-wrapped pointers. A pointer carried inside an SNS notification envelope.
- Your own fields. Body fields you configure that hold a bucket and key, or a full
s3://bucket/keyor S3 URL.
If your producers use their own field names, add them under Settings ▸ External payloads. Tarmac does not treat a random JSON bucket/key pair as an offload pointer unless you name those fields or the message carries an Extended Client size attribute. Custom names are matched case-sensitively.
Automatic and manual retrieval
Automatic retrieval is off by default. Tarmac still detects pointers and stores them locally, but it makes no S3 request until you click Retrieve in the inspector.
Turn automatic retrieval on under Settings ▸ External payloads and Tarmac fetches detected payloads up to the threshold you set, 256 KiB by default. Those show Auto-retrieved. Anything larger, or anything whose size Tarmac cannot determine, waits for you to click Retrieve. A retrieval that fails shows Retrieval failed, and you can retry it by hand. A payload you already retrieved is never lost this way: if the same message is received again and the S3 object is gone or unreachable, Tarmac keeps the stored copy.
Manual retrieval is hard-capped at 10 MiB, and the automatic threshold cannot exceed that cap.
Retrieved payloads are stored with the message in the local cache, appear in a dedicated S3 tab pretty-printed like the body, and get their own copy button. Rows with external payloads show an S3 chip in the message list, colored by status. If you fetch a message again and its pointer has not changed, Tarmac keeps the payload it already retrieved and does not download it a second time.
You need s3:GetObject on the payload bucket
Payload retrieval uses your Tarmac credentials, not the producer's. If
messages show "Retrieval failed" with an access error, grant s3:GetObject on
the offload bucket to the identity Tarmac is connected as.
The pointer and the object have different lifetimes
Deleting a queue message does not delete its S3 object. Redriving or sending a pointer message sends the pointer, not a copy of the object. So a resent pointer message is only useful while the S3 object still exists. Check the bucket's lifecycle rules before you replay old messages. Staged deletes in Tarmac never remove S3 objects either, so offloaded payloads stay until your lifecycle policy cleans them up.
LocalStack
The LocalStack connection preset works with LocalStack S3, so extended-client payloads retrieve there too. See Connecting to AWS.