AWS SQS Troubleshooting Workflows

Recipes that combine Tarmac's features into complete jobs. Each one assumes you've read Getting Started. The reasoning behind the steps lives in SQS Essentials.

Staged actions and Queue Redrive need Tarmac Pro. Inspection and Live Tail work on Free.

Diagnose a failing consumer

Symptoms: DLQ depth climbing, or messages cycling without being processed.

1. On the source queue, open Queue Tools ▸ View Topology. Confirm which DLQ it feeds and note the Max receives threshold.
2. Fetch a sample from the source queue. The default 15-minute visibility gives you time to inspect from the cache.
3. Open a few messages and check System ▸ Receive Count. A high receive count on an old message means it is being received over and over without a successful delete. Look at the payload, handler failures, processing time against the visibility timeout, and errors thrown after receipt.
4. Switch to the DLQ, fetch, and open Facet Filters. Facet on error and status fields in the body to bucket failures by cause instead of reading messages one by one.
5. Inspect a representative from the biggest bucket. The Envelope tab shows SNS wrapping and metadata, and the S3 chip shows whether Tarmac detected or retrieved an external payload.

You end up with a failure taxonomy with counts, which is what the fix's PR description needs.

Drain a DLQ after a fix

1. Deploy and verify the consumer fix first. Redriving into a broken consumer only increments receive counts.
2. Select the DLQ and fetch with a visibility timeout sized to your review. Deleting originals requires live receipt handles.
3. Filter to the messages the fix addresses, multi-select, and choose Stage Send. Choose the destination explicitly, and enable Delete originals after successful send only if you want the originals gone. Use Stage Delete for hopeless messages and single-message Stage Edit & Send for salvageable payloads.
4. Review Staged mode and choose Publish Actions before the visibility window closes. If delete-after-send cannot get a valid receipt handle, fetch that message again and re-stage it.
5. Live Tail the DLQ and watch for boomerangs. Redriven messages reappearing there mean the fix missed a case.

For an everything-goes-back redrive with no triage, use Queue Tools ▸ Queue Redrive. See DLQ & Redrive.

Repair and replay a poison message

A message with malformed data your consumer can never process:

1. Fetch with a generous visibility timeout, since you'll want to delete the original afterward.
2. Open the message and choose Stage Edit & Send. Fix the payload and choose the source queue as the destination.
3. Leave Delete originals after successful send off to verify the repaired copy first, or enable it to delete only after AWS confirms the send.
4. On FIFO queues, Tarmac keeps the original message group ID and publishes staged messages in order within a group. Retrying a staged send won't create a second FIFO delivery.

Inspect production safely

• Connect with a read-only profile, described in Connecting to AWS. Fetching, caching, searching, and inspecting need no write permissions.
• Verify the identity in the top bar says the account you think it does.
• Keep the default 15-minute visibility, fetch once, and do all analysis in the local cache. Your consumers cannot see those messages while the timer runs.
• Mind queues with a low maxReceiveCount. Your receives count toward the DLQ threshold like anyone else's.
• On a shared machine, open Settings and choose Delete all local data… when you're done, to remove cached payloads.

Develop against LocalStack

1. In the credentials modal choose AWS-compatible endpoint and apply the LocalStack preset (http://localhost:4566, test/test).
2. Create queues with awslocal sqs create-queue --queue-name my-queue. Tarmac lists queues, it doesn't create them.
3. Queue operations use the compatible endpoint. Fetch, Live Tail, staged actions, and extended-client S3 payloads all work when LocalStack exposes its S3 service.
4. Use Live Tail while you exercise your producer code for an instant "did my message actually send, and what does it look like?" loop. The inspector's unwrapped JSON view catches double-stringified bodies immediately, the classic producer bug.

Watch a deployment

During a rollout, Live Tail plus the cache make a lightweight canary.

1. Live Tail the queue your new producer writes to and confirm the shape and attributes of fresh messages as they arrive. On Standard queues it samples about one message every 3 seconds. On FIFO it samples once per session. It also stops itself rather than sample a message close to the queue's redrive limit. Treat it as a spot check, not a firehose.
2. Stop the tail before you change queues, then select the DLQ and Fetch Messages when you need a direct failure check. Use timestamps in the cached sample to separate rollout failures from older ones.
3. After the rollout, fetch from both queues and use the date-range controls on the deploy window to compare message shapes before and after.