SNS and SQS fan-out: one publish, two queues, and a filter that sorts the orders

Amazon SNS with Amazon SQS, in plain words

Amazon SNS (Simple Notification Service) is publish/subscribe messaging. A sender publishes once to a topic, and SNS pushes a copy to every subscription: SQS queues, Lambda functions, HTTP endpoints, email and more. SNS doesn't keep messages for later, so if nobody is subscribed when you publish, the message is simply not delivered anywhere.

Amazon SQS (Simple Queue Service) is the opposite shape: a queue that holds messages until a consumer pulls them, for up to 14 days. Consumers work at their own pace, and a slow or broken worker doesn't lose data; the messages wait. Subscribing several queues to one topic is the fan-out pattern: one event, such as a new order, lands in a billing queue, a shipping queue and an audit queue, and each team's workers process their copy independently.

Each subscription can carry a filter policy, a small JSON document that decides which messages it receives. By default it checks the message attributes the publisher sets; it can be switched to check fields inside a JSON message body instead. Two settings trip people up: the queue's access policy must allow the topic to send to it, and a queue encrypted with AWS KMS needs a customer managed key that SNS is allowed to use. There's no notable SNS or SQS launch in the last two weeks; the most recent SNS change is that message data protection closed to new customers on April 30, 2026.

The AI angle

Generative AI calls are slow, rate-limited and sometimes throttled, which is exactly what a queue is good at absorbing. A common shape: a "document uploaded" event goes to an SNS topic; one SQS queue feeds a worker that calls a Bedrock model or starts a Knowledge Base ingestion at a rate your quota allows, while a second queue feeds an audit or indexing job. If the model call fails, the message stays in the queue and is retried, instead of being lost in a failed request. Filter policies can send only certain document types or customer tiers to the more expensive model.

Hands-on lab: Fan out three orders to two queues and filter the big ones

From AWS CloudShell, create an SNS topic and two SQS queues, give the topic permission to write to them, subscribe both (one with a numeric filter policy and raw delivery), publish three orders and see which queue got what. Then delete everything.

1. Open CloudShell and create one topic and two queues

   Open CloudShell from the console's top bar. The random suffix keeps the names unique in your account. The audit queue will receive every order; the big queue should only receive large ones. The last line saves the variables to a file, so if CloudShell restarts you can run source lab-vars.sh and carry on.

   REGION=us-east-1
   SUFFIX=$RANDOM
   TOPIC_ARN=$(aws sns create-topic --region "$REGION" \
     --name "lab-orders-$SUFFIX" --query TopicArn --output text)
   AUDIT_URL=$(aws sqs create-queue --region "$REGION" \
     --queue-name "lab-audit-$SUFFIX" --query QueueUrl --output text)
   BIG_URL=$(aws sqs create-queue --region "$REGION" \
     --queue-name "lab-big-$SUFFIX" --query QueueUrl --output text)
   qarn() { aws sqs get-queue-attributes --region "$REGION" --queue-url "$1" \
     --attribute-names QueueArn --query Attributes.QueueArn --output text; }
   AUDIT_ARN=$(qarn "$AUDIT_URL"); BIG_ARN=$(qarn "$BIG_URL")
   declare -p REGION SUFFIX TOPIC_ARN AUDIT_URL BIG_URL AUDIT_ARN BIG_ARN > lab-vars.sh
   cat lab-vars.sh

   Done

2. Let only this topic write to each queue

   A new queue accepts messages only from its own account's IAM principals, so SNS needs permission in the queue's access policy. The aws:SourceArn condition limits that permission to your one topic, which guards against the confused deputy problem. jq turns the policy into the escaped JSON string that the Policy attribute expects.

   allow_topic() {
     jq -n --arg q "$1" --arg t "$TOPIC_ARN" '{Policy: ({Version: "2012-10-17",
       Statement: [{Effect: "Allow", Principal: {Service: "sns.amazonaws.com"},
         Action: "sqs:SendMessage", Resource: $q,
         Condition: {ArnEquals: {"aws:SourceArn": $t}}}]} | tojson)}' > queue-attrs.json
     aws sqs set-queue-attributes --region "$REGION" --queue-url "$2" \
       --attributes file://queue-attrs.json
   }
   allow_topic "$AUDIT_ARN" "$AUDIT_URL"
   allow_topic "$BIG_ARN" "$BIG_URL"
   aws sqs get-queue-attributes --region "$REGION" --queue-url "$BIG_URL" \
     --attribute-names Policy --query Attributes.Policy --output text | jq .

   Done

3. Subscribe both queues, one with a filter policy

   The audit queue subscribes with no filter, so it gets every message. The big queue gets a filter policy that matches only when the order_total message attribute is 100 or more, and raw message delivery, so its messages arrive without the SNS JSON envelope. Because you own both the topic and the queues, the subscriptions are confirmed straight away. AWS says filter policy changes can take up to 15 minutes to fully apply, so the step waits a minute before you publish.

   aws sns subscribe --region "$REGION" --topic-arn "$TOPIC_ARN" \
     --protocol sqs --notification-endpoint "$AUDIT_ARN" --return-subscription-arn
   jq -n '{RawMessageDelivery: "true",
     FilterPolicy: ({order_total: [{numeric: [">=", 100]}]} | tojson)}' > sub-attrs.json
   aws sns subscribe --region "$REGION" --topic-arn "$TOPIC_ARN" \
     --protocol sqs --notification-endpoint "$BIG_ARN" \
     --attributes file://sub-attrs.json --return-subscription-arn
   aws sns list-subscriptions-by-topic --region "$REGION" --topic-arn "$TOPIC_ARN" \
     --query 'Subscriptions[].[Endpoint, SubscriptionArn]' --output table
   sleep 60

   Done

4. Publish three made-up orders

   One publish call per order. Two carry an order_total attribute of type Number; the third has no attribute at all. SNS answers each call with a MessageId and then delivers copies to the matching subscriptions on its own. Publishing succeeds even if a queue refuses the delivery, so the queues are where you check the result.

   pub() {
     aws sns publish --region "$REGION" --topic-arn "$TOPIC_ARN" \
       --message "$1" --query MessageId --output text "${@:2}"
   }
   pub "order A: 2 notebooks" --message-attributes 'order_total={DataType=Number,StringValue=25}'
   pub "order B: 1 laptop" --message-attributes 'order_total={DataType=Number,StringValue=1250}'
   pub "order C: no total given"

   Done

5. Read both queues and compare

   Long polling (--wait-time-seconds) waits for messages to arrive. Messages on the audit queue are SNS envelopes, so jq pulls out the Message field and the attribute SNS copied into the envelope. On the big queue the body is your original text, and with raw delivery the attribute arrives as an SQS message attribute instead. A standard queue can return fewer messages than are waiting in a single call; if the audit list is short, run the first command again.

   aws sqs receive-message --region "$REGION" --queue-url "$AUDIT_URL" \
     --max-number-of-messages 10 --wait-time-seconds 10 --output json \
     | jq -r '.Messages[]?.Body | fromjson | [.Message, (.MessageAttributes.order_total.Value // "-")] | @tsv'
   aws sqs receive-message --region "$REGION" --queue-url "$BIG_URL" \
     --max-number-of-messages 10 --wait-time-seconds 10 \
     --message-attribute-names All \
     --query 'Messages[].[Body, MessageAttributes.order_total.StringValue]' --output text

   Done

for S in $(aws sns list-subscriptions-by-topic --region "$REGION" \
    --topic-arn "$TOPIC_ARN" --query 'Subscriptions[].SubscriptionArn' --output text); do
  aws sns unsubscribe --region "$REGION" --subscription-arn "$S"
done
aws sns delete-topic --region "$REGION" --topic-arn "$TOPIC_ARN"
aws sqs delete-queue --region "$REGION" --queue-url "$AUDIT_URL"
aws sqs delete-queue --region "$REGION" --queue-url "$BIG_URL"
rm -f queue-attrs.json sub-attrs.json lab-vars.sh
sleep 60
aws sns list-topics --region "$REGION" \
  --query "Topics[?contains(TopicArn, 'lab-orders-$SUFFIX')]"
aws sqs list-queues --region "$REGION" --queue-name-prefix "lab-" \
  --query "QueueUrls[?contains(@, '$SUFFIX')] || \`[]\`"

Gotchas

• Forgetting the queue access policy is the classic mistake. The publish call still succeeds, because SNS delivers to subscribers afterwards, but the message never shows up in the queue. Scope the permission to one topic with aws:SourceArn, or to your account with aws:SourceAccount when many topics feed one queue.
• AWS says adding or changing a filter policy can take up to 15 minutes to fully take effect. Don't conclude a filter is wrong from a test run a few seconds after you saved it.
• A filter policy checks message attributes unless you set FilterPolicyScope to MessageBody. If the policy names a property the message doesn't have, the message doesn't match, as order C shows. Attribute-based filtering is free, but body-based filtering is billed on the payload data SNS scans.
• Without raw message delivery, every SQS message body is an SNS JSON envelope, and the original text sits in its Message field. Decide one way per subscription and make the consumer code match.
• For a queue encrypted with AWS KMS, SNS needs a customer managed key whose key policy allows sns.amazonaws.com to use kms:GenerateDataKey* and kms:Decrypt. The lab's queues use the default SQS-managed encryption, so no key policy is involved.