- Separate permission to invoke a function from permission to act on a customer record.
- Test duplicate delivery and retries before they repeat a business action.
- Use narrowly scoped execution roles and retain correlation evidence without logging secrets.
Consider a hypothetical invoice workflow: an API accepts a document, a queue starts processing, and a function writes a result to a customer record. There may be no server for your team to patch, but there are several opportunities to authorize the wrong action or repeat a legitimate one.
Draw that event path before reviewing individual functions. Include the trigger, queue or topic, execution identity, data stores, external calls, and failure destination. Mark where the application establishes which customer is allowed to affect which record.
Separate invocation from business authorization
A valid event source does not prove that every field in an event is trustworthy. Validate schema, size, identifiers, and allowed operations. Derive the tenant or customer boundary from trusted authentication context where possible, and check access before reading or updating the referenced object.
For the invoice example, test whether one test customer can submit another customer's record identifier. The expected outcome is rejection without a cross-customer read or write. Use controlled test data and capture the authorization result.
For webhooks, verify the provider's supported signature and freshness mechanism before processing. For internal queues, limit who can publish and validate messages anyway. A compromised upstream service should not gain unrestricted authority merely because it can send to the queue.
Treat retries as a business rule
AWS's Lambda best practices note that event source mappings can process records more than once and recommend idempotent processing. Review the delivery semantics of your actual trigger; do not assume all serverless integrations behave identically.
Choose an idempotency identifier tied to the business operation and customer scope. Store processing state so a repeated event cannot issue a duplicate invoice or payment. Test failure after the external side effect but before the function records success. That boundary is where a simple "processed" flag can fall short.
Define retry limits, failure routing, and ownership of the dead-letter destination. A poison message should not consume unbounded retries. When replaying failed events, recheck whether the action is still valid and preserve the original correlation identifier.
Limit the function's authority
Give each execution role only the resource actions the workflow needs. A function that reads one document location should not receive broad storage administration or unrestricted secret access. Keep deployment permissions separate from runtime permissions.
Use a managed secret mechanism appropriate to the platform, restrict retrieval, and test rotation. Do not log full events by default: headers, document content, and tokens can leak into an otherwise useful diagnostic record. Record selected fields such as correlation ID, operation, result, and a safe resource identifier.
Test failure and containment
In a nonproduction environment, submit a malformed event, a duplicate, an unauthorized customer identifier, and a request that causes a controlled downstream failure. Check that each reaches the intended outcome and leaves enough evidence to explain it.
Set concurrency and rate controls against the capacity of downstream services, then monitor invocation volume, permission denials, timeouts, queue age, and unusual configuration changes. Treat spending alerts as supplementary signals, not a guaranteed real-time cutoff.
Finally, document how to pause the event source without losing queued work, preserve evidence, revoke a compromised execution identity, and resume processing safely. Those steps belong beside the deployment procedure so an on-call engineer can use them before a backlog becomes a business outage.
Turn this response guidance into clearer roles, containment decisions, evidence paths, and executive briefing rhythm.