Throttling Email Sending (AWS SES) w/ Lucee

dev
1

We need to send out a large amount of emails using AWS SES (Simple Email Service). Our customer has a send limit of 50 emails / second. This is plenty for them during normal operation, but sending out an email to all their customers (25k) would take forever.

Is there a way to configure Lucee to only send like 40 emails / second (leave overhead for their normal emails? Batch schedule them all and let them trickle out slowly?

Any thoughts or help would be appreciated.

:slight_smile:

2

First: This would be best suited for an email service that can handle the volume. The cost of writing and supporting your own solution is likely to be much higher as you run into unknowns rolling your own clever when it works solution.

The rest is theoretical, untested, and not recommended.
If I were to solve this, I might use a message queue. (it may be memory for a single server or table if I need to support multiple servers / need a record).

  • All emails go into the queue: (from, to, fail-to, subject, body, priority, etc)
    • email types are sorted so bulk is always last, new regular email always skips ahead
  • An “email worker” (unique scheduled task?) runs every minute.
    • Unique makes sure another doesn’t start until this one completes.
  • The email worker runs in a loop and exits after 1 minute (or 60 iterations)
  • Each iteration is responsible for the following:
    • Get next batch of email with high and normal priority first
    • sleep until 1 sec of time has passed from lastBatchSent iteration (I may be inclined to take a 1 sec idle sleep hit on the first iteration or add more code)
    • Loop through the email batch.
    • Mark batch complete / Set lastBatchSent = GetTickCount() or some timer for designating when the batch was sent.

I mentioned sleep before sending because I have questions and doubt about this 50/sec limit. If I send 51, what happens to message 51? A delay? Silently dropped?

If sleep is done after, the timing could be off a bit:
Start at 01:01:01.000
50 messages are sent and that ends at 01:01:01.222
Sleep for 778
50 more sent ending at 01:01:02.111 (technically this is up to 100 messages within 1 second). By controlling the start, you can fine tune your buffer between sends.

3

Thanks Jason.

I noticed one typo in my original message. We are trying to send 250k emails, NOT 25k. We have looked at email delivery company options, but at that scale, it gets pretty expensive.

We currently have a very similar approach implemented on our clients server, but I was hoping to be able to spool the mail and let JavaMail handle the send limits. I seem to recall that ColdFusion had a spool option and possibly a send limit. Is there anyway to set this up in Lucee?

1 Like
4

there is a sendtime attribute, haven’t tried it myself

also you might consider using the async=false, which means the mail gets sent (i.e. not spooled) when the cfmail tag is called

1 Like
5

Woof! :dog:
I think it’s safe to assume cfmail is the wrong starting point once you hit that volume. If you find some params that’ll work let us know.

I’d investigate the API / template features they have in SES and bulk email. Run some tests (they have that documented somewhere with doing test sending to check/verify you’re under the account send limits).

Thank you for the fun thought experiment. :nerd_face: