I was ready to hate on this article, but it's actually pretty good. Having implemented all of these, I have a few notes:
Api gateway calls are limited to 30 seconds. This is obviously a problem if your processing takes longer than 30 seconds, especially for direct calls.
Provide a way to shut off processing and continue ingesting. Ingestion can never stop, because handling client retries is a nightmare.
Retried/poison sqs messages are a problem. They will eventually clog up your queue id not handled. Throw them away or send them to an error queue.
The best ingestion method depends on your response time and cost requirements. If possible, ingestion without API calls at all is the best solution. If not, then lambdas front ended by cloudflare or fastly are great. Lambdas, as long as they don't do work, scale unbelievably well and are unbelievably cheap.
Oh lastly, dom't let your edge lambdas touch your database. Always moderate via processing, because if you don't you (or a ddos, or a bad cloent) will crush your db.
I was ready to hate on this article, but it's actually pretty good. Having implemented all of these, I have a few notes:
Api gateway calls are limited to 30 seconds. This is obviously a problem if your processing takes longer than 30 seconds, especially for direct calls.
Provide a way to shut off processing and continue ingesting. Ingestion can never stop, because handling client retries is a nightmare.
Retried/poison sqs messages are a problem. They will eventually clog up your queue id not handled. Throw them away or send them to an error queue.
The best ingestion method depends on your response time and cost requirements. If possible, ingestion without API calls at all is the best solution. If not, then lambdas front ended by cloudflare or fastly are great. Lambdas, as long as they don't do work, scale unbelievably well and are unbelievably cheap.
Oh lastly, dom't let your edge lambdas touch your database. Always moderate via processing, because if you don't you (or a ddos, or a bad cloent) will crush your db.