Hi guys,
Looking for a bit of advice.
I’m about to build an app and am wondering if event gateways specifically the directory watcher is the best solution on its own or if I should also implement some kind of queue or somthing else. The app will be receiving potentially 100’s of files per minute via FTP that need to be processed (at the beginning probably not but later)
Now I could use a directory watcher to check the directory every minute or a scheduled task to do the same. To me the directory watcher would make more sense since it will only trigger if a file actually lands in the directory and not run every minute whether there are new files or not.
The question is, if there are 100’s of files and it takes longer to process the 100’s of files than the minute interval the directory watcher will probably spawn another thread with the list of newly arrived files, if this continues I could run into a memory or performance problem by using all my threads.
I was thinking something along these lines.
I could use the directory watcher to get the incoming files and add them to some kind of queue, if the queue is empty it could add them and then start a worker, if the queue is not empty then no worker is started and the files are just added to the queue. Using either an external pub/sub or messaging system (rabbitmq, Kafka etc?)
How would you approach this? I’m running on a managed VPS
Cheers and Thanks
Gary