This seems like a case where a mutex would be the solution. You will need to atomically check-and-set the mutex using UpdateAsync from something like DataStore or MemoryStoreSortedMap. The latter is not intended to be used this way, but maybe it could be used to circumvent request limits if those become a problem
local valid = false
if not mutex then
valid = true
mutex = true
if valid then
-- do your stuff
The mutex can be much richer than just a boolean. Something that may become necessary is storing the active lock’s start time so that you can set it to automatically expire in the case where a server holds the mutex for too long.
Each server wanting to do some processing will need to poll frequently which may be a bit slow and use up your request budget. It might be possible to lighten the weight a little by using MessagingService at the same time to notify all servers of the mutex being unlocked so they don’t need to poll as much, but I don’t think we can get rid of polling completely.
I assume when processing the queue, you’re also going to want to remove things from the queue, which means it may be necessary to lock the entire queue for add/removes (and not just call ReadAsync once). I think it would be worth testing to see how bad the edge cases are if you don’t lock everything, since being able to add and remove regardless of the processing state will be much faster.