Given the following scenario:
I have a system that creates, updates and deletes records. For each of these actions I need to do something (lets say write the events to a log as a silly example) however I need to process these events for each record in order – Meaning I can’t log the delete before I have done the create, or any of the previous updates. I also can’t log the update before I have logged the create.
I am investigating Queues in order to preserve sequence. However I don’t really want RecordID_2 to be held up behind RecordID_14 The records do not need to be processed in sequence as much as the actions on each record have to. Hence I don’t think I can/should use one queue.
As I don’t have hundreds of RecordID_XX active at the same time, I was thinking of having a queue for each RecordID_XX so if several updates can in for that one RecordID each event for that record would be added to that same queue and be processed in order (I.e. Create first, Update_1 after Create is complete, Update_2 is processed after Update_1 is complete etc), however if additional events for a different record came in they would be added to their own queue. If the queue is empty for a period of time it simply gets deleted. I realize that this may result in a queue getting one message and then being deleted as there were no updates before the idle timeout expired. (This does not seem at all efficient)
Based on Andres T Finnell’s excellent answer to this question.
I was thinking of doing the following
Producer (Web Service) -> Queue_Original <- Dispatcher -> RecordID_14
Some of the “logging” may take long. So I want to be able to have a few consumers listening for these queues.
Lets say I have Consumer_1 and Consumer_2 (I may want to add Consumer_3 later to assist with growing load)
What I would like is Consumer_1 to do a getDistinations()
where the broker will return [RecordID_14, RecordID_2, RecordID_8, RecordID_15]
Is it possible for Consumer_1 to iterate through the list of queues returned from the broker looking for the first available queue that does not have a Consumer_X connected to it and begin processing the 1st message on this queue? And then each subsequent Consumer to do the same until it finds the next queue without a Consumer connected to it?
Am I going down the wrong path completely? Is there a better approach to handling this scenario?