asynchronous - How to control sequence of Kafka Messages sent Asynchronously -


i have developed kafka version : 0.9.0.1 application.

one important aspect of application messages need consumed in correct sequence. because propagating database rows between 2 databases. means need ensure records within each unit of work sent uow-insert arriving before uow-update.

if use asynchronous message production, how can guarantee messages consumed in correct sequence?

i employ kafka producer send callback notified whether or not each message successfuly sent.

i have acks=all, retries=0, batch.size=16384 , kafka topics have single partition.

my consumer can cope duplicate messages, e.g. in case of retries being necessary, consumers cannot cope messages out of sequence.

my retry approach fail fast, e.g. messages fails send report records log record sequence number (lrsn) or relative byte address (rba) , stop sending messages.

then reset source database logs reported lrsn or rba , restart message production.

for example send messages

message               uow m1                    uow-0000  m2                    uow-0000  m3                    uow-0000  m4                    uow-0001  m5                    uow-0001  m6                    uow-0001  m7                    uow-0001  m8                    uow-0002

when message m5 failed send successfully, stop sending anymore messages. have issue consumer have received messages m1, m2, m3, m4, m6, m7, & m8.

to recover situation reset source database log reader reported lrsn or rba of m5 , start resending messages point.

now consumer receives

message               uow                    duplicate m5                    uow-0001                  no m6                    uow-0001                  yes m7                    uow-0001                  yes m8                    uow-0002                  yes

with approach speed of asynchronous messaging , "hopefully" messages consumed in desired sequence.


-

Comments

Post a Comment

Popular posts from this blog

amazon web services - S3 Pre-signed POST validate file type? -

i know it's possible put limit on file size content-length-range header. possible validate file type? https://docs.aws.amazon.com/amazons3/latest/dev/httppostforms.html#policyconditions i see there content-type header, if set say, audio/mp3 allow mp3 files , return error if file not mp3? i found previous question answers mention validating file size: s3 direct upload restricting file size , type you can specify content-type @ post request. you can specify @ signature, post must done content-type: { "expiration": "2007-12-01t12:00:00.000z", "conditions": [ {"acl": "public-read" }, {"bucket": "johnsmith" }, {"content-type: "audio/mp3"} ] } creating html form (using aws signature version 4) edit: @ previous question found, checking content-type.
Read more

c# - Check Keyboard Input Winforms -

i wondering if instead of doing this protected override void onkeydown(keyeventargs e) { if (e.keycode == keys.a) console.writeline("the key down."); } i set bool method , this: if(keydown(keys.a)) // whatever here i've been sitting here ages trying figure out how it. can't wrap head around it. in case wondering, plan call bool inside different method, check input. since want perform action after pressing key, using keydown event enough. but in cases suppose want check if specific key down in middle of process, can use getkeystate method way: [dllimport("user32.dll", charset = charset.auto, exactspelling = true)] public static extern short getkeystate(int keycode); public const int key_pressed = 0x8000; public static bool iskeydown(keys key) { return convert.toboolean(getkeystate((int)key) & key_pressed); } you should know, each time check key state using example iskeydown(keys.a) method returns true if ke...
Read more