Home Forums Kamanja Forums Features & Releases Offsetting engine kafka input queue topics

This topic contains 4 replies, has 3 voices, and was last updated by  Archived_User79 1 year, 8 months ago.

  • Author
    Posts
  • #13232 Reply

    Archived_User1
    Participant

    Hi

    How can I offset engine Kafka input topics so we can replay test data? ea. Let say there are 1000 messages in the input queue and I want to run only 500 messages, how can offset starting point to 500 and where is it stored?

    Thanks
    Mohamed

  • #13233 Reply

    Archived_User28
    Participant

    Hi Mohamed,

    At this moment the answer is NO (you can not reset offset).

    I have the following questions:

    1. We are supporting other than Kafka also. At this moment we can give a tool to reset the partitions counters for queues (atleast kafka queues for now). But my question is do you maintain the offsets externally for each partition of the topic/queue to reset it back?
    2. What do you want to do with output Adapter (queue)? you simply ignore the results we have in ouput adapters (queue) and consider only new data?
    3. What about Database state? because that data is already processed and the data stage (history of the messages & model results state) already updated with that data processed before. Do you want to revert it back or what do you want to do with that?

    Thanks
    Pokuri

  • #13234 Reply

    Archived_User1
    Participant

    Hi Pokuri,

    Q1. yes, we want to maintain them externally for each partition of the topic and we should be able to reset offset point the we want to start from . for example if the there are 1000 messages in the topic/partition queue and we want run 100, we want simply reset offset point to 900. Once messages are consumed we should be able to do get kafka/app/partition_number to make sure offset point is reset to 1000 again.

    Q2. Yes, we want to measure only the new data and ignore the data that already being processed

    Q3. No, we don’t want to revert back, we just want update with new data.

    Thanks
    Mohamed

  • #13236 Reply

    Archived_User1
    Participant

    Pokuri,

    When can you give us the tool that allows us to reset kafka partitions counters?

    Thanks
    Mohamed

  • #13238 Reply

    Archived_User79
    Participant

    Mohamed,

    We are currently focused on releasing Kamanja version 1.1 and revisit this request as an add-on only after 1.1 is releasing.

    Thanks,

    Mahesh Lalwani
    VP of Engineering | ligaDATA


    Find the latest on our open-source project at http://www.kamanja.org

Reply To: Offsetting engine kafka input queue topics
Your information: