Home Forums Kamanja Forums Problems & Solutions Kamanja error logging concern

This topic contains 0 replies, has 1 voice, and was last updated by  Saaim 1 year, 5 months ago.

  • Author
    Posts
  • #17566 Reply

    Saaim

    The reason for calling this out as a Production Issue is to update the error logging in kamanja, as it creates confusion for Production Support purposes.

    These are the relevant config settings in Kamanja:

    request.required.acks : the default is 0 which means that there is no waiting for acknowledgement. Low latency; weakest durability. Kamanja is using 1, which waits for an acknowledgement from the current leader. This is not bulletproof but is usually considered acceptable. The -1 setting waits to make sure that the message was successfully replicate which guarantees delivery at the cost of performance. This usually should be used this when someone specifically says that they need guaranteed delivery.
    request.timeout.ms: The default is 10000 ms. This is the amount of time the broker will wait trying to send the message before it sends back an error. For example, today it must have taken longer than 10 seconds to elect a new leader, which is reasonable.

    message.send.max.retries: The default is 3.

    So basically, out of the box, Kamanja will try three times, at ten second intervals, to initiate contact with a topic. If we have 5 servers in prod and uat with a replication factor of 3, we should definitely be able to get a response within 30 seconds. If we don’t, throw an Error.

    If you go to Kamanja’s KafkaSimpleConsumer class, check out lines 335 and 336, you see the following code:
    LOG.error(“KAFKA ADAPTER: Failure fetching topic “qc.topic”, partition ” + partitionId + “, retrying”)
    LOG.warn(“KAFKA ADAPTER: Error fetching topic ” + qc.topic + “, partition ” + partitionId + “, recreating kafka leader for this partition”)

    This occurs in a while loop that checks to see if the process has failed too many times.
    https://github.com/LigaData/Kamanja/blob/e664fe1344538ec9e58031f59db4ffe1b8850f3e/trunk/InputOutputAdapters/KafkaSimpleInputOutputAdapters/src/main/scala/com/ligadata/InputAdapters/KafkaSimpleConsumer.scala
    When I look at lines 335 and 336, I think this is a typo. Every other log entry at that level is a warning and the state certainly hasn’t changed between lines 335 and 336.. If the number of retries were exceeded or if a leader or topic did not exist, then throw an error. Which is already done in the findLeader method
    } catch {
    case e: Exception =>

    {         LOG.error(“KAFKA ADAPTER – Fatal Error for FindLeader for partition ” + inPartition, e)       }
    Line 335 should be changed from error to warn.

Reply To: Kamanja error logging concern
Your information: