Home Forums Kamanja Forums Data Science & Models Container Deserialization

This topic contains 2 replies, has 3 voices, and was last updated by  Archived_User36 1 year, 8 months ago.

  • Author
    Posts
  • #13334 Reply

    Archived_User57
    Participant

    Heman Duraiswamy:

      Hi Pokuri,

    Further to our call yesterday, you said that we already have an application that does:
    deserialize the container model stored in the Hbase
    KV Init which updates the container in hbase and notifies the engine to clear the cache and reload from the hbase
    Can you please send the github links to these applications at your earliest possible??

    Thanks,
    HD

    Sreenivasulu Pokuri:

      Hi Heman,

    Here are those details.

    Extract Data will extract the data out of our serialized format and produce JSON (through GSON) for the given message/container. Optionally you can ask for one PartitionKey and One Primarykey in partition key.

    Application: https://github.com/ligaDATA/Kamanja/tree/Sprint8Features/trunk/Utils/ExtractData

    File: https://github.com/ligaDATA/Kamanja/blob/Sprint8Features/trunk/Utils/ExtractData/src/main/scala/com/ligadata/ExtractData/ExtractData.scala

    Code lines you need to follow: From 147 to 399. Most of those small functions are useful for you. Some may not.

    KVInit is one of the place where we send the notification through Zookeeper to all other running instances whoever is watching for updating data to refresh their caches.

    Application: https://github.com/ligaDATA/Kamanja/tree/Sprint8Features/trunk/Utils/KVInit

    File: https://github.com/ligaDATA/Kamanja/blob/Sprint8Features/trunk/Utils/KVInit/src/main/scala/com/ligadata/tools/kvinit/KVInit.scala

    Code lines: 739 to 764

    Code:

    if (zkConnectString != null && zkNodeBasePath != null && zkConnectString.size > 0 && zkNodeBasePath.size > 0) {

    logger.info(“Notifying Engines after updating is done through Zookeeper.”)

    var zkcForSetData: CuratorFramework = null

    try {

    val dataChangeZkNodePath = zkNodeBasePath + “/datachange”

    CreateClient.CreateNodeIfNotExists(zkConnectString, dataChangeZkNodePath) // Creating

    zkcForSetData = CreateClient.createSimple(zkConnectString, zkSessionTimeoutMs, zkConnectionTimeoutMs)

    val changedContainersData = Map[String, List[List[String]]]()

    changedContainersData(typename) = savedKeys.toList

    val datachangedata = (“txnid” -> transId.toString) ~

    (“changeddatakeys” -> changedContainersData.map(kv =>

    (“C” -> kv._1) ~

    (“K” -> kv._2)))

    val sendJson = compact(render(datachangedata))

    zkcForSetData.setData().forPath(dataChangeZkNodePath, sendJson.getBytes(“UTF8”))

    } catch {

    case e: Exception => {

    logger.error(“Failed to send update notification to engine.”)

    }

    } finally {

    if (zkcForSetData != null)

    zkcForSetData.close

    }

    } else {

    logger.error(“Failed to send update notification to engine.”)

    }

    Thanks
    Pokuri

    Mohammed Abusaad:

      Thank you Pokuri for your guidance.

    I was thinking of packaging certain components of Kamanja, then importing them into my project.

    The file that you pointed me to (ExtractData.scala) contains a singleton object with functions that are mostly private. How can I deserialize in my external project without repeating the code? Do you have a running example?

    Best regards,
    ~Mohammed

    William Tarver:

      I’m afraid I’m not terribly familiar with the ExtractData project. I can take a look at it. What exactly are you trying to achieve? Do you want to be able to use the capabilities of ExtractData as part of external source code?

    Heman Duraiswamy:

      Scenario:

     

      For fusion project, Kamanja is processing a set of input message, what we call as “Indicators”, and saves these as a containers in HBase (as a serialized object)

    Problem statement:
    Abusaad is building an external app to manage these indicators – meaning he is required to access these indicators, edit it and save it back to the hbase. The problem is because the containers are saved in serialized manner in Hbase, he has to deserialize the object inorder to access the value. While discussing this problem with Pokuri, he suggested to look into this ExtractData project as it handles similar use case.

    Hopefully this gives you a little bit of back-ground, else, please let me know and I can schedule a quick conf call between 3 of us.

    Thanks and appreciate your help!

    Daniel Kozin:

      Hi,

    Reading stored information from Kamanja storage is not supported right now outside of kamanja. However, this is a requirement that should be implemented in a very near future. As far as I know, the Roadmap is being prioritized so the tentative support date should be available shortly.

    Dan

    Mohammed Abusaad:

      I’l read HBase data through Kamanja, rather than bypassing it.
  • #13335 Reply

    Archived_User7
    Participant

    what do you mean that you are reading from Kamanja, rather than bypassing??

    You are not intending to read it from Kamanja – rite?? – you wanted to read from your java application – correct? – I believe what Dan means by reading from Kamanja is my actually reading within the context of your metadata model.

    Please detail the all the steps that you are doing and we can evaluate a solution.

    • #13336 Reply

      Archived_User36
      Participant

      We are going to add feature to make the containers accessible by storing the data in more standardized serial format (AVRO or similar). This works both ways (Kamanja is the owner/writer, external systems are readers as well as external systems are owners and Kamanja (+components) are readers). This is essential to inter operate with other systems where the information stored in containers need to be accessed or updated outside Kamanja.

Reply To: Container Deserialization
Your information: