Home Forums Kamanja Forums Data Science & Models How to store binary data in Kamanja containers?

This topic contains 5 replies, has 3 voices, and was last updated by  Archived_User42 1 year, 6 months ago.

  • Author
    Posts
  • #13496 Reply

    Archived_User42
    Participant

    hi all,

    We need to store java serialized binary data in containers – can someone please let me know if we can do this using KVInit? if yes, please direct me to an example. The examples from SampleApplications are using CSV or delimited data only.

    thanks,

  • #13497 Reply

    Archived_User7
    Participant

    Is this serialized binary data a field of some data structure? In any case, currently there is no mechanism in the message deserializer to decode some input data into array of bytes. Best bet is to encode the binary data as hex decimal or 64 base string and store as string and then decode up on retrieving into memory. We will add proper conversion function as part of transformation work that will be doing in near future.

    Thanks

  • #13498 Reply

    Archived_User42
    Participant

    Thanks Krishna. We encoded the binary data in base 64 string, but now we are getting below error while loading it in HBase store using KVInit:

    ERROR com.ligadata.keyvaluestore.HBaseAdapter – Failed to save a list of objects in table(s) :Row length 52139 is > 32767

    I suppose it is to do with some setting for the storage adapter – can someone please let me know what setting has to be done to allow insertion of bigger row values in the data store?

    Thanks,

  • #13499 Reply

    Archived_User42
    Participant

    Hi all,

    Can someone please let me know about the below issue at the earliest possible?

    Thanks,

  • #13500 Reply

    Archived_User19
    Participant

    Looks like you are keeping the KEY (ROW) value as 50KB. HBase allows only max Short.MAX_VALUE as the ROW value.

    Can you please check once whether you are keeping all that stuff as KEY or not?

    Thanks

  • #13501 Reply

    Archived_User42
    Participant

    Yes Pokuri, that was the issue. I added a separate key and was able to insert the data in the data store.

    Thanks for the help!

Reply To: How to store binary data in Kamanja containers?
Your information: