Home Forums Kamanja Forums Data Science & Models Transformation and Storage Model

This topic contains 9 replies, has 5 voices, and was last updated by  Archived_User79 1 year, 8 months ago.

  • Author
    Posts
  • #13185 Reply

    Archived_User19
    Participant

    How do we create a model that only transforms and stores the data into a storage container without producing any alerts?

  • #13186 Reply

    Archived_User36
    Participant

    There is no reason the model written cannot ALWAYs return false. When a false is returned, no results are produced by the engine from the model results returned by the model.

    On Fri, Sep 4, 2015 at 12:03 AM, Nithin Bose <nit…@ligadata.com> wrote:
    How do we create a model that only transforms and stores the data into a storage container without producing any alerts?

    You received this message because you are subscribed to the Google Groups “kamanja” group.
    To unsubscribe from this group and stop receiving emails from it, send an email to kamanja+u…@ligadata.com.
    To post to this group, send email to kam…@ligadata.com.
    Visit this group at http://groups.google.com/a/ligadata.com/group/kamanja/.

  • #13187 Reply

    Archived_User36
    Participant

    I should have said null not false. Return null result from execute.

  • #13189 Reply

    Archived_User19
    Participant

    Great. With this approach is it easy to configure storage into a flat or defined schema into any repo? Like hive or hdfs?

  • #13190 Reply

    Archived_User57
    Participant

    It was an interesting question and peeked my curiosity. I have looked the engine code and it
    seem to ignore “null” results. I have tried the scenario with one of the sample scala models(using watchOutputQueue.sh to observe the output)

    If I don’t return null, I see the results in “watch” window. If I return null, I don’t see anything.
    Nothing seem to break and I was able to save values in a container.

    I wonder there are any other side effects ? Also, It is not clear “doing nothing in case of a null result” is by design.

    Pokuri will have the correct answer 🙂

    Ramana

  • #13193 Reply

    Archived_User79
    Participant

    Model ouput result is not the way we should do this. Use container for transformed data and let model save the resulted row in container.

  • #13195 Reply

    Archived_User36
    Participant

    I would say that certain enhancements will be needed to several components (e.g., EnvContext, storage and probably message compiler) to write to defined schema. We have been talking about the need to have the engine, one of its models, or even an external application be able to update a table that can be read by any party participating in the application. There is definitely some work to do to make this happen.

    For example, the twitter demo web app writes its topics and selection criteria to the table-able storage and the model instances that use such data pick that up when notified that a change has been made. From the model perspective the EnvContext is the gateway to the persistent storage’s table.

    This is a useful pattern imo. Try to be more specific about what you are thinking.

  • #13196 Reply

    Archived_User7
    Participant

    I believe it may also be possible to go a slightly different route that will avoid using a model or external tool entirely.

    I believe it is possible to use messages instead of containers to populate lookup tables.

    This way, instead of using a model or kvinit, one may simply push the messages being used as a lookup table and fetch those from whatever model needs them.

  • #13198 Reply

    Archived_User36
    Participant

    It is true that we can build messages or containers and write them with a model to the EnvContext even today. This is good… don’t get me wrong, but what I am thinking here is adding additional “parties” to the mix… e.g., the twitter demo writes to its Cassandra table with the fields it naturally knows in its application. That information in turn is pulled as needed by the models through the EnvContext interface either in total or perhaps by a projection of the fields in the table-able storage.

    As mentioned in my previous post, this requires some work. What it kind of means is that a kamanja application has a way to integrate more tightly with web applications (for one). The alternative is to push work to the web application, construct messages, push them to the queue, build a model that ingests them, and then build the model(s) to consume them.

    I guess all I am saying is that supporting tables in the storage can ease integration into the application being built of which kamanja is an important component managing but not the only one.

  • #13199 Reply

    Archived_User79
    Participant

    The requirement is just not to store/update messages, but have ability to create new messages by doing transformations on incoming messages and store them in containers/messages. If we had fully flushed out Output message implementation, it would be easier (in that case, models don’t need to directly save). Until that is available, best option is to have models take ownership of saving/updating containers.

    Thanks

    -Krishna

Reply To: Transformation and Storage Model
Your information: