Data Cycle

This page describes in detail how the client produced data is distributed in the Scolvo Development Platform.

The data change distribution is an essential topic in Scolvo Development Platform as it makes possible to:

  • Spread the project specific information changes among the central components and connected clients,

  • Be reactive on the client UIs, if the business use-case request it,

  • Give prompt information about the changes to the connected external systems.

There are two main cases for data changes described here in details:

  • Client data change flow,

  • And Backend data change flow.

Client data change flow

In this flow a user of a connected client make changes in the local database and sends this information towards the Backend component too. The Backend receives the Data Change Request and triggers the appropriate events before and after executing the changes in the Backend database, and finally distributes the changes to all connected clients (including the originating one). The connected clients finally receive the description of the changes and react according to the implemented business logic (they can accept and execute or ignore the changes). Below you can find code examples for each step with hints and descriptions.

Step 1: A client makes changes

It is a usual case, that based on a UI action (e.g. push the Save button) the client creates a new data to be inserted locally, and it sends the change to Backend at the same time. In our example when the user activates AddClientDataChange button, then the SVM invokes the onAddClientDataChange() event, where the data changes are implemented. In this example case we use the note table defined in Playground and use it for our actual purpose extending it with unique data.

The client creates new data and handles it on ClientDataChangePage page
 1import {
 2  /mobile/repository/NoteRepository
 3}
 4
 5function displayClientDataChangePage(originId) {
 6  debug("Display ClientDataChange Page ...");
 7  var dataRows = clientDataChangeGetData();
 8
 9  var pageData = {
10    "ClientDataChangePage": {
11      "ClientDataChangeList": dataRows
12    }
13  };
14  display(ClientDataChangePage, pageData, originId);
15}
16
17function clientDataChangeGetData() {
18  return getNotes().map(function (noteDao) {
19    return {
20      "id": noteDao.id,
21      "title": noteDao.title,
22      "content": noteDao.content
23    };
24  });
25}
26
27function clientDataChangeRefreshList(originId) {
28  fireEvent(createRefreshItemTargetEvent("ClientDataChangeList", getNotesData()), "ClientDataChangePage");
29}
30
31function clientDataChangeRemoveIdFromList(recordId, originId) {
32  fireEvent(createRemoveItemTargetEvent("ClientDataChangeList", recordId), originId);
33}
34
35function onClientDataChangePageLoaded(originId) {}
36
37page ClientDataChangePage {
38  layout: vertical;
39  template: general;
40  settingsVisible: true;
41  scolvoMenuVisible: true;
42
43  list ClientDataChangeList {
44    template: listVerticalNormal;
45    filterVisible: true;
46    span: 0;
47    paged: false;
48    itemTemplate: listItemMultiLine;
49
50    actions: [
51      AddClientDataChange
52    ]
53
54    columns: [
55      mainText => title,
56      subText => content
57    ]
58  }
59}
60
61function onClientDataChangeListFilterCount(originId) {
62  return clientDataChangeGetData().size();
63}
64
65function onClientDataChangeListSelectionChanged(originId) {
66  debug("On list selection changed: " + $IN.data);
67  // Do nothing actually
68}
69
70function onAddClientDataChange(originId) {
71  var newTitle = dateToString(nowInMillis(), "yyyy.MM.dd. HH:mm");
72  var newContent = "The content of " + newTitle;
73  var dataDao = {
74    "id": uuid(),
75    "title": newTitle,
76    "content": newContent,
77    "changeType": "INSERT"
78  };
79
80  insertTypeDefinition("note", dataDao);
81  sendDataChange("note", [dataDao]);
82  fireEvent(createAddItemTargetEvent("ClientDataChangeList", dataDao), "ClientDataChangePage");
83}

The following main actions are done in the onAddClientDataChange() event implementation:

  • A DAO object is created with a unique id and details,

  • The local database is changed by the insertTypeDefinition() function call,

  • The sendDataChange() function call sends the change detils to the Backend for processing,

  • The fireEvent call with the created AddItemTargetEvent object refreshes the UI and populates a new item in the list.

Step 2: The Backend receives the change

When the Backend receives a Data Change Request, then for each row it executes a pre, and post change event with appropriate name. The event names are calculated based on the changeType and on the type definition name (table name). As in our example the note table is used, the following events are in focus:

  • onNoteInserting(originId) - event executed before changing the Backend database (pre INSERT script), it can block the insert if failed

  • onNoteInserted(originId) - event executed after changing the Backend database (post INSERT script), executed asynchronously, no effect on database operation

  • onNoteUpdating(originId) -event executed before changing the Backend database (pre UPDATE script), it can block the update if failed

  • onNoteUpdated(originId) - event executed after changing the Backend database (post UPDATE script), executed asynchronously, no effect on database operation

  • onNoteDeleting(originId) - event executed before changing the Backend database (pre DELETE script), it can block the delete if failed

  • onNoteDeleted(originId) - event executed after changing the Backend database (post DELETE script), executed asynchronously, no effect on database operation

In these event hooks it is possible to implement customer specific logic according to the requirements (e.g. sending information about the change to an external system). In the actual example an email is sent to the creator user with the new record details. This kind of event definitions are usually placed to TypeDefinitionLifeCycle.scolvo under the backend script directory.

The context of the event has the following information (the context can be accessed through $IN.data map):

  • clientId - the unique ID of the client,

  • createdBy - the ID of the creator user in User table,

  • messageCreated - the time stamp of the Data Change message,

  • language - the used locale of the client (browser or mobile client),

  • map - the DAO object of the change in a Map<String, Object>

The creator user deatils are retrieved by the createdBy ID containing name and email for the sendEmail() built-in function call.

Backend data change events for note table in TypeDefinitionLifeCycle.scolvo
 1// Note related data change event definitions:
 2function onNoteInserting(originId) {}
 3
 4function onNoteInserted(originId) {
 5  var noteDao = $IN.data.map;
 6  debug("The request information is: " + $IN.data);
 7  var creatorUser = getUserById($IN.data.createdBy);
 8  if (creatorUser != null) {
 9      debug("The creator user's email is: " + creatorUser.email);
10      sendEmail(
11          getEnv("EMAIL_USERNAME"),
12          creatorUser.email,
13          "",
14          "New Note data created",
15          "You have created a new data: " + noteDao + ". Have a nice day!",
16          null);
17      debug("The email has been sent successfully");
18  }
19}
20
21function onNoteUpdating(originId) {}
22
23function onNoteUpdated(originId) {}
24
25function onNoteDeleting(originId) {}
26
27function onNoteDeleted(originId) {}

Step 3: The Backend distributes the change to the clients

When the data change is executed in the central database the Backend sends a Data Change Datagram message to the clients. The message sending is usually parallel to the post event execution in the Backend component.

Step 4: An online client receives the change

Every online client gets the Data Change Datagram message, and a specific event with name onDataChangeDg(originId) executed to process it. This event is usually placed to DataChange.scolvo file. The implementation of the event has to reflect the business requirements. In our case the mobile script set (mobile apps and the Webapp client) logic is to handle note table changes and execute them on local database. This event is the place to handle the reactive behavior of the system if required. In this case it is implemented to refresh the list on our page.

Note: The built-in function insertOrReplaceTypeDefinition() called to execute INSERT database operation as the originating client receives the changes too and the note table already contains the data.

The onDataChangeDg(originId) event imlpementation in mobile/DataChange.scolvo script file
 1import {
 2  /mobile/SessionUser,
 3  /mobile/repository/NoteRepository,
 4  /mobile/example/ClientDataChangePage
 5}
 6
 7function onDataChangeDg(originId) {
 8  debug("Data change message arrived ...");
 9  if (getCurrentUser() == null) {
10    return null;
11  }
12
13  var changeset = $IN.changeset;
14  var typeDefinition = $IN.typeDefinition;
15  debug("Data change message arrive with type definition: " + typeDefinition + ", size: " + changeset.size());
16
17  if (typeDefinition == "note") {
18    changeset.each(function(dao) {
19      if (dao.changeType == "INSERT") {
20        insertOrReplaceTypeDefinition(typeDefinition, dao);
21      } else if (dao.changeType == "UPDATE") {
22        updateTypeDefinition(typeDefinition, dao.id, dao);
23      } else {
24        deleteTypeDefinition(typeDefinition, dao.id);
25      }
26    });
27    clientDataChangeRefreshList(originId);
28  }
29}

The event context has the DataChangeDg object, what can be accessed through the context by the $IN expression. The object has following accessible keys:

  • typeDefintion - the name of the database type,

  • typeDefinitionVersion - the version of the change, time in milliseconds of UTC time-zone,

  • changeset - collection of data rows containing the changeType describing the desired database operation (INSERT, UPDATE, DELETE).

When updating or deleting data by the clients

When the business logic requires update or deletion of a business object, the example code above has to be modified. For the update operation the changeType is UPDATE for the delete operation the appropriate changeType is the DELETE. The built-in function for refreshing the list of the entities is different too: for the update case the createReplaceItemTargetEvent(), for the delete case the createRemoveItemTargetEvent() built-in function has to be used.

Changes for the update and delete cases

A set of row actions is introduced on the list data to be able to do specific operation on the selected row, as these (e.g. the update and delete) operations require the unique Id of the data. To be able to show these row actions the layout of the list is switched to listItemCard, and a new set of row operations and clicked events are introduced.

Changes for Update and Delete operations in the ClientDataChangePage.scolvo
  1import {
  2  /mobile/repository/NoteRepository
  3}
  4
  5function displayClientDataChangePage(originId) {
  6  debug("Display ClientDataChange Page ...");
  7  var dataRows = clientDataChangeGetData();
  8
  9  var pageData = {
 10        "ClientDataChangePage": {
 11        "ClientDataChangeList": dataRows
 12    }
 13  };
 14  display(ClientDataChangePage, pageData, originId);
 15}
 16
 17function clientDataChangeGetData() {
 18  return getNotes().map(function (noteDao) {
 19    return buildClientDataChangeListRowFromDao(noteDao);
 20  });
 21}
 22
 23function clientDataChangeRefreshList(originId) {
 24  fireEvent(createRefreshItemTargetEvent("ClientDataChangeList", getNotesData()), "ClientDataChangePage");
 25}
 26
 27function clientDataChangeRemoveIdFromList(recordId, originId) {
 28  fireEvent(createRemoveItemTargetEvent("ClientDataChangeList", recordId), originId);
 29}
 30
 31function onClientDataChangePageLoaded(originId) {}
 32
 33page ClientDataChangePage {
 34  layout: vertical;
 35  template: general;
 36  settingsVisible: true;
 37  scolvoMenuVisible: true;
 38
 39  list ClientDataChangeList {
 40    template: listVerticalNormal;
 41    itemTemplate: listItemCard;
 42    filterVisible: true;
 43    span: 0;
 44    paged: false;
 45
 46    actions: [
 47      AddClientDataChange
 48    ]
 49
 50    columns: [
 51      title => title,
 52      text => content
 53    ]
 54  }
 55}
 56
 57function onClientDataChangeListFilterCount(originId) {
 58  return clientDataChangeGetData().size();
 59}
 60
 61function onClientDataChangeListSelectionChanged(originId) {
 62  debug("On list selection changed: " + $IN.data);
 63  // Do nothing actually
 64}
 65
 66function onAddClientDataChange(originId) {
 67  var newTitle = dateToString(nowInMillis(), "yyyy.MM.dd. HH:mm");
 68  var newContent = "The content of " + newTitle;
 69  var dataDao = {
 70    "id": uuid(),
 71    "title": newTitle,
 72    "content": newContent,
 73    "changeType": "INSERT"
 74  };
 75
 76  insertTypeDefinition("note", dataDao);
 77  sendDataChange("note", [dataDao]);
 78  fireEvent(createAddItemTargetEvent("ClientDataChangeList", buildClientDataChangeListRowFromDao(dataDao)), "ClientDataChangePage");
 79}
 80
 81function buildClientDataChangeListRowFromDao(dataDao) {
 82  return {
 83      "id": dataDao.id,
 84      "title": dataDao.title,
 85      "content": dataDao.content,
 86      "actions": ["ModifyClientData", "DeleteClientData"]
 87    };
 88}
 89
 90function onModifyClientData(originId) {
 91  debug("Modify data id is: " + $IN.data.id);
 92  var newTitle = $IN.data.title + " .";
 93  var newContent = $IN.data.content + " .";
 94  var dataDao = {
 95    "id": $IN.data.id,
 96    "title": newTitle,
 97    "content": newContent,
 98    "changeType": "UPDATE"
 99  };
100  updateTypeDefinition("note", dataDao.id, dataDao);
101  sendDataChange("note", [dataDao]);
102
103  fireEvent(createReplaceItemTargetEvent("ClientDataChangeList", buildClientDataChangeListRowFromDao(dataDao)), "ClientDataChangePage");
104}
105
106function onDeleteClientData(originId) {
107  debug("Delete data: is to be deleted is:" + $IN.data.id);
108  var dataDao = {
109    "id": $IN.data.id,
110    "changeType": "DELETE"
111  };
112  deleteTypeDefinition("note", dataDao.id);
113  sendDataChange("note", [dataDao]);
114
115  fireEvent(createRemoveItemTargetEvent("ClientDataChangeList", dataDao.id), "ClientDataChangePage");
116}

Backend data change flow

This data flow is used when a scheduled backend logic, data import or external system trigger results in data change to be distributed among clients. In the below example a periodic Backend trigger produces some data, and a mobile client reacts to that.

Repeated jobs in Backend

The following example defines a periodic data creation for above defined data structure.

Job definition and Backend side produced data distribution to clients in Jobs.scolvo script
 1import {
 2  /backend/Broadcast
 3}
 4
 5function initScheduledJobs() {
 6  addJob("0 */5 * ? * * *", "DataCreationJob", "ProduceData", "CreateData");
 7}
 8
 9//========================== Execute initScheduledJobs
10initScheduledJobs();
11
12function onCreateData(originId) {
13  debug("CreateData scheduled job - START");
14
15  var newTitle = "From Backend - " + dateToString(nowInMillis(), "yyyy.MM.dd. HH:mm");
16  var newContent = "The content of " + newTitle;
17  var dataDao = {
18    "id": uuid(),
19    "title": newTitle,
20    "content": newContent,
21    "changeType": "INSERT"
22  };
23  insertTypeDefinition("note", dataDao);
24
25  publishBroadcast("note", [dataDao]);
26  debug("CreateData scheduled job - END");
27}

As result the Backend produces new data in every 5 minutes, and if there is a connected client with opened Note listing page, then the client refreshes the list when a new data arrives.