Replication Process
The following picture illustrates how TMS Echo performs data replication.
The whole process is detailed here:
Every database operation the application performs through TMS Aurelius is captured and logged, creating a "Change Log" in the database.
2. Routing
For each node (database) registered as a remote node (i.e., that should receive data from the source code), the router creates an outgoing batch with the log of changes. From the original change log, each outgoing batch can will receive its own change log, which can be the same or a subset of the original one, so each target node will have its own set of data to receive.
3. Transfering Data (Push/Pull)
This step is the communication with the remote node. Or, in other words, is how the batch is transferred from one database (node) to another. The transfer can be requested by the application being executed in the source code (which will be a "Push" to target node) or can be requested by the application being executed in the target node (which will be a "Pull" from the source node). The way data is transferred depends on how the IEchoRemoteNode interface is created. It can be through a TMS XData Rest/Json server, or can be direct database connection (i.e., the application in the source node access the database in the target node directly).
The data transferred will be come an incoming batch at the target database (node).
This is the step where all the incoming batches are read, the change log in the batch is processed and for each item in the log the respective operation is performed in the target database. In other words, that's when the data is effectively updated in the remote database.
Logging Changes
For TMS Echo to replicate data correctly, it needs to log all changes made to the database. Strictly speaking, TMS Echo replicates changes, not data. So logging the changes is a crucial thing for it to work. Also, Echo only logs data changed through Aurelius, so all your database manipulation must be done using TMS Aurelius. If you perform any operation directly in database with using Aurelius, or if you do not subscribe the events to log changes made by Aurelius, such data will not be replicated.
To log changes for replication, TMS Echo relies on TMS Aurelius event system. You need to create the Echo listener and subscribe it:
uses {...}, Echo.Listeners;
// Subscribe listeners at the begining of your application,
// before any database operation is performed
// The subscription is made by TMappingExplorer. In this case we're
// subscribing to the default mapping explorer (calling SubscribeListeners without params)
FEchoSubscriber := TEchoEventSubscriber.Create;
FEchoSubscriber.SubscribeListeners;
// At the end of application, unsuscribe the listeners,
// this will stop logging
FEchoSubscriber.UnsubscribeListeners;
FEchoSubscriber.Free;
Keep FEchoSubscriber instance reference alive for the whole time of the application (or at least while logging is important), and that is enough. Remember that Echo will save the log to internal tables it uses, so you must be sure the database schema is updated and has those tables created.
As described in TMS Aurelius documentation about the event system, listeners are registered per TMappingExplorer object. The SubscribeListeners method, when used without parameters, will subscribe to the default explorer (TMappingExplorer.Default). If you have multiple explorers in your application (for example because you use a multi-model design or any other reason), then you have to be sure to subscribe the listeners to all the TMappingExplorer objects used by your application which maps to data that you want to be replicated:
FEchoSubscriber.SubscribeListeners(TMappingExplorer.Get('Finance'));
FEchoSubscriber.SubscribeListeners(TMappingExplorer.Get('Marketing'));
FEchoSubscriber.SubscribeListeners(AnotherMappingExplorerInstance);
and also unsubscribe from all of them:
FEchoSubscriber.UnsubscribeListeners(TMappingExplorer.Get('Finance'));
FEchoSubscriber.UnsubscribeListeners(TMappingExplorer.Get('Marketing'));
FEchoSubscriber.UnsubscribeListeners(AnotherMappingExplorerInstance);
Routing
Routing is the process of processing logged changes and create batches targeting each remote node registered to receive data. This is a more detailed description of the routing mechanism:
Get a list of the logged changes that has not been routed yet (status is "unrouted");
Create a batch for each registered remote node;
Include the list of changes in the batch, so those changes are scheduled to be sent to that specific node;
Flag the list of logged changes as routed (status becomes "routed") so they are not sent again.
Once batches are created, they can be transferred to the remote nodes.
To execute the routing process, you just need to call TEcho.Route method:
Echo.Route;
And that is enough. It's important that you call Route from time to time in your application to create the batches. If you don't do so, batches are never created, and data is never transferred. It's up to you to decide the best way (how and when) you should execute the routing process.
It's also while routing that you can filter which data is transferred to each node. For example, supposed you are routing data at the server database to two different nodes, "Client1" and "Client2", and you don't want to send invoices to Client1. You can implement a custom logic this way:
uses {...}, Echo.Main, Echo.Entities;
Echo.Route(
procedure(Log: TEchoLog; Node: TEchoNode; var Route: boolean)
begin
if SameText(Log.EntityClass, 'AppEntities.TEchoInvoice')
and (Node.Id = 'Client1') then
Route := false;
end;
);
The above code means: if you are routing to the node with id "Client1", and you are trying to route a change in an object of class "AppEntities.TEchoInvoice", then don't route it.
So any change in TEchoInvoice object will only be send to "Client2", not "Client1".
Be aware that routing happens on both client and server side (or in better words, in all nodes) so you need to know exactly the source code (local database) and the target node (Node parameter) to implement your logic.
Transfering Data (Push/Pull)
Once your application is periodically logging changes and routing data, you will eventually have several outgoing batches targeting several remote nodes, waiting to be transferred to them. So you can transfer data to and from remote nodes, using the Push/Pull mechanism.
Here is how Push (to a specific remote node) works:
Get all outgoing batches targeting the remote node that has not been sent yet (status "Pending");
Send each of those batches to the remote node;
An incoming batch with same data will be locally saved in the database represented by the remote node;
The outgoing batch in the source node will be flagged as "Sent", to avoid being resent;
The incoming batch in the target node will be flagged as "Imported", meaning it's ready to be loaded.
Here is how Pull (from a specific remote node) works:
Asks for all outgoing batches in the remote node targeting the local node (status "Pending");
Receive each of those batches from the remote node;
An incoming batch with same data will be locally saved in the database represented by the local node;
The incoming batch in the local node will be flagged as "Imported", meaning it's ready to be loaded.
The remote node will be asked to flag the outgoing batch as "Sent", to avoid being resent;
So in very short words, "Push" sends local changes to remote node, and "Pull" receives remote changes to local node.
To Push or Pull, just call the respective method using the IEchoRemoteNode interface:
var
RemoteNode: IEchoRemoteNode;
{...}
RemoteNode.Push; // push changes to server
{...}
RemoteNode.Pull; // pull changes from server
It's up to your application to periodically call Push and Pull to transfer pending batches from one node to another. If you don't do that, batches will be indefinitely pending to be sent and nodes won't receive data from each other.
Push/pull just transfer batches from one node to the other. It doesn't mean data will be updated in the database. You need to perform batch loading to finish the replication process and effectively have your data present in the database.
Batch Loading
Batch loading is the last stage of the replication process. After an outgoing batch is transferred from from source node to target node, an incoming batch is created in the target node. The incoming batch then needs to be loaded into the database, meaning that the changes in the batch are effectively applied (inserts, deletes, updates are performed).
Batch loading process described:
Retrieve all incoming batches pending to be loaded (status is "imported");
For each pending batch, get the list of changes in the batch;
Apply each change (insert, update, etc.) in the database;
Flag the batch so it won't be loaded again (status becomes "loaded").
To perform the batch load, just call TEcho.BatchLoad method:
Echo.BatchLoad;
It's up to your application to periodically call BatchLoad to apply received changes to the database.
Batch loading is the most critical process because it's where the replication effectively happens, i.e., where data from source node is effectively applied in the target node. This means that this is where errors and conflicts can arise. TMS Echo provides several mechanisms to minimize and solve those problems:
Fallback
When loading a batch, errors might arise because the data being loaded might be incompatible with existing data in the database. To minimize such problems, TMS Echo provides a fallback mechanism that automatically tries to solve some problems.
On Insert
If an Insert operation fails, TMS Echo checks if the object being inserted already exists in the database. If it does, it automatically tries an Update on that object with the same data and continue normally.
Fallback fails if the object does not exist (which means the Insert operation failed due to other reason than a key conflict with existing object) or if the fallback Update operation also fails.
On Update
If an Update operation fails, TMS Echo checks if the object being updated exists in the database. If it does not, it automatically tries to Insert such object again and continue normally.
Fallback fails if the object does exist in the database (which means the Update operation failed due to other reason than the object not existing in the database) or if the fallback Insert operation also fails.
On Delete
If a Delete operation fails, TMS Echo checks if the object being deleted exists in the database. If it does not, it just ignores the Delete operation and continue normally, since object has already been deleted.
Fallback fails if the object does exist in the database, which means the Delete operation failed due to other reason than the object not existing in the database.
If a fallback operation is performed and fails, an EFallbackException exception is raised, and it includes information about both exceptions: the original one which caused the fallback to be tried, and the one that was raised when fallback operation was performed.
Predictive Fallback
The automatic fallback can be predictive or not. By default, it's not predictive, which means Echo will try to execute the load operation normally, and if an exception is raised, then the fallback operation will be tried, if applicable.
You an alternatively set the fallback to predictive. This means for Echo will first check (execute an SQL) to see if the load operation will fail. Then it will already execute the fallback operation, if applicable. Predictive fallback is a must if you are using PostgreSQL, since it doesn't allow a transaction to continue if an exception is raised. To set the predictive fallback, just set the PredictiveFallback to true in the TEcho object:
Echo.PredictiveFallback := True;
Even though predictive fallback avoids raising exceptions in advance and allows fallback operations to be used in PostgreSQL, it makes the batch loading operation slower because it will execute an SQL SELECT statement before each load operation where fallback applies.
Disabling fallback
If for some reason you don't want Echo to perform the automatic fallback, you can disable it by setting the FallbackEnabled
property to false:
Echo.FallbackEnabled := False;