Data Retention
XCALLY section | Plugins → Data Retention |
On this page |
Overview
The Data Retention plugin can be installed on the XCALLY server to manage the database without requiring advanced system expertise.
With a defined time interval for data retention, the plugin can automatically:
Delete, backup, or copy data stored in XCALLY database tables
Manage files such as voice recordings, voicemails, emails, and interaction attachments
This makes storage management simple and efficient, helping you to:
Prevent storage from running out of space
Avoid service interruptions
Organize and handle historical data properly
Please note that a dedicated license is required to use the Data Retention plugin. For further details, refer to the price list or contact your sales account.
Benefits and target markets
Automatically manages XCALLY database
Schedules backup and data transfer activities
Schedules and performs deletion of redundant data
Optimizes database performarce
Reduces systemic expertise for task planning
Data retention can be a very useful feature, applicable to different examples of target markets:
BPO | FINANCIAL INSTITUTIONS |
For efficient management of voice recordings, emails, and interaction attachments to avoid running out of storage space and ensure continuity of services | To securely store and manage sensitive customer communications in compliance with data protection regulations and reduce the risk of breaches |
HEALTH CARE | E-COMMERCE |
For managing patient call records and sensitive data in compliance with privacy regulations, ensuring that obsolete information is properly archived or deleted. | For archiving customer communications and order confirmations, improving historical data management for future analysis without compromising storage space. |
UTILITIES |
|
For managing customer communications for service requests and emergencies, optimizing storage space and ensuring continuity of services. |
|
Why choose data retention?
Efficient database storage management: prevents databases from filling up with old or redundant data, ensuring uninterrupted services and easier historical data management;
Automated processes: simplifies tasks like deleting, backing up, and copying data, reducing manual effort;
Prevention of storage issues: Helps avoid running out of storage space, ensuring smooth operation and improved database performance.
Installation
To install Data Retention plugin:
Upload the Data Retention zip archive through the admin interface on XCALLY Motion V3, from the AppZone → Plugins section.
After the upload is complete, you can find the Data Retention plugin row. Click on the three dots button menu and select Install plugin.
Go under Plugins → Data Retention.
In the Configuration section of the Data Retention plugin, you need to configure the database and the server details. Insert the below information:
credentials for motion2 database connection
path of the folder where the database data is going to de dumped
details of your server, including your server reachable url and an admin api key
As alternative, it is also possible to manually edit the config.json file in the zip archive.
Restart the plugin from the AppZone → Plugins section after saving the configuration!
IMPORTANT: Every time you change the database configuration, remember to enable also the Seed, since this allows the plugin to fill the database with the required configurations needed to work properly.
The seed will be deactivated automatically every time the system tries to create those configurations.
TL; DR: Enable the Seed during the first configuration.
How it works
Below how the Data Retention works:
You configure one or more automations, routines that are executed periodically, based on an interval that can take one or multiple values;
For each automation, you can create multiple actions, that will be executed sequentially, when the specified interval for that automation is met. The actions are queued, so the next action is not executed until the previous one has not finished or failed. The queue is common through all the automations, so there is no possibility of two actions of two different automations to be executed at the same time, since all the actions are put in the same queue;
Each action can be based on a table configuration, so for every table, there are specific operations allowed. For example, for some tables you can only dump or delete rows, while for others you also have files, and you can copy/delete/link them;
If the default table configurations are not enough, it’s always possible to create a custom table configuration, specifying your table and then choosing the allowed operations;
Every time the rows from a table are dumped, a dump file is created, and you can download or delete it;
Everything that happens in background is recorded and can be checked in the log files.
Automations
The Automations view allows you to manage your automated routines. From this tab you can create new automations or modify and delete them.
To create a new automation:
Go under Plugins → Data Retention
Select Automations tab
Click the + button
Enter the details:
Name: The name of the automation
Interval: When the automation is to be executed. You can choose the basic unit of repetition (Hour, Day, Week, etc..) and then you can select several sub-parts of that unit (e.g., every Hour at both 0 and 30 minutes)
Skip actions if still in execution: As mentioned before, the actions are all queued in a single queue, and executed after the previous one is completed.
In a remote case, it could happen that, while the action is still waiting to be executed, the automation interval is respected and the system tries to insert the action into the queue again. If you want to avoid placing the action in the queue while it is still to be executed, you can switch on this option.Enabled: activate/deactivate the automation
Click Add Automation
Every time an automation is executed, the Last Exec field is automatically updated.
In addition, the Interval column shows when the next execution of the automation is scheduled to take place.
After you create an automation, it’s possible to quick edit it by clicking on 3 dots menu of the specific automation and edit it.
Actions
Under Plugins → Data Retention → Automations in the Actions tab, you can create, modify, or delete automation actions.
When the specified interval for an action is reached, it is added to a queue and executed sequentially. The Last Exec column is updated each time the action is executed.
To create a new Action:
Go under Plugins → Data Retention
Select Automations tab
Click the three dots button menu next to the Automation of interest and select Edit
Go under Actions tab
Click the + button
Enter the details:
Table Configuration: table configuration you want to use for this action, which includes the table and the types of operations you can perform for that table.
You can choose between the default table configurations or the custom ones (see Custom Table Configurations)Retention Actions: The operations that are allowed for the specified table. Depending on which operations you select, you’ll have additional fields displayed. The possible actions are:
Remove table rows: Remove the selected rows from the table
Dump Table rows: Dump the selected table content into a sql file
Remove files from disk: Remove the files related to each one of the selected rows on the table
Backup files on disk: Backup the files related to each one of the selected rows from the table
Backup files, create symlink to new files and remove: The same as ‘Backup files on disk’, with the addition that the original file is removed and in its place a symbolic link is created pointing to the backup file (The action automatically includes the backup and the removal part
Destination File Path: If you select an operation that includes the backup of the files, you’ll need to insert the destination folder where your files will be copied.
If you have a network resource, simply mount it as a partition locally and configure it appropriately according to the cloud you use
Keep last: How much data to keep in the database (to be ignored). Basically, as it is a data retention tool, you specify how much data you want to keep, while the remaining data is processed by the application, dumped, removed etc.. (e.g., if you want to dump all data starting from 1 month ago, you can insert 1 Month)
Work on data range: You can choose to work on a limited data range, instead of working on all data prior to the date specified by the Keep Last setting.
If you enable this, you’ll see two new input fields for the Work on setting, with which you specify how much data to consider going back from the Keep Last setting.Conditions: possible conditions to be considered when querying the table to extract the data to be processed. You can leave it blank if you wish to process all data.
Custom Table Configurations
Under Plugins → Data Retention → Custom Table Configuration, you can customize table configurations. This is useful, for example, when the default table configurations available for your actions are not sufficient.
From this section, you can modify, or delete a custom table configuration clicking on the three dots button menu.
If a custom table configuration is currently used in one of your actions, deletion will not be allowed unless you first remove the association or delete the action.
To create a new Custom Table Configuration, follow the below steps:
Go under Plugins → Data Retention → Custom Table Configuration
Click the + button
Enter the below information:
Name: name of table configuration
Custom Table: enable this option if you want to use a table different from the default ones in the Motion2 database.
When enabled, you can also specify the Date Search Column, i.e. the column that will be used to perform retention operations when filtering table data.Table: name of database table you would like to manage
Supported Retention Actions: the actions supported by this configuration (e.g., backup rows, remove rows, backup files, etc...).
Original File Path: if you enable one or more retention actions that require handling files, you must specify the file location.
This field supports table column autocomplete, so you can enter the name of a column from the selected table, and its value will be used as the file path.
You can do this inserting the name of the column between double curly brackets like this: {{column_name}}.
Original File Path Base Folder: if you enable one or more retention actions that involve handling files, and your files are organized in a subfolder structure that you want to preserve in the backup folder, you can specify the base path of the original folder. The system will then replicate the subfolder hierarchy inside the backup folder defined in the action configuration.
Dumps
When table data is dumped, the system generates a file with the .sql extension in the folder specified in the plugin configuration tab. You can view the complete list of all table dumps under Plugins → Data Retention → Custom Table Configuration.
Data dump file names follow this format:
Name of the dumped table
Upper datetime limit of the extracted data
Datetime of when the data was extracted"
From the three-dots menu on the right, you can download or delete the dump file, or restore it. Please note that while a restore is running, no other restore operations can be performed until the active one is completed.
Restores
Under Plugins → Data Retention → Restores section, you can view the list of restores with the relative status (such as running, error, completed).
In case of error, you can click on 3 dots button and select “Show error” to get more details about it.
Logs
The application stores log files for each component or background function executed. You can access these logs under Plugins → Data Retention → Log, where you can view the last update time for each log, download it, or delete it.
The logs allow you to check the proper functioning of the application, and to eventually check for errors.
You can find logs in folder /var/log/xcally/data-retention
As mentioned, the log files are organized by component and by type, and can be of four types:
<component_name>-combined.log, includes the list of all complete logs for that component;
<component_name>-combined.date.log (e.g. api-combined.2024-01-01.log) : these files gather the log details for the specified component, including both debug and error messages, and for the day specified in the date part of the name. The logs are rotated daily, meaning that you’ll have a new combined log file for each component for each day. The rotation limit is set to 30 days, so after 30 days the oldest logs will be deleted;
<component_name>-error.log, includes the list of all error logs for that component;
<component_name>-error.date.log: these files gather the log details related to errors only, and for the day specified in the date part of the name. The logs are rotated daily, meaning that you’ll have a new combined log file for each component for each day. The rotation limit is set to 30 days, meaning that after 30 days the oldest logs will be deleted.
Troubleshooting
If the plugin is not working, you can check whether it is active. Navigate to Settings → System → Processes and verify in the list if the plugin is running (ONLINE label).
If not, go to App Zone → Plugins, open the plugin list, click the three-dots menu of the specific plugin, select Uninstall plugin, and then Install plugin again.
Please note that uninstalling only stops the processes; it does not remove the plugin itself, so all previously entered configurations are preserved.