Reports as a primary way of communication between different departments
In many big companies fast and clear communication between different stakeholders is a key for sustainable growth. One of the most popular ways to convey information nowadays still are graphical presentations or reports. Indeed, a tremendous amount of data could be crystallized in a couple of plots with KPI figures and relevant comments. However, human efforts are required for building such a report: data gathering, processing, aggregation, putting it in a relevant graphical format. These activities are usually done by analysts, data scientists, product managers, or similar roles. But in the age of automation, the following question arises: why not build Bot once and let previously involved people work on more meaningful tasks for your business?
Problem setup
In this post we will go through the process how to actually implement this kind of Bot on Botprise platform. The setup is next:
- The customer has several office branches from which sales data is gathered from.
- In addition to it, there is data from CRM tool and website logs.
- Once a week, data is uploaded by the customer’s sales team to the dedicated S3 bucket in Botprise cloud.
- After upload of all necessary files, workflow for report generation should be triggered.
How it could be accomplished within the platform?
Add trigger to workflow
- Let’s start from the beginning – how to trigger workflow on files upload. In the Workflow Studio there is a section – Workflow triggers.
- User should select “On files upload” in the dropdown menu,
- Add S3 bucket, folder, where files will be uploaded, file names and periodicity. Once workflow is saved, successful uploading of all files in a particular location with required suffixes (as shown in UI) will trigger the workflow.
Aggregation block
Automation Unit for merging several csv files with same columns into one file (merge different business branches’ sales data into one datasheet).
Deduplication block
Decision Unit, which represents ML/AI algorithms processing block. To create it, users need to go to Developer Workbench and train DU with provided data. Pre-trained DUs by Botprise are available as well. After deployment of DU in the end of DU creation process, it would be available in the list of workflow items.
Particular DU used in this workflow is based on Hidden Markov Random Field, which is generalisation of HMM. K-means is applied on top of it, which gives us an algorithm that groups similar rows together. For example, if in different reports clients’ company name was written differently, this DU makes entity canonicalization and produces data with canonicalized entities. For example, all entries:
- Botprise, Inc
- Botprise Inc
- Botprise company
- Botprise
would be substituted by Botprise, Inc. It is possible to add additional logic there, such as aggregation on other fields.
DUs’ input and output fields are specified during creation as well, which is also displayed in the workflow. In current case, csv file address and corresponding bucket address are specified, as well as output file and bucket for results are specified.
IP data enhancement block
An Automation Unit which is an enhanced version of whois tool. User selects the bucket and cvs file location where the “ip” column is provided. AU will produce a new file, where ip will be joined with the provided csv.
Report generation block
An Automation Unit. Input for this block is a json file with the report configurations. Inside the configuration, plot types and file locations should be specified.
Emails sender block
An Automation Unit. During AU creation, after choosing AU type, several fields should be configured: topic, Body of email, receivers, bucket and attached files locations.
Graphical view of workflow
Once a user created the listed above blocks, (s)he could build workflow from them in Workflow Studio. Picture below represents how it looks like with the current UI.
How report looks like
Below is an example of a 3-pages report on a real customer data. For available types of plots and examples of config files, see documentation.
Received email
Here is an email received from the bot after execution of all steps:
What’s next?
There are plenty of areas of potential improvements to this workflow even with current features available, but we decided not to overcomplicate things in this post. List of such features includes:
- Add function to repeatedly pin via email or slack employees responsible for file upload – every week (or other time interval) they will get the reminder to upload files with direct link to upload.
- Add other processing steps
- Add more plots/text to plot
- Use ML to filter website visits
Our development plan includes adding new features as well:
- Use AI/ML to gather interesting properties of data automatically
- Have a tab on the platform with drag-and-drop report building
If you have suggestions or questions, use cases needed to be implemented – please reach us via support@botprise.com