Automatic Insights, far beyond dashboard

Do you need insights on customers, products, finance… quick, understandable, actionable business responses… many promise, very few deliver it… We can help you, read on and decide.

Insights? We Already Have Dashboards

Those of working on data analytics know this well: beyond statistics, mathematics, and machine learning, we need a technique as old as humanity itself: storytelling.

Ultimately, a productive data analysis session is about crafting a narrative in business knowledge terms that we can convey to decision-makers. A narrative, of course, backed by data and analytics. Not just any data, but those that provide relevant knowledge leading to action. What is often called Insights. In short: organizations need insights to build a data-driven narrative that support their decisions.

If we google ‘insights,’ we’ll find interesting definitions and many resources on how to obtain them with Business Intelligence. These Business Intelligence tools are now in every office. They offer self-service data analysis for business experts and have evolved in flexibility, performance, data integration, visualization versatility… All BI tools promise this—relevant knowledge, insights—but do they deliver?

For a data point or discovery to be considered an insight, it is understood that:

• It must be unique, distinct from the obvious or trivial in the business
• It must be useful to improve a business problem or create something
• It must lead to a decision

In short, an insight is the result of transforming data into a type of actionable knowledge that helps companies define strategic and action plans.

Do We Use Dashboards? Well, There’s One Guy in Marketing and Another in Finance…

Dashboards remain ‘reactive’ in data exploration: the user must select objects, filters, metrics, dates to obtain a chart that could lead to discovering relevant knowledge. And thus, by launching hypotheses, validating, or refuting them with data, they search, like for a gold nugget, for that insight on which to build the narrative.

However, the possible combinations of attributes, metrics, times… are enormous and grow exponentially as we enrich the dashboard and expand the data feeding it. The user then suffers from data overabundance, especially when they have less time to explore systematically and decision-making must be faster. A common issue of our times.

The reality is that dashboards end up in the hands of employees with more time and technical skills, who explore them as far as time allows, often inconsistently and without the strategic vision of the executive. A loop: more and better data, which takes more time to explore, time that is increasingly scarce. And the promised insights? Somewhere over there, buried in data.

That’s Right, I Need an Assistant Good with Numbers… and Patient

And this is where we, unica360, enter the story. Our specialty, beyond designing and setting up dashboards—which we also do—is extracting relevant knowledge through advanced analytical techniques. And we ask ourselves, why let the user experiment with attribute, metric, and time configurations based on their intuition or experience when we have mathematics to tell us when something is unique? Let the algorithms tell us what to look at.

And this is what we call automatic insight extraction. That is, using Machine Learning algorithms to shed light on historical or quasi-real-time data to extract this knowledge. Let the ‘AI machine’ separate the gold nuggets and present them to the users. Note that this approach is substantially different from other applications where a black box is created to generate predictions—based on Machine Learning, Deep Learning—which we often cannot explain (which we also do and is valuable). Here, it is essential that the resulting rules are interpretable.

Automatic insight extraction only proposes to the user, presenting realities that stand out from the obvious, like pieces that fit together, but the final conclusion results from human reflection. The user can now combine these data pieces, forming a mental image of correlations, trends, or anomalies to build their picture, turning them into risks, threats, opportunities, and creating a narrative. Their storytelling.

Automatic insight extraction is, in this sense, an excellent assistant with numbers, and very productive. As a concept, it aligns with the role of the copilot, popularized as a generative AI application, but it is more transparent and, therefore, trustworthy. The machine does the initial data screening so the human can further filter those gold nuggets that serve their current task.

I’m Interested in These Insights, What Are They Like?

At unica360, we have been working along these lines for years, initially without having conceptualized it yet in the semantics of insight. Over many years of customer intelligence consultancy, segmentation, predictive analytics based on Machine Learning, we have found that:

• Many organizations need transparency in the predictive systems we have developed for them—black boxes based on machine learning—and to make them interpretable
• In other cases, they are high-level profiles who do not have time to search for relevance in the corporate dashboard, as described earlier
• Or technical staff, account managers, and product managers who need clues to the underlying causes of changes they see at an aggregated level in their KPIs—okay, the trend has changed, but… who, what pulls the average in the customer and product mix?

This demand led us to develop an analysis methodology, our automatic insight extraction, which essentially always includes:

• A “subject, verb, predicate” phrase
• Modeling of relevance
• Navigation through results

Of course, this is a generalization; the final format, presentation, or language of the proposed insights varies across different projects depending on objectives, user profiles, explanatory variables, and available targets.

Alright, So How Does This Automatic Insight Extraction Work?

Our solution is based on KPI time series restricted to segments of the overall sample. For instance, the KPI could be visits to a URL associated with a product, say a bicycle, and the segment could be “women under 25 from Barcelona reached through the Christmas campaign.”

We differentiate between static and dynamic insights:

• Static insights: provide information about the average correlation/propensity of the segment to the KPI recorded over a long period T -> not based on temporal evolution
• Dynamic insights: inform us of unexpected fluctuations or changes in recent time sales (W) -> temporal evolution is the basis for identifying deviations from the expected series

Insight Structure: Subject, Verb, Predicate

We classify the recorded behavior so that the 3 elements “segment, behavior, and KPI” follow the structure of “subject, verb, and predicate,” resulting in what we call an insight, for example: women under 25 from Barcelona reached through the Christmas campaign (2,376 records) have sharply increased their inquiries about bicycles (8.9% over an expected value of 2.4%).

Insights Relevance

We model the relevance of each insight by integrating the extent of the alteration in the series, the segment size, and a subjective weight associated with each KPI that adjusts according to user feedback.

The interesting thing about the relevance indicator is that it allows us to rank and select the most important elements from a set of many thousands of results generated by algorithms, which would be unmanageable manually by a human.

Navigating Results, User-Selected Insights, and Feedback

Additionally, we can navigate through this vast set of insights by filtering by segment or KPI in cases where focused searching is necessary. As part of this human filtering, an interface can enable the user to rate and provide feedback on the relevance of each insight. The analytical engine that assigns relevance can learn from user feedback, improving relevance assignment in the future.

The following image shows an insights rules interface in the context of customer behavior in the insurance sector:

And here is a similar one, analyzing massive tourist mobility data at the municipal level. The algorithms ‘bring to light’ unexpected and relevant behaviors among these tourists, representing high-value discoveries for the analyst.

And here the time series for tourists from Switzerland to Cádiz, where we can see how the real series far exceeds the prediction based on historical data, with its seasonality, since it spikes from July 2023:

And the same for German tourists in Valladolid:

I Like These Insights, How Can I Use Them?

Our automatic insight extraction engine functions as an API and integrates with the dashboard of virtually any BI software. As always in data analytics, adaptation to context and business rules is critical, and each implementation includes expert Data Science consultant support to ensure this fit.

This way, every morning, the business user can check the insights generated for them by the system, with a coffee, as they finalize their day’s agenda. Quick, easy, relevant, actionable, in business language… these are genuine insights, generated automatically. And thus, we fulfill what we promised :-).

If you want to start obtaining automatic insights, tell us about your case, and we’ll help you.