Data Discovery 2.0: Going Further Than Traditional Analytics Tools

by Jonas Poelmans, MSc, PhD



Posted on November 15, 2017 at 8:00 PM


Data Discovery is different from traditional data mining for several reasons. Traditional data mining techniques often aim at fully automated extraction of patterns from data. Black box techniques may give accurate predictions on the current dataset but are not understandable for the end users and often fail to perform for unseen data presented on future occasions. Traditional data mining techniques often fail in uncovering truly valuable knowledge since they typically focus on statistical patterns that explain the mainstream of the data records. Often, however, real knowledge is hidden between the anomalies and exceptional cases found in the data. A human expert cannot sift through millions of data records manually but he disposes of unique knowledge about the domain under investigation.

Combining the computational power of a computer with the creative power of the human mind

Data Discovery aims at synergistically combining the computational power of a computer with the creative power of the human mind. In his book “How to Create a Mind”, the famous computer scientist and futurist Ray Kurzweil postulated that partial order structures such as concept lattices are one of the few mathematical techniques capable of bridging this gap. Our software engages the user with intuitive visualizations to explore the data and steer the discovery process. This offers him insight into the patterns describing the bulk of the data but also allows him to selectively pick out smaller patterns for deeper analysis. The user may also let the software discover patterns automatically to explain certain phenomena in the area under investigation of interest for the user.

“Unknown Unknowns”: the hidden pitfall of traditional data mining techniques

In February 2002, Donald Rumsfeld, the US secretary of State for Defense coined the notion of “unknown unknowns”: “there are things we do not know we don't know”. We strongly believe that a structure should not be preimposed on the data because this may lead to a loss of crucial information valuable to the knowledge discovery process. Our models allow for the visualization of emerging patterns in the data and are all characterized by their rich semantic expressiveness. This allows the user to look at the data with an open mind and minimal modeling efforts by the user before the start of his knowledge discovery exercise. We therefore believe that we dispose of a unique product for discovering “unknown unknowns” and to analyze them in full depth in real-time.

“There are things we do not know we don't know.”

Donald Rumsfeld

Interactive data discovery solution: Clarida Excalibur

The Clarida ExcaliburTM Platform offered by Clarida Technologies Ltd. is a data analytics software system based on state of the art techniques from the fields of combinatorial algebra, lattice theory and formal concept analysis. The system was developed to identify hidden patterns in large volumes of high dimensional data. The system offers the user the ability to interactively explore the data in a semi-automated and fully automated manner. The software will identify profiles consisting of early warning indicators and possible red flags which can be used later on for early recognition of successful or risky cases. It is important to stress that each of these patterns are fully transparent and their performance is exactly known for past cases. Any source of textual, web, social network or numeric data can be analyzed with the software.

We look forward to hearing your comments and feedback. And welcome your visit to our website: www.claridatech.com. Thank you for reading!


Jonas Poelmans, MSc, PhD

Jonas Poelmans is in charge of Data Analytics at Clarida Technologies.


Have a question?

We'd love to hear from you, why not try one of the following ways:
EMAIL: info@claridatech.com
PHONE: +44 203 5144 631
ADDRESS: Regent Street 207, London, United Kingdom

Organization-wide Intelligence

Learn how to bring everything together and become an intelligent organization:
Read More