Clearpace's NParchive technology is based on a body of research work that created algorithms for the automatic transformation of structured data into a potentially more space-efficient form - a form in which repeated instances of patterns were replaced by pointers to a single memory location where the original pattern was stored.

While theoretically interesting, the work had two main drawbacks which meant that practical exploitation was simply not viable. These drawbacks were:

• The level of de-duplication achieved was dependent upon the order in which the fields were processed. The discovery of the optimal ordering of the fields was known to be NP-hard, meaning that the technique would not effectively scale to many fields.

• The records in their de-duplicated form were rendered effectively unqueryable, and any attempt to query the data necessitated its expansion back to its original form.

Clearpace have addressed both of these problems with a combination of discoveries and innovative solutions that turn this elegant theory into a powerful and deployable mechanism for efficiently storing and querying data.

Clearpace's solutions to these drawbacks are:

• The discovery of a heuristic that allows us to determine a near-optimal ordering of the fields in time proportional to the overall data size. This makes it practical to apply the algorithm efficiently to large data sets and is a work-around for what is essentially an NP-hard problem.

• The discovery of algorithms, equivalent to those in conventional database theory, that resolve queries against the data in its compressed form without the requirement to expand it back to its original form. Furthermore, the fact that the upper bound on the time required to perform these operations (even "sort", "group by" and "join") is directly proportional to the volume of information in the data set gives the solution excellent performance and scalability characteristics.

Clearpace's technology is essentially a high performance, in-memory data store with a very small storage footprint that services queries extremely rapidly and efficiently. Find out more...