Varada provides query acceleration using a combination of indexing technologies to speed up data searches, filters, and joins, together with smart cache management to speed up data access and improve performance.
To do this, Varada creates a full copy of your data catalog. When queries hit tables under the Varada catalog, the Query Acceleration Engine autonomously identifies which datasets to accelerate and how to optimally balance performance and the available storage. It then indexes the relevant columns using a nanoblock indexing mechanism, and considers the data type, structure, and distribution of data in each nanoblock to create the optimal index.
The Query Acceleration Engine also utilizes machine learning to monitor which datasets and columns are frequently used, as well as which part of the data lake needs to be accelerated in order to meet the performance requirements of high-priority workloads. It implements acceleration instructions that dynamically operationalize datasets within your data lake.
Updated 5 months ago