You can use the following REST commands to work with acceleration instructions.

Create an Instruction

Create one or more instructions, each of which is attached to a column in a table.

After an instruction is set, the data needs to be accessed via a query to trigger the warmup. This improves efficiency by ensuring that only data that is required is warmed up. Data that is not required will not be warmed up, even if it falls under the instruction definition.

Synopsis

curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[{
  \"colNameId\": \"<column name>\",
  \"predicates\": [],
  \"schema\": \"<schema name>\",
  \"table\": \"<table name>\",  
  \"colWarmUpType\": \"<COL_WARM_UP_TYPE_BASIC|COL_WARM_UP_TYPE_DATA|COL_WARM_UP_TYPE_LUCENE>\",
  \"priority\": \"<priority value>\",
  \"ttl\": \"<PnDTnHnMn>\"
    },
    {...}
  ]
}
]
[...]"

Mandatory Properties

Property	Description	Legal Values
colNameId	The name of the column to which the instruction will be attached.	String
schema	The name of the table schema.	String
table	The name of the table that contains the specified column.	String
predicates	The table partitions to be materialized.	Array If no predicates are required, use an empty array `[]`.
colWarmUpType	The materialization type Varada will perform on the specified column in the specified table. You can specify any combination of warm-up types, each with its own priority and TTL.	`COL_WARM_UP_TYPE_BASIC` (create an adaptive index for the column) `COL_WARM_UP_TYPE_DATA` (create a data cache for the column) `COL_WARM_UP_TYPE_LUCENE` (create a Lucene index for a string column)
priority	The instruction priority. Negative priority will prevent the data cache or index creation.	Numeric values
ttl	The TTL threshold for the instruction, after which the instruction becomes inactive and the associated column materialization is reverted if the system has reached the upper storage limit. Use `PT0M` as the TTL value so that the instruction is never deactivated.	String in ISO-8601 duration format (PnDTnHnMn) specifying a period of minutes, hours, or days.

Example

Create three instructions, two for column ss_promo_sk, and the other for column ss_customer_sk, both on the partition of date=01-01-2020.

Request

curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[
    {
       \"colNameId\": \"ss_promo_sk\",
       \"schema\": \"default\",
       \"table\": \"storesalesflat\",
       \"colWarmUpType\": \"COL_WARM_UP_TYPE_DATA\",
       \"priority\": 8,
       \"ttl\": \"PT1440M\",
       \"predicates\": [
       {
              \"type\":\"PartitionValue\",
               \"columnId\" : \"date\", 
               \"value\" : \"01-01-2021\"
       }
      ]
    },
    {
       \"colNameId\": \"ss_promo_sk\",
       \"schema\": \"default\",
       \"table\": \"storesalesflat\",
       \"colWarmUpType\": \"COL_WARM_UP_TYPE_LUCENE\",
       \"priority\": 8,
       \"ttl\": \"PT1440M\",
       \"predicates\": [
       {
         \"type\":\"PartitionValue\",
         \"columnId\" : \"date\", 
         \"value\" : \"01-01-2021\"
       }
      ]
    },
    {
       \"colNameId\": \"ss_customer_sk\",
       \"schema\": \"default\",
       \"table\": \"storesalesflat\",
       \"colWarmUpType\": \"COL_WARM_UP_TYPE_BASIC\",
       \"priority\": 8,
       \"ttl\": \"PT1440M\",
       \"predicates\": [
       {
         \"type\":\"PartitionValue\",
         \"columnId\" : \"date\", 
         \"value\" : \"01-01-2021\"
       }
      ]
  }
]"

Response

[{"id":449933183,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_DATA","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":1480046781,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_LUCENE","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":1251737023,"schema":"default","table":"storesalesflat","colNameId":"ss_customer_sk","colWarmUpType":"COL_WARM_UP_TYPE_BASIC","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]}]

Edit an Instruction

Edit one or more existing instructions. For each instruction you want to edit, you must specify the Rule ID and all the properties, including both the properties you want to update, as well as the properties that will remain unchanged.

Synopsis

curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[{
  \"id\":<instruction_id>,
  \"colNameId\": \"<column name>\",
  \"predicates\": [],
  \"schema\": \"<schema name>\",
  \"table\": \"<table name>\",  
  \"colWarmUpType\": \"<COL_WARM_UP_TYPE_BASIC|COL_WARM_UP_TYPE_DATA|COL_WARM_UP_TYPE_LUCENE>\",
  \"priority\": \"<priority value>\",
  \"ttl\": \"<PnDTnHnMn>\"
    },
    {...}
  ]
}
]
[...]"

Mandatory Properties

Property	Description	Legal Values
id	The ID of the instruction you want to edit.	Integer
colNameId	The name of the column to which the instruction will be attached.	String
schema	The name of the table schema.	String
table	The name of the table that contains the specified column.	String
predicates	The table partitions to be materialized.	Array If no predicates are required, use an empty array `[]`.
colWarmUpType	The materialization type Varada will perform on the specified column in the specified table. You can specify any combination of warm-up types, each with its own priority and TTL.	`COL_WARM_UP_TYPE_BASIC` (create an adaptive index for the column) `COL_WARM_UP_TYPE_DATA` (create a data index for the column) `COL_WARM_UP_TYPE_LUCENE` (create a Lucene index for a string column)
priority	The instruction priority.	Numeric values
ttl	The TTL threshold for the instruction, after which the instruction becomes inactive and the associated column materialization is reverted if the system has reached the upper storage limit. Use `PT0M` as the TTL value so that the instruction is never deactivated.	String in ISO-8601 duration format (PnDTnHnMn) specifying a period of minutes, hours, or days.

Example

Update instruction 449933183, and change the priority of the COL_WARM_UP_TYPE_DATA warm-up type from 8 to 1.

Request

curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[
    {
       \"id\":449933183,
       \"colNameId\": \"ss_promo_sk\",
       \"schema\": \"default\",
       \"table\": \"storesalesflat\",
       \"colWarmUpType\": \"COL_WARM_UP_TYPE_DATA\",
       \"priority\": 1,
       \"ttl\": \"PT1440M\",
       \"predicates\": [
       {
              \"type\":\"PartitionValue\",
               \"columnId\" : \"date\", 
               \"value\" : \"01-01-2021\"
       }
      ]
    }
]"

Response

[{"id":449933183,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_DATA","priority":1.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]}]

Read All Instructions

Get a list of all the instructions defined in your system.

Synopsis

curl -X GET "http://<coordinator-host>:8088/warmup/warmup-rule-get" -H "accept: application/json"

Response

A list of rules. For example:

[{"id":1480046781,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_LUCENE","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":1251737023,"schema":"default","table":"storesalesflat","colNameId":"ss_customer_sk","colWarmUpType":"COL_WARM_UP_TYPE_BASIC","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":449933183,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_DATA","priority":1.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]}]

View Warmup Status

View the warm up status.

Synopsis

pcli

select 
sum(warm_started) as warm_started,  
sum(warm_accomplished) as warm_accomplished,  
sum(warm_failed) as warm_failed,  
sum(warm_skipped_due_demoter) as warm_skipped_due_demoter  
from jmx.current."io.varada.presto:type=VaradaStatsWarmingService,name=warming-service.varada"

You can identify that Varada is not currently warming up any columns as follows:

warm_started - warm_failed - warm_skipped_due_demoter = warm_accomplished
The warm_started value does not increase when you run the command several times

View Storage Utilization

Shows the worker node storage utilization.

Synopsis

curl -X POST -H "Content-Type: application/json" http://<coordinator-host>:8088/debug-tools -d '{"commandName" : "all","@class" : "io.varada.presto.execution.debugtools.DebugToolData"}' | jq .

(Allocated 8k pages / Storage_capacity) * 100 shows the storage utilization percentage per worker node.
.

Get Warmed Up Columns

Gets a list of the currently warmed-up columns.

Synopsis

curl -X GET "http://<coordinator-host>:8088/row-group/row-group-count" -H "accept: application/json"

Response

A list of columns. Each element in the list represents a column and an acceleration type, and the corresponding count of accelerated row groups. BASIC represents index acceleration, and DATA represents data cache acceleration.

In the following example, 20 row groups of the tripid column are accelerated with an index and data cache.

v2_demo_trips.trips_data.tripid.COL_WARM_UP_TYPE_DATA": 20, v2_demo_trips.trips_data.tripdid.COL_WARM_UP_TYPE_BASIC": 20

Delete an Instruction

Deletes a specified instruction or instructions. When you delete an instruction, the column materialization (data/index) is deleted if there are not sufficient storage resources available for materializing other columns specified in the active instructions.

Synopsis

curl -X DELETE "http://<coordinator-host>:8088/warmup/warmup-rule-delete" -H "accept: application/json" -H "Content-Type: application/json" -d '[<rule_ID>]'

where rule_ID is the identification number of the instruction you want to delete.

Example

Delete instruction 384.

curl -X DELETE "http://<coordinator-host>:8088/warmup/warmup-rule-delete" -H "accept: application/json" -H "Content-Type: application/json" -d '[384]'