Acceleration Instruction Commands
You can use the following REST commands to work with acceleration instructions.
Create an Instruction
Create one or more instructions, each of which is attached to a column in a table.
After an instruction is set, the data needs to be accessed via a query to trigger the warmup. This improves efficiency by ensuring that only data that is required is warmed up. Data that is not required will not be warmed up, even if it falls under the instruction definition.
Synopsis
curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[{
\"colNameId\": \"<column name>\",
\"predicates\": [],
\"schema\": \"<schema name>\",
\"table\": \"<table name>\",
\"colWarmUpType\": \"<COL_WARM_UP_TYPE_BASIC|COL_WARM_UP_TYPE_DATA|COL_WARM_UP_TYPE_LUCENE>\",
\"priority\": \"<priority value>\",
\"ttl\": \"<PnDTnHnMn>\"
},
{...}
]
}
]
[...]"
Mandatory Properties
Property | Description | Legal Values |
---|---|---|
colNameId | The name of the column to which the instruction will be attached. | String |
schema | The name of the table schema. | String |
table | The name of the table that contains the specified column. | String |
predicates | The table partitions to be materialized. | Array If no predicates are required, use an empty array [] . |
colWarmUpType | The materialization type Varada will perform on the specified column in the specified table. You can specify any combination of warm-up types, each with its own priority and TTL. | COL_WARM_UP_TYPE_BASIC (create an adaptive index for the column)COL_WARM_UP_TYPE_DATA (create a data cache for the column)COL_WARM_UP_TYPE_LUCENE (create a Lucene index for a string column) |
priority | The instruction priority. Negative priority will prevent the data cache or index creation. | Numeric values |
ttl | The TTL threshold for the instruction, after which the instruction becomes inactive and the associated column materialization is reverted if the system has reached the upper storage limit. Use PT0M as the TTL value so that the instruction is never deactivated. | String in ISO-8601 duration format (PnDTnHnMn) specifying a period of minutes, hours, or days. |
Example
Create three instructions, two for column ss_promo_sk, and the other for column ss_customer_sk, both on the partition of date=01-01-2020.
Request
curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[
{
\"colNameId\": \"ss_promo_sk\",
\"schema\": \"default\",
\"table\": \"storesalesflat\",
\"colWarmUpType\": \"COL_WARM_UP_TYPE_DATA\",
\"priority\": 8,
\"ttl\": \"PT1440M\",
\"predicates\": [
{
\"type\":\"PartitionValue\",
\"columnId\" : \"date\",
\"value\" : \"01-01-2021\"
}
]
},
{
\"colNameId\": \"ss_promo_sk\",
\"schema\": \"default\",
\"table\": \"storesalesflat\",
\"colWarmUpType\": \"COL_WARM_UP_TYPE_LUCENE\",
\"priority\": 8,
\"ttl\": \"PT1440M\",
\"predicates\": [
{
\"type\":\"PartitionValue\",
\"columnId\" : \"date\",
\"value\" : \"01-01-2021\"
}
]
},
{
\"colNameId\": \"ss_customer_sk\",
\"schema\": \"default\",
\"table\": \"storesalesflat\",
\"colWarmUpType\": \"COL_WARM_UP_TYPE_BASIC\",
\"priority\": 8,
\"ttl\": \"PT1440M\",
\"predicates\": [
{
\"type\":\"PartitionValue\",
\"columnId\" : \"date\",
\"value\" : \"01-01-2021\"
}
]
}
]"
Response
[{"id":449933183,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_DATA","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":1480046781,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_LUCENE","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":1251737023,"schema":"default","table":"storesalesflat","colNameId":"ss_customer_sk","colWarmUpType":"COL_WARM_UP_TYPE_BASIC","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]}]
Edit an Instruction
Edit one or more existing instructions. For each instruction you want to edit, you must specify the Rule ID and all the properties, including both the properties you want to update, as well as the properties that will remain unchanged.
Synopsis
curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[{
\"id\":<instruction_id>,
\"colNameId\": \"<column name>\",
\"predicates\": [],
\"schema\": \"<schema name>\",
\"table\": \"<table name>\",
\"colWarmUpType\": \"<COL_WARM_UP_TYPE_BASIC|COL_WARM_UP_TYPE_DATA|COL_WARM_UP_TYPE_LUCENE>\",
\"priority\": \"<priority value>\",
\"ttl\": \"<PnDTnHnMn>\"
},
{...}
]
}
]
[...]"
Mandatory Properties
Property | Description | Legal Values |
---|---|---|
id | The ID of the instruction you want to edit. | Integer |
colNameId | The name of the column to which the instruction will be attached. | String |
schema | The name of the table schema. | String |
table | The name of the table that contains the specified column. | String |
predicates | The table partitions to be materialized. | Array If no predicates are required, use an empty array [] . |
colWarmUpType | The materialization type Varada will perform on the specified column in the specified table. You can specify any combination of warm-up types, each with its own priority and TTL. | COL_WARM_UP_TYPE_BASIC (create an adaptive index for the column)COL_WARM_UP_TYPE_DATA (create a data index for the column)COL_WARM_UP_TYPE_LUCENE (create a Lucene index for a string column) |
priority | The instruction priority. | Numeric values |
ttl | The TTL threshold for the instruction, after which the instruction becomes inactive and the associated column materialization is reverted if the system has reached the upper storage limit. Use PT0M as the TTL value so that the instruction is never deactivated. | String in ISO-8601 duration format (PnDTnHnMn) specifying a period of minutes, hours, or days. |
Example
Update instruction 449933183
, and change the priority of the COL_WARM_UP_TYPE_DATA
warm-up type from 8
to 1
.
Request
curl -X POST "http://<coordinator-host>:8088/warmup/warmup-rule-set" -H "Content-Type: application/json" -d "[
{
\"id\":449933183,
\"colNameId\": \"ss_promo_sk\",
\"schema\": \"default\",
\"table\": \"storesalesflat\",
\"colWarmUpType\": \"COL_WARM_UP_TYPE_DATA\",
\"priority\": 1,
\"ttl\": \"PT1440M\",
\"predicates\": [
{
\"type\":\"PartitionValue\",
\"columnId\" : \"date\",
\"value\" : \"01-01-2021\"
}
]
}
]"
Response
[{"id":449933183,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_DATA","priority":1.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]}]
Read All Instructions
Get a list of all the instructions defined in your system.
Synopsis
curl -X GET "http://<coordinator-host>:8088/warmup/warmup-rule-get" -H "accept: application/json"
Response
A list of rules. For example:
[{"id":1480046781,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_LUCENE","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":1251737023,"schema":"default","table":"storesalesflat","colNameId":"ss_customer_sk","colWarmUpType":"COL_WARM_UP_TYPE_BASIC","priority":8.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]},{"id":449933183,"schema":"default","table":"storesalesflat","colNameId":"ss_promo_sk","colWarmUpType":"COL_WARM_UP_TYPE_DATA","priority":1.0,"ttl":86400.000000000,"predicates":[{"type":"PartitionValue","columnId":"date","value":"01-01-2021"}]}]
View Warmup Status
View the warm up status.
Synopsis
pcli
select
sum(warm_started) as warm_started,
sum(warm_accomplished) as warm_accomplished,
sum(warm_failed) as warm_failed,
sum(warm_skipped_due_demoter) as warm_skipped_due_demoter
from jmx.current."io.varada.presto:type=VaradaStatsWarmingService,name=warming-service.varada"
You can identify that Varada is not currently warming up any columns as follows:
warm_started
-warm_failed
-warm_skipped_due_demoter
=warm_accomplished
- The
warm_started
value does not increase when you run the command several times
View Storage Utilization
Shows the worker node storage utilization.
Synopsis
curl -X POST -H "Content-Type: application/json" http://<coordinator-host>:8088/debug-tools -d '{"commandName" : "all","@class" : "io.varada.presto.execution.debugtools.DebugToolData"}' | jq .
(Allocated 8k pages / Storage_capacity) * 100
shows the storage utilization percentage per worker node.
.
Get Warmed Up Columns
Gets a list of the currently warmed-up columns.
Synopsis
curl -X GET "http://<coordinator-host>:8088/row-group/row-group-count" -H "accept: application/json"
Response
A list of columns. Each element in the list represents a column and an acceleration type, and the corresponding count of accelerated row groups. BASIC
represents index acceleration, and DATA
represents data cache acceleration.
In the following example, 20 row groups of the tripid
column are accelerated with an index and data cache.
v2_demo_trips.trips_data.tripid.COL_WARM_UP_TYPE_DATA": 20, v2_demo_trips.trips_data.tripdid.COL_WARM_UP_TYPE_BASIC": 20
Delete an Instruction
Deletes a specified instruction or instructions. When you delete an instruction, the column materialization (data/index) is deleted if there are not sufficient storage resources available for materializing other columns specified in the active instructions.
Synopsis
curl -X DELETE "http://<coordinator-host>:8088/warmup/warmup-rule-delete" -H "accept: application/json" -H "Content-Type: application/json" -d '[<rule_ID>]'
where rule_ID is the identification number of the instruction you want to delete.
Example
Delete instruction 384
.
curl -X DELETE "http://<coordinator-host>:8088/warmup/warmup-rule-delete" -H "accept: application/json" -H "Content-Type: application/json" -d '[384]'
Updated 8 months ago