Working with Materializations
Continuously materializing query results into an external data store
The results of a Fenl query can be written to an external data store and kept up to date as the data underlying the query changes using materializations. A materialization is similar to a query, except that the results are updated any time data is added to a table used by the query. Materializations can be used to populate feature vectors in a variety of feature stores to be used in production for low-latency inference.
All methods on this page use from the materialization
object. Be sure to import it before running any method:
from kaskada import materialization
Materialization Methods
Creating a Materialization
To create a materialization, we'll start by describing the expression we'd like to materialize. In this case, we're interested in some purchase statistics for each user. This definition depends on business logic and might require some iteration to get just right.
Using fenlmagic
%%fenl --var purchase_stats
{
time: Purchase.purchase_time,
entity: Purchase.customer_id,
max_amount: Purchase.amount | max(),
min_amount: Purchase.amount | min(),
}
Notice that we added --var purchase_stats
to the beginning of the magic block; this causes the extension to assign the query string to the variable purchase_stats
when the block is run.
We can use this variable to create a materialization using the Python client without re-typing the expression:
redis_db = 1
redis_host = "redisai"
redis_port = 6379
destination = materialization.RedisAIDestination(redis_db, redis_host, redis_port)
materialization.create_materialization(
name = "PurchaseStats",
destination = destination,
query = purchase_stats,
)
Using python
Materializations can also be created without using fenlmagic:
materialization.create_materialization(
name = "PurchaseStats",
destination = destination,
query = "{
time: Purchase.purchase_time,
entity: Purchase.customer_id,
max_amount: Purchase.amount | max(),
min_amount: Purchase.amount | min(),
}"
)
List Materializations
The list materializations method returns all materializations defined for your user. An optional search string can filter the response set.
Here is an example of listing tables:
materialization.list_materializations("Purchase")
Get Materialization
You can get a materialization using its name:
materialization.get_materialization("PurchaseStats")
Updating a Materialization
Materializations are currently immutable. Updating a materialization requires deleting that materialization and then re-creating it with a new expression.
Deleting a materialization
You can delete a materialization using its name:
materialization.delete_materialization("PurchaseStats")
Deleting a materialization does not delete any data persisted in the external data store.
Updated over 1 year ago