-
Azure DF: storage-event Triggers
A problem: I have an ADF Pipeline. I would like that the Pipeline was automatically started when a BLOB file with certain name appears in certain folder of my Azure Storage.
A solution: create a Trigger of type Storage event:
( Read more... ) -
Dremio for Data Consumer - 4
Calculated fields
The context menu (one that we call on a column) has an item Calculated Field that allows us to construct calculated fields:
( Read more... ) -
Dremio for Data Consumers - 3
JOINs
Dremio has a functionality Join that helps to build joined SQL statements visually:
( Read more... ) -
Dremio for Data Consumers - 2
Tagging data
In addition to the wiki that we have created, we have the ability to add tags to further enhance the value of the dataset. Tags are a great feature that allows us to provide key descriptors to any dataset. Best of all, these tags are also searchable. After a dataset is tagged, when we do a search for a dataset, Dremio will also ( Read more... ) -
Dremio for Data Consumers
Describing data
One common request that you will notice when collaborating among teams, is that there should be a description of the data that we are working with. This doesn’t only help you have a broader understanding of what the data is about but also provides your team context of the dataset when they find it.
In Dremio we can describe data ( Read more... ) -
Data Reflections course - 8
10 reflection best practices
Here are some considerations and suggestions when designing your raw and aggregate reflections. By no means are they exhaustive, and each deployment will have nuances in terms of data and resources that strongly influence the appropriate best practices.
In general, Dremio recommends you experiment and measure, and ask ( Read more... ) -
Data Reflections course - 7
Reflection refresh
Refresh policies
As physical datasets change, reflections should be refreshed.
- Scheduled refresh: You select a refresh interval -- typically some number of weeks or days, but some use cases call for multiple refreshes per hour. Dremio recommends an expiration interval longer than a refresh interval in the event that a refresh ( Read more... ) -
Data Reflections course - 7
SortingBoth raw and aggregating reflections support sorting ( Read more... ) -
Data Reflections course - 6
Anchor and supporting anchor datasets
Anchor datasets are datasets used to configure a reflection. Since reflections from anchor datasets can be used to accelerate queries to any ancestor, sibling, or descendant of the anchor dataset, the anchor dataset can be where the reflection will have the most impact to the largest number of queries.
( Read more... )