Caches¶
Caches in general refer to the CacheMachines that connect kernels and the CacheData inside them.
CacheData¶
CacheData objects hold data and/or metadata. Most of the time a CacheData only holds data, and is used for the purpose of moving data through the execution graph. Sometimes that data needs to have some associated metadata that is used by the business logic of the algorithms. And finally, there are cases where metadata needs to be passed from one node to another or one kernel to another and in those cases you can have a CacheData object that has metadata but no actual data. The purpose of this is to only have one kind of message for sending both data and metadata.
The data held by a CacheData can be in one of several different states depending on the underlying CacheData type it is. The different kinds of CacheData at the moment are: - GPUCacheData - CPUCacheData - CacheDataLocalFile - CacheDataIO - ConcatCacheData
The purpose of the CacheData object is to hold data that is not necessarily materialized and can be passed around
without necessarily materializing it until it has to be. Most kernels will obtain data in the form of a CacheData from a CacheMachine
and use it to create a task to be executed by the task executor. That data in the CacheData will only then be materialized when the
decache() method is called, right before the data is processed by the executor, using the do_process() function of the kernel.
The materialization process performed by the decache() method will depend on the type of CacheData it is.
GPUCacheData¶
Data Representation: A GPUCacheData holds data that is already effectively materialized, because data representation in a GPUCacheData is already a BlazingTable.
Typical Usage: A GPUCacheData is typically used when there is space in the GPU to hold it. Its the most optimal data representation since its the closest to being materialized.
Decache process: The decache() function call of GPUCacheData simply moves the std::unique_ptr<BlazingTable> holding the data out. See BlazingTable in the Data Structures Page
CPUCacheData¶
Data Representation: A CPUCacheData holds data in system memory. The data representation in a CPUCacheData is a BlazingHostTable, which is effectively the exact same memory layout. of a GPUCacheData but in system memory. Note that a BlazingHostTable can use pinned memory from a memory pool, or regular host memory from a memory pool. See BlazingHostTable in the Data Structures Page
Typical Usage: A CPUCacheData is typically used when there is not enough space in the GPU but there is enough space in the CPU. Its the second most optimal data representation. It is also used for holding data that has just been received from another node.
Decache process: The decache() function call of CPUCacheData will materialize the data by copying it to GPU making a BlazingTable.
CacheDataLocalFile¶
Data Representation: A CacheDataLocalFile holds data in the local filesystem. The data representation in a CacheDataLocalFile is an orc file. The location where CacheDataLocalFile orc files are stored is defined by the path set in the BLAZING_CACHE_DIRECTORY config option.
Typical Usage: A CacheDataLocalFile is typically used when there is not enough space in the GPU nor in system memory.
Decache process: The decache() function call of CacheDataLocalFile will materialize the data by reading the orc file into a BlazingTable.
NOTE: In the future a CacheDataLocalFile could be implemented by a CacheDataIO so that its not limited to being a local orc file, but instead
the file format and filesystem is more generic.
CacheDataIO¶
Data Representation: A CacheDataIO holds data that is going to be obtained from a data source. Its representation is in the form of a combination of a data_handle, a data_parser and a Schema which can be found in IO Page.
Typical Usage: A CacheDataIO is what is generated by a table scan. It represents data that will be generated by reading a file or dataframe that comes from a table.
Decache process: The decache() function call of CacheDataIO will read and parse a file in the case of file based data sources, or create a BlazingTable from a
DataFrame provided by the user when creating a table.
ConcatCacheData¶
Data Representation: A ConcatCacheData holds a vector of CacheData that will be concatenated togather. The different CacheData can be of any type.
Typical Usage: A ConcatCacheData is sometimes used for concatenating several CacheData for data coming from several nodes, or in some algorithms several CacheData can be combined without materializing them, producing a ConcatCacheData.
Decache process: The decache() function call of ConcatCacheData will call decache() for all the individual CacheData and then concatenate them, before returning a BlazingTable.
WaitingQueue¶
The WaitingQueue is a structure used by the CacheMachine that serves as a FIFO queue that is threadsafe and has waiting mechanisms so that you try to retireve an item from the queue, it will wait until there is an item available. Additionally it can have a state where it can be “finished”, in which case, any function calls waiting for an item to be returned, just return a nullptr.
The WaitingQueue class is templated, but when used in the CacheMachine, the template object is a message which is simply a unique_ptr<CacheData> and a string which served as a message_id.
The message_id is used in several places in the code to identify or label a CacheData.
Additionally, the WaitingQueue has facilities for knowing how many CacheData objects have been added or how many bytes have been added.
CacheMachine¶
CacheMachines are an abstraction built on top of WaitingQueues that provide facilities to add or remove BlazingTables or other data structures from a CacheMachine. The CacheMachine will convert it to and from a CacheData before adding it or after removing it from a WaitingQueue.
A CacheMachine has an understanding of different memory resources and therefore can automatically decide if a BlazingTable should be converted to a GPUCacheData, CPUCacheData or CacheDataLocalFile before added into the WaitingQueue that will hold the CacheData. When you add data into a CacheMachine, it checks the memory consumption of the node by asking the memory resources. If the consumption is below a certain threshold, then the data is maintained in GPU memory. It is converted into a GPUCacheData and added to the CacheMachine. If consumption is above the device memory threshold, then it checks the next tier in the CacheMachine, the CPU cache. It checks the memory consumption of the CPU memory resource. If it is below that threshold, it converts the BlazingTable into a CPUCacheData, where it copied all the data to host. If the CPU memory consumption is above a certain threshold, then it goes into the next tier, the Disk Cache. For the disk cache, the data is placed in an ORC file and a CacheDataLocalFile is created to keep track of it.
Aside from the standard CacheMachine, another specialty type: ConcatenatingCacheMachine. The ConcatenatingCacheMachine will concatenate batches so that the resulting batch is not too small. It is used in several places and for some places, it is configurable. It is useful to concatenate batches to increase performance, since operating on really small batches can be detrimental to performance.