Package org.apache.sysds.runtime.io

Class ReaderDeltaParallel

java.lang.Object

org.apache.sysds.runtime.io.MatrixReader

org.apache.sysds.runtime.io.ReaderDelta

org.apache.sysds.runtime.io.ReaderDeltaParallel

public class ReaderDeltaParallel
extends ReaderDelta

Parallel native Delta Lake matrix reader. Delta tables are stored as one or more parquet data files; this reader decodes those files concurrently (one task per data file) and then concatenates the per-file row-major buffers into the dense output in the original file order.

The expensive part of a Delta read is the parquet decode, which the kernel performs per data file; parallelizing across files is therefore the natural way to bridge the gap to the (near-raw) binary reader. A table backed by a single data file (the default for tables <= the parquet target file size) cannot be split this way, so the reader transparently falls back to the sequential ReaderDelta path in that case.

Constructor Summary

Constructors

Constructor	Description
ReaderDeltaParallel()

Method Summary

Modifier and Type	Method	Description
MatrixBlock	readMatrixFromHDFS(String fname, long rlen, long clen, int blen, long estnnz)

Methods inherited from class org.apache.sysds.runtime.io.ReaderDelta

readMatrixFromInputStream

Methods inherited from class org.apache.sysds.runtime.io.MatrixReader

checkValidInputFile

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ReaderDeltaParallel

public ReaderDeltaParallel()

Method Details

readMatrixFromHDFS

public MatrixBlock readMatrixFromHDFS(String fname,
 long rlen,
 long clen,
 int blen,
 long estnnz)
                               throws IOException,
DMLRuntimeException

Overrides:

readMatrixFromHDFS in class ReaderDelta

Throws:

IOException

DMLRuntimeException