Package org.apache.sysds.runtime.matrix.data

Class LibMatrixReorg

java.lang.Object
org.apache.sysds.runtime.matrix.data.LibMatrixReorg
public class LibMatrixReorg extends Object
MB: Library for selected matrix reorg operations including special cases and all combinations of dense and sparse representations. Current list of supported operations: - reshape, - r' (transpose), - rdiag (diagV2M/diagM2V), - rsort (sorting data/indexes) - rmempty (remove empty) - rexpand (outer/table-seq expansion)

  • Field Details

    • PAR_NUMCELL_THRESHOLD

      public static long PAR_NUMCELL_THRESHOLD

    • PAR_NUMCELL_THRESHOLD_SORT

      public static final int PAR_NUMCELL_THRESHOLD_SORT
      See Also:

    • SHALLOW_COPY_REORG

      public static final boolean SHALLOW_COPY_REORG
      See Also:

    • SPARSE_OUTPUTS_IN_CSR

      public static final boolean SPARSE_OUTPUTS_IN_CSR
      See Also:

    • TRANSPOSE_IN_PLACE_DENSE_LEGACY

      public static final boolean TRANSPOSE_IN_PLACE_DENSE_LEGACY
      See Also:

  • Method Details

    • isSupportedReorgOperator

      public static boolean isSupportedReorgOperator(ReorgOperator op)

    • reorg

      public static MatrixBlock reorg(MatrixBlock in, MatrixBlock out, ReorgOperator op)

    • reorgInPlace

      public static MatrixBlock reorgInPlace(MatrixBlock in, ReorgOperator op)

    • transpose

      public static MatrixBlock transpose(MatrixBlock in)

    • transpose

      public static MatrixBlock transpose(MatrixBlock in, MatrixBlock out)

    • transpose

      public static MatrixBlock transpose(MatrixBlock in, int k)

    • transpose

      public static MatrixBlock transpose(MatrixBlock in, int k, boolean allowCSR)

    • transpose

      public static MatrixBlock transpose(MatrixBlock in, MatrixBlock out, int k)

    • transpose

      public static MatrixBlock transpose(MatrixBlock in, MatrixBlock out, int k, boolean allowCSR)

    • countNNZColumns

      public static Future<int[]> countNNZColumns(MatrixBlock in, int k, ExecutorService pool) throws InterruptedException, ExecutionException
      Throws:
      InterruptedException
      ExecutionException

    • countNNZColumnsFuture

      public static List<Future<int[]>> countNNZColumnsFuture(MatrixBlock in, int k, ExecutorService pool) throws InterruptedException
      Throws:
      InterruptedException

    • transposeInPlace

      public static MatrixBlock transposeInPlace(MatrixBlock in, int k)

    • rev

      public static MatrixBlock rev(MatrixBlock in, MatrixBlock out)

    • rev

      public static MatrixBlock rev(MatrixBlock in, MatrixBlock out, int k)

    • rev

      public static void rev(IndexedMatrixValue in, long rlen, int blen, ArrayList<IndexedMatrixValue> out)

    • roll

      public static MatrixBlock roll(MatrixBlock in, MatrixBlock out, int shift)

    • roll

      public static MatrixBlock roll(MatrixBlock input, MatrixBlock output, int shift, int numThreads)

    • roll

      public static void roll(IndexedMatrixValue in, long rlen, int blen, int shift, ArrayList<IndexedMatrixValue> out)

    • diag

      public static MatrixBlock diag(MatrixBlock in, MatrixBlock out)

    • sort

      public static MatrixBlock sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret)

    • sort

      public static MatrixBlock sort(MatrixBlock in, MatrixBlock out, int[] by, boolean desc, boolean ixret, int k)
      Parameters:
      in - Input matrix to sort
      out - Output matrix where the sorted input is inserted to
      by - The Ordering parameter
      desc - A boolean, specifying if it should be descending order.
      ixret - A boolean, specifying if the return should be the sorted indexes.
      k - Number of parallel threads
      Returns:
      The sorted out matrix.

    • reshape

      public static MatrixBlock reshape(MatrixBlock in, int rows, int cols, boolean rowwise)
      CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.
      Parameters:
      in - input matrix
      rows - number of rows
      cols - number of columns
      rowwise - if true, reshape by row
      Returns:
      output matrix

    • reshape

      public static MatrixBlock reshape(MatrixBlock in, MatrixBlock out, int rows, int cols, boolean rowwise)
      CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.
      Parameters:
      in - input matrix
      out - output matrix
      rows - number of rows
      cols - number of columns
      rowwise - if true, reshape by row
      Returns:
      output matrix

    • reshape

      public static MatrixBlock reshape(MatrixBlock in, MatrixBlock out, int rows, int cols, boolean rowwise, int k)
      CP reshape operation (single input, single output matrix) NOTE: In contrast to R, the rowwise parameter specifies both the read and write order, with row-wise being the default, while R uses always a column-wise read, rowwise specifying the write order and column-wise being the default.
      Parameters:
      in - input matrix
      out - output matrix
      rows - number of rows
      cols - number of columns
      rowwise - if true, reshape by row
      k - The parallelization degree
      Returns:
      output matrix

    • reshape

      public static List<IndexedMatrixValue> reshape(IndexedMatrixValue in, DataCharacteristics mcIn, DataCharacteristics mcOut, boolean rowwise, boolean outputEmptyBlocks)
      MR/SPARK reshape interface - for reshape we cannot view blocks independently, and hence, there are different CP and MR interfaces.
      Parameters:
      in - indexed matrix value
      mcIn - input matrix characteristics
      mcOut - output matrix characteristics
      rowwise - if true, reshape by row
      outputEmptyBlocks - output blocks with nnz=0
      Returns:
      list of indexed matrix values

    • rmempty

      public static MatrixBlock rmempty(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
      CP rmempty operation (single input, single output matrix)
      Parameters:
      in - The input matrix
      ret - The output matrix
      rows - If we are removing based on rows, or columns.
      emptyReturn - Return row/column of zeros for empty input
      select - An optional selection vector, to remove based on rather than empty rows or columns
      Returns:
      The result MatrixBlock

    • rmemptyUnsafe

      public static MatrixBlock rmemptyUnsafe(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)

    • rmemptyEarlyAbort

      public static MatrixBlock rmemptyEarlyAbort(MatrixBlock in, MatrixBlock ret, boolean rows, boolean emptyReturn, MatrixBlock select)
      Handle the early-termination cases of removeEmpty that do not require scanning for empty rows/columns.
      Parameters:
      in - The input matrix
      ret - The output matrix, reused for the empty-input case
      rows - If removing based on rows, or columns
      emptyReturn - Return a row/column of zeros for empty input
      select - An optional selection vector
      Returns:
      The early-abort result, or null if no early termination applies and the caller must continue. For the select-all case the returned block is the input in itself (a shallow alias, not a copy).

    • rmempty

      public static void rmempty(IndexedMatrixValue data, IndexedMatrixValue offset, boolean rmRows, long len, long blen, ArrayList<IndexedMatrixValue> outList)
      MR rmempty interface - for rmempty we cannot view blocks independently, and hence, there are different CP and MR interfaces.
      Parameters:
      data - ?
      offset - ?
      rmRows - ?
      len - ?
      blen - block length
      outList - list of indexed matrix values

    • rexpand

      public static MatrixBlock rexpand(MatrixBlock in, MatrixBlock ret, double max, boolean rows, boolean cast, boolean ignore, int k)
      CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.
      Parameters:
      in - Input matrix
      ret - Output matrix
      max - Number of rows/cols of the output
      rows - If the expansion is in rows direction
      cast - If the values contained should be cast to double (rounded up and down)
      ignore - Ignore if the input contain values below zero that technically is incorrect input.
      k - Degree of parallelism
      Returns:
      Output matrix rexpanded

    • rexpand

      public static MatrixBlock rexpand(MatrixBlock in, MatrixBlock ret, int max, boolean rows, boolean cast, boolean ignore, int k)
      CP rexpand operation (single input, single output), the classic example of this operation is one hot encoding of a column to multiple columns.
      Parameters:
      in - Input matrix
      ret - Output matrix
      max - Number of rows/cols of the output
      rows - If the expansion is in rows direction
      cast - If the values contained should be cast to double (rounded up and down)
      ignore - Ignore if the input contain values below zero that technically is incorrect input.
      k - Degree of parallelism
      Returns:
      Output matrix rexpanded

    • fusedSeqRexpand

      public static MatrixBlock fusedSeqRexpand(int seqHeight, MatrixBlock A, double w)
      The DML code to activate this function:

      ret = table(seq(1, nrow(A)), A, w)

      Parameters:
      seqHeight - A sequence vector height.
      A - The MatrixBlock vector to encode.
      w - The weight matrix to multiply on output cells.
      Returns:
      A new MatrixBlock with the table result.

    • fusedSeqRexpand

      public static MatrixBlock fusedSeqRexpand(int seqHeight, MatrixBlock A, double w, MatrixBlock ret, boolean updateClen)
      The DML code to activate this function:

      ret = table(seq(1, nrow(A)), A, w)

      Parameters:
      seqHeight - A sequence vector height.
      A - The MatrixBlock vector to encode.
      w - The weight scalar to multiply on output cells.
      ret - The output MatrixBlock, does not have to be used, but depending on updateClen determine the output size.
      updateClen - Update clen, if set to true, ignore dimensions of ret, otherwise use the column dimension of ret.
      Returns:
      A new MatrixBlock or ret.

    • fusedSeqRexpand

      public static MatrixBlock fusedSeqRexpand(int seqHeight, MatrixBlock A, double w, MatrixBlock ret, boolean updateClen, int k)
      The DML code to activate this function:

      ret = table(seq(1, nrow(A)), A, w)

      Parameters:
      seqHeight - A sequence vector height.
      A - The MatrixBlock vector to encode.
      w - The weight matrix to multiply on output cells.
      ret - The output MatrixBlock, does not have to be used, but depending on updateClen determine the output size.
      updateClen - Update clen, if set to true, ignore dimensions of ret, otherwise use the column dimension of ret.
      k - Parallelization degree
      Returns:
      A new MatrixBlock or ret.

    • rexpandSingleRow

      public static int rexpandSingleRow(int row, double v2, double w, int[] retIx, double[] retVals, boolean updateClen, int maxOutCol)

    • checkRexpand

      public static void checkRexpand(MatrixBlock in, boolean ignore)
      Quick check if the input is valid for rexpand, this check does not guarantee that the input is valid for rexpand
      Parameters:
      in - Input matrix block
      ignore - If zero valued cells should be ignored

    • rexpand

      public static void rexpand(IndexedMatrixValue data, double max, boolean rows, boolean cast, boolean ignore, long blen, ArrayList<IndexedMatrixValue> outList)
      MR/Spark rexpand operation (single input, multiple outputs incl empty blocks)
      Parameters:
      data - Input indexed matrix block
      max - Total nrows/cols of the output
      rows - If the expansion is in rows direction
      cast - If the values contained should be cast to double (rounded up and down)
      ignore - Ignore if the input contain values below zero that technically is incorrect input.
      blen - The block size to slice the output up into
      outList - The output indexedMatrixValues (a list to add all the output blocks to / modify)

    • countNnzPerColumn

      public static int[] countNnzPerColumn(MatrixBlock in)

    • countNnzPerColumn

      public static int[] countNnzPerColumn(MatrixBlock in, int rl, int ru)

    • mergeNnzCounts

      public static int[] mergeNnzCounts(int[] cnt, int[] cnt2)

    • copyMtx

      public static void copyMtx(MatrixBlock in, MatrixBlock out, int inStart, int outStart, int copyLen, boolean isAllocated, boolean copyTotalNonZeros)

    • copyDenseMtx

      public static void copyDenseMtx(MatrixBlock in, MatrixBlock out, int inIdx, int outIdx, int copyLen, boolean isAllocated, boolean copyTotalNonZeros)

    • removeEmptyRows

      public static MatrixBlock removeEmptyRows(MatrixBlock in, MatrixBlock ret, boolean emptyReturn, boolean selectNull, boolean[] flags, int rlen2)
      Remove selected rows, based on the boolean array given. Note this function is internal use only, and require a boolean vector to be constructed first.
      Parameters:
      in - Input to remove rows from
      ret - Output to assign the result into
      emptyReturn - If the output is allowed to be empty.
      selectNull - If the original caller did not have a selection matrix.
      flags - The boolean selection vector to specify which rows to keep.
      rlen2 - The number of true values in the flags argument.
      Returns:
      Another reference to the ret matrix input argument.

    • transposeInPlaceDenseBrenner

      public static void transposeInPlaceDenseBrenner(MatrixBlock in, int k)
      Transposes a dense matrix in-place using following cycles based on Brenner's method. This method shifts cycles with a focus on less storage by using cycle leaders based on prime factorization. The used storage is in O(n+m). Quadratic matrices should be handled outside this method (using the trivial method) for a speedup. This method is based on: Algorithm 467, Brenner, https://dl.acm.org/doi/pdf/10.1145/355611.362542.
      Parameters:
      in - The input matrix to be transposed.
      k - The number of threads.

    • transposeInPlaceTensor

      public static boolean transposeInPlaceTensor(MatrixBlock in, int[] shape, int[] perm)
      Performs prime in-place tensor transposition for arbitrary permutations.
      Parameters:
      in - Tensor stored as MatrixBlock
      shape - Original shape informtion of tensor
      perm - Permutation of tensor

    • findSplitIndex

      public static int findSplitIndex(int[] perm)
      Finds a split index for a tensor permutation that allows reduction of the permutation to a 2D matrix transpose.
      Parameters:
      perm - permutation of tensor axes
      Returns:
      split index {i} if reducible, otherwise {-1}

    • permute

      public static MatrixBlock permute(MatrixBlock in, int[] inDims, int[] perm)

    • permute

      public static MatrixBlock permute(MatrixBlock in, int[] inDims, int[] perm, int k)