Package org.apache.sysds.runtime.transform.encode

Class ColumnEncoder

java.lang.Object
org.apache.sysds.runtime.transform.encode.ColumnEncoder
All Implemented Interfaces:
Externalizable, Serializable, Comparable<ColumnEncoder>, Encoder
Direct Known Subclasses:
ColumnEncoderBagOfWords, ColumnEncoderBin, ColumnEncoderComposite, ColumnEncoderDummycode, ColumnEncoderFeatureHash, ColumnEncoderPassThrough, ColumnEncoderRecode, ColumnEncoderUDF, ColumnEncoderWordEmbedding
public abstract class ColumnEncoder extends Object implements Encoder, Comparable<ColumnEncoder>
Base class for all transform encoders providing both a row and block interface for decoding frames to matrices.
See Also:

  • Field Details

    • APPLY_ROW_BLOCKS_PER_COLUMN

      public static int APPLY_ROW_BLOCKS_PER_COLUMN

    • BUILD_ROW_BLOCKS_PER_COLUMN

      public static int BUILD_ROW_BLOCKS_PER_COLUMN

  • Method Details

    • initEmbeddings

      public void initEmbeddings(MatrixBlock embeddings)

    • apply

      public MatrixBlock apply(CacheBlock<?> in, MatrixBlock out, int outputCol)
      Apply Functions are only used in Single Threaded or Multi-Threaded Dense context. That's why there is no regard for MT sparse!
      Specified by:
      apply in interface Encoder
      Parameters:
      in - Input Block
      out - Output Matrix
      outputCol - The output column for the given column
      Returns:
      same as out

    • apply

      public MatrixBlock apply(CacheBlock<?> in, MatrixBlock out, int outputCol, int rowStart, int blk)

    • isApplicable

      public boolean isApplicable()
      Indicates if this encoder is applicable, i.e, if there is a column to encode.
      Returns:
      true if a colID is set

    • isApplicable

      public boolean isApplicable(int colID)
      Indicates if this encoder is applicable for the given column ID, i.e., if it is subject to this transformation.
      Parameters:
      colID - column ID
      Returns:
      true if encoder is applicable for given column

    • prepareBuildPartial

      public void prepareBuildPartial()
      Allocates internal data structures for partial build.
      Specified by:
      prepareBuildPartial in interface Encoder

    • getDomainSize

      public int getDomainSize()

    • buildPartial

      public void buildPartial(FrameBlock in)
      Partial build of internal data structures (e.g., in distributed spark operations).
      Specified by:
      buildPartial in interface Encoder
      Parameters:
      in - input frame block

    • build

      public void build(CacheBlock<?> in, double[] equiHeightMaxs)

    • build

      public void build(CacheBlock<?> in, Map<Integer,double[]> equiHeightMaxs)

    • mergeAt

      public void mergeAt(ColumnEncoder other)
      Merges another encoder, of a compatible type, in after a certain position. Resizes as necessary. ColumnEncoders are compatible with themselves and EncoderComposite is compatible with every other ColumnEncoders. MultiColumnEncoders are compatible with every encoder
      Parameters:
      other - the encoder that should be merged in

    • updateIndexRanges

      public void updateIndexRanges(long[] beginDims, long[] endDims, int colOffset)
      Update index-ranges to after encoding. Note that only Dummycoding changes the ranges.
      Specified by:
      updateIndexRanges in interface Encoder
      Parameters:
      beginDims - begin dimensions of range
      endDims - end dimensions of range
      colOffset - is applied to begin and endDims

    • getColMapping

      public MatrixBlock getColMapping(FrameBlock meta)
      Obtain the column mapping of encoded frames based on the passed meta data frame.
      Parameters:
      meta - meta data frame block
      Returns:
      matrix with column mapping (one row per attribute)

    • writeExternal

      public void writeExternal(ObjectOutput os) throws IOException
      Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd serialization.
      Specified by:
      writeExternal in interface Externalizable
      Parameters:
      os - object output
      Throws:
      IOException - if IOException occurs

    • readExternal

      public void readExternal(ObjectInput in) throws IOException
      Redirects the default java serialization via externalizable to our default hadoop writable serialization for efficient broadcast/rdd deserialization.
      Specified by:
      readExternal in interface Externalizable
      Parameters:
      in - object input
      Throws:
      IOException - if IOException occur

    • getColID

      public int getColID()

    • setColID

      public void setColID(int colID)

    • shiftCol

      public void shiftCol(int columnOffset)

    • setEstMetaSize

      public void setEstMetaSize(long estSize)

    • getEstMetaSize

      public long getEstMetaSize()

    • setEstNumDistincts

      public void setEstNumDistincts(int numDistincts)

    • getEstNumDistincts

      public int getEstNumDistincts()

    • computeMapSizeEstimate

      public void computeMapSizeEstimate(CacheBlock<?> in, int[] sampleIndices)

    • compareTo

      public int compareTo(ColumnEncoder o)
      Specified by:
      compareTo in interface Comparable<ColumnEncoder>

    • getBuildTasks

      public List<DependencyTask<?>> getBuildTasks(CacheBlock<?> in)

    • getBuildTask

      public Callable<Object> getBuildTask(CacheBlock<?> in)

    • getPartialBuildTask

      public Callable<Object> getPartialBuildTask(CacheBlock<?> in, int startRow, int blockSize, HashMap<Integer,Object> ret, int p)

    • getPartialMergeBuildTask

      public Callable<Object> getPartialMergeBuildTask(HashMap<Integer,?> ret)

    • getApplyTasks

      public List<DependencyTask<?>> getApplyTasks(CacheBlock<?> in, MatrixBlock out, int outputCol, int[] sparseRowPointerOffsets)