Skip to main content
`Ctrl`+`K`

# smallpond.logical.node.DataSetPartitionNode

smallpond.logical.node.DataSetPartitionNode(ctx: Context, input_deps: Tuple[ Node , ...], npartitions: int, *, partition_by_rows=False, random_shuffle=False, data_partition_column=None)

  Partition the outputs of input_deps into n partitions.

  ## Parameters

  npartitions

    The number of partitions. The input files or rows would be evenly distributed to npartitions partitions.

  partition_by_rows, optional

    Evenly distribute rows instead of input files into npartitions partitions, by default distribute by files.

  random_shuffle, optional

    Random shuffle the list of paths or parquet row groups (if partition_by_rows=True) in input datasets.

  data_partition_column, optional

    Partition by files based on the partition keys stored in data_partition_column if specified. Default column name used by HashPartitionNode is DATA_PARTITION_COLUMN_NAME.

  ## Examples

  See unit test test_load_partitioned_datasets in test/test_partition.py.