Skip to main content
`Ctrl`+`K`

# smallpond.logical.node.Node

class smallpond.logical.node.Node(ctx: Context, input_deps: Tuple[ Node , ...], output_name: str | None = None, output_path: str | None = None, cpu_limit: int = 1, gpu_limit: float = 0, memory_limit: int | None = None)

  The base class for all nodes.

  __init__(ctx: Context, input_deps: Tuple[ Node , ...], output_name: str | None = None, output_path: str | None = None, cpu_limit: int = 1, gpu_limit: float = 0, memory_limit: int | None = None) → None

    The base class for all nodes in logical plan.

    ## Parameters

    ctx

      The context of logical plan.

    input_deps

      Define the inputs of this node.

    output_name, optional

      The prefix of output directories and filenames for tasks generated from this node. The default output_name is the class name of the task created for this node, e.g. HashPartitionTask, SqlEngineTask, PythonScriptTask, etc.

      The output_name string should only include alphanumeric characters and underscore. In other words, it matches regular expression [a-zA-Z0-9_]+.

      If output_name is set and output_path is None, the path format of output files is: {job_root_path}/output/{output_name}/{task_runtime_id}/{output_name}-{task_runtime_id}-{seqnum}.parquet where {task_runtime_id} is defined as {job_id}.{task_id}.{sched_epoch}.{task_retry_count}.

    output_path, optional

      The absolute path of a customized output folder for tasks generated from this node. Any shared folder that can be accessed by executor and scheduler is allowed although IO performance varies across filesystems.

      If both output_name and output_path are specified, the path format of output files is: {output_path}/{output_name}/{task_runtime_id}/{output_name}-{task_runtime_id}-{seqnum}.parquet where {task_runtime_id} is defined as {job_id}.{task_id}.{sched_epoch}.{task_retry_count}.

    cpu_limit, optional

      The max number of CPUs would be used by tasks generated from this node. This is a resource requirement specified by the user and used to guide task scheduling. smallpond does NOT enforce this limit.

    gpu_limit, optional

      The max number of GPUs would be used by tasks generated from this node. This is a resource requirement specified by the user and used to guide task scheduling. smallpond does NOT enforce this limit.

    memory_limit, optional

      The max memory would be used by tasks generated from this node. The memory limit is automatically calculated based memory-to-cpu ratio of executor machine if not specified. This is a resource requirement specified by the user and used to guide task scheduling. smallpond does NOT enforce this limit.

  Methods

  `__init__`(ctx, input_deps[, output_name, ...]) | The base class for all nodes in logical plan.
  `add_perf_metrics`(name, value) |
  `create_task`(runtime_ctx, input_deps, ...) |
  `get_perf_stats`(name) |
  `slim_copy`() |
  `task_factory`(task_builder) |

  Attributes

  `enable_resource_boost` |
  `num_partitions` |