Skip to main content
`Ctrl`+`K`

# Tasks

    # create a runtime context
    runtime_ctx = RuntimeContext(JobId.new(), data_root)
    runtime_ctx.initialize(socket.gethostname(), cleanup_root=True)

    # create a logical plan
    plan = create_logical_plan()

    # create an execution plan
    planner = Planner(runtime_ctx)
    exec_plan = planner.create_exec_plan(plan)

You can then execute the tasks in a scheduler, see Execution.

## RuntimeContext

`RuntimeContext`(job_id, job_time, data_root, *) | The configuration and state for a running job.
`JobId`([hex, bytes, bytes_le, fields, int, ...]) | A unique identifier for a job.
`TaskId` | A unique identifier for a task.
`TaskRuntimeId`(id, epoch, retry) | A unique identifier for a task at runtime.
`PartitionInfo`([index, npartitions, dimension]) | Information about a partition of a dataset.
`PerfStats`(cnt, sum, min, max, avg, p50, p75, ...) | Performance statistics for a task.

## ExecutionPlan

`ExecutionPlan`(ctx, root_task, logical_plan) | A directed acyclic graph (DAG) of tasks.

## Tasks

`Task`(ctx, input_deps, partition_infos[, ...]) | The base class for all tasks.
`ArrowBatchTask`(ctx, input_deps, partition_infos) |
`ArrowComputeTask`(ctx, input_deps, ...[, ...]) |
`ArrowStreamTask`(ctx, input_deps, partition_infos) |
`DataSinkTask`(ctx, input_deps, ...[, type, ...]) |
`DataSourceTask`(ctx, dataset, partition_infos) |
`EvenlyDistributedPartitionProducerTask`(ctx, ...) |
`HashPartitionArrowTask`(ctx, input_deps, ...) |
`HashPartitionDuckDbTask`(ctx, input_deps, ...) |
`HashPartitionTask`(ctx, input_deps, ...[, ...]) |
`LoadPartitionedDataSetProducerTask`(ctx, ...) |
`MergeDataSetsTask`(ctx, input_deps, ...[, ...]) |
`PandasBatchTask`(ctx, input_deps, partition_infos) |
`PandasComputeTask`(ctx, input_deps, ...[, ...]) |
`PartitionConsumerTask`(ctx, input_deps, ...) |
`PartitionProducerTask`(ctx, input_deps, ...) |
`ProjectionTask`(ctx, input_deps, ...) |
`PythonScriptTask`(ctx, input_deps, ...[, ...]) |
`RangePartitionTask`(ctx, input_deps, ...) |
`RepeatPartitionProducerTask`(ctx, input_deps, ...) |
`RootTask`(ctx, input_deps, partition_infos) |
`SplitDataSetTask`(ctx, input_deps, ...) |
`SqlEngineTask`(ctx, input_deps, ...[, udfs, ...]) |
`UserDefinedPartitionProducerTask`(ctx, ...[, ...]) |