Skip to main content
`Ctrl`+`K`

# smallpond.init

smallpond.init(job_id: str | None = None, job_time: float | None = None, job_name: str | None = None, data_root: str | None = None, num_executors: int | None = None, ray_address: str | None = None, bind_numa_node: bool | None = None, platform: str | None = None, _remove_output_root: bool = True, **kwargs) → Session

  Initialize smallpond environment.

  This is the entry point for smallpond:

      import smallpond
      sp = smallpond.init()

  By default, it will use a local ray cluster as worker node. To use more worker nodes, please specify the argument:

      sp = smallpond.init(num_executors=10)

  It will create an task to run the workers.

  ## Parameters

  All parameters are optional. If not specified, read from environment variables. If not set, use default values.

  job_id (SP_JOBID)

    Unique job id. Default to a random uuid.

  job_time (SP_JOB_TIME)

    Job create time (seconds since epoch). Default to current time.

  job_name (SP_JOB_NAME)

    Job display name. Default to the filename of the current script.

  data_root (SP_DATA_ROOT)

    The root folder for all files generated at runtime.

  num_executors (SP_NUM_EXECUTORS)

    The number of executors. Default to 0, which means all tasks will be run on the current node.

  ray_address (SP_RAY_ADDRESS)

    If specified, use the given address to connect to an existing ray cluster. Otherwise, create a new ray cluster.

  bind_numa_node (SP_BIND_NUMA_NODE)

    If true, bind executor processes to numa nodes.

  memory_allocator (SP_MEMORY_ALLOCATOR)

    The memory allocator used by worker processes. Choices: “system”, “jemalloc”, “mimalloc”. Default to “mimalloc”.

  platform (SP_PLATFORM)

    The platform to use. Choices: “mpi”. By default, it will automatically detect the environment and choose the most suitable platform.

  _remove_output_root

    If true, remove the “{data_root}/output” directory after the job is finished. Default to True. This is only used for compatibility. User should use write_parquet for saving outputs.

  ## Spawning a new job

  If the environment variable SP_SPAWN is set to “1”, it will spawn a new job to run the current script.