Skip to main content
`Ctrl`+`K`

# smallpond.logical.dataset.CsvDataSet

class smallpond.logical.dataset.CsvDataSet(paths: List[str], schema: Dict[str, str], delim=',', max_line_size: int | None = None, parallel=True, header=False, root_dir: str | None = '', recursive=False, columns: List[str] | None = None, union_by_name=False)

  A set of csv files.

  __init__(paths: List[str], schema: Dict[str, str], delim=',', max_line_size: int | None = None, parallel=True, header=False, root_dir: str | None = '', recursive=False, columns: List[str] | None = None, union_by_name=False) → None

    Construct a dataset from a list of paths.

    ## Parameters

    paths

      A path or a list of paths or path patterns. e.g. [‘data/100.parquet’, ‘/datasetA/*.parquet’].

    root_dir, optional

      Relative paths in paths would be resolved under root_dir if specified.

    recursive, optional

      Resolve path patterns recursively if true.

    columns, optional

      Only load the specified columns if not None.

    union_by_name, optional

      Unify the columns of different files by name (see https://duckdb.org/docs/data/multiple_files/combining_schemas#union-by-name).

  Methods

  `__init__`(paths, schema[, delim, ...]) | Construct a dataset from a list of paths.
  `log`([num_rows]) | Log the dataset to the logger.
  `merge`(datasets) | Merge multiple datasets into a single dataset.
  `partition_by_files`(npartition[, random_shuffle]) | Evenly split into npartition datasets by files.
  `reset`([paths, root_dir, recursive]) | Reset the dataset with new paths, root_dir, and recursive flag.
  `sql_query_fragment`([filesystem, conn]) | Return a sql fragment that represents the dataset.
  `to_arrow_table`([max_workers, filesystem, conn]) | Load the dataset to an arrow table.
  `to_batch_reader`([batch_size, filesystem, conn]) | Return an arrow record batch reader to read the dataset.
  `to_pandas`() | Convert the dataset to a pandas dataframe.

  Attributes

  `schema` |
  `delim` |
  `max_line_size` |
  `parallel` |
  `header` |
  `absolute_paths` | An ordered list of absolute paths of the given file patterns.
  `columns` | The columns to load from the dataset files.
  `empty` | Whether the dataset is empty.
  `num_files` | The number of files in the dataset.
  `num_rows` | The number of rows in the dataset.
  `paths` | The paths to the dataset files.
  `recursive` | Whether to resolve path patterns recursively.
  `resolved_paths` | An ordered list of absolute paths of files.
  `root_dir` | The root directory of paths.
  `udfs` |
  `union_by_name` | Whether to unify the columns of different files by name.