Skip to main content
`Ctrl`+`K`

# DataFrame

DataFrame is the main class in smallpond. It represents a lazily computed, partitioned data set.

A typical workflow looks like this:

    import smallpond

    sp = smallpond.init()

    df = sp.read_parquet("path/to/dataset/*.parquet")
    df = df.repartition(10)
    df = df.map("x + 1")
    df.write_parquet("path/to/output")

## Initialization

`smallpond.init`([job_id, job_time, job_name, ...]) | Initialize smallpond environment.

## Loading Data

`Session.from_items`(items) | Create a DataFrame from a list of local Python objects.
`Session.from_arrow`(table) | Create a DataFrame from a pyarrow Table.
`Session.from_pandas`(df) | Create a DataFrame from a pandas DataFrame.
`Session.read_csv`(paths, schema[, delim]) | Create a DataFrame from CSV files.
`Session.read_json`(paths, schema) | Create a DataFrame from JSON files.
`Session.read_parquet`(paths[, recursive, ...]) | Create a DataFrame from Parquet files.

## Partitioning Data

`DataFrame.repartition`(npartitions[, ...]) | Repartition the data into the given number of partitions.

## Transformations

Apply transformations and return a new DataFrame.

`Session.partial_sql`(query, *inputs, **kwargs) | Execute a SQL query on each partition of the input DataFrames.
`DataFrame.map`(sql_or_func, *[, schema]) | Apply a function to each row.
`DataFrame.map_batches`(func, *[, batch_size]) | Apply the given function to batches of data.
`DataFrame.flat_map`(sql_or_func, *[, schema]) | Apply a function to each row and flatten the result.
`DataFrame.filter`(sql_or_func, **kwargs) | Filter out rows that don't satisfy the given predicate.
`DataFrame.limit`(limit) | Limit the number of rows to the given number.
`DataFrame.partial_sort`(by, **kwargs) | Sort rows by the given columns in each partition.
`DataFrame.random_shuffle`(**kwargs) | Randomly shuffle all rows globally.

## Consuming Data

These operations will trigger execution of the lazy transformations performed on this DataFrame.

`DataFrame.count`() | Count the number of rows.
`DataFrame.take`(limit) | Return up to limit rows.
`DataFrame.take_all`() | Return all rows.
`DataFrame.to_arrow`() | Convert to an arrow Table.
`DataFrame.to_pandas`() | Convert to a pandas DataFrame.
`DataFrame.write_parquet`(path) | Write data to a series of parquet files under the given path.
`DataFrame.write_parquet_lazy`(path) | Write data to a series of parquet files under the given path.

## Execution

DataFrames are lazily computed. You can use these methods to manually trigger computation.

`DataFrame.compute`() | Compute the data.
`DataFrame.is_computed`() | Check if the data is ready on disk.
`DataFrame.recompute`() | Always recompute the data regardless of whether it's already computed.
`Session.wait`(*dfs) | Wait for all DataFrames to be computed.