Documentation: Python - Rust - Node.js - R | StackOverflow: Python - Rust - Node.js - R | User guide | Discord
## Polars: Blazingly fast DataFrames in Rust, Python, Node.js, R, and SQL Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using [Apache Arrow Columnar Format](https://arrow.apache.org/docs/format/Columnar.html) as the memory model. - Lazy | eager execution - Multi-threaded - SIMD - Query optimization - Powerful expression API - Hybrid Streaming (larger-than-RAM datasets) - Rust | Python | NodeJS | R | ... To learn more, read the [user guide](https://docs.pola.rs/). ## Python ```python >>> import polars as pl >>> df = pl.DataFrame( ... { ... "A": [1, 2, 3, 4, 5], ... "fruits": ["banana", "banana", "apple", "apple", "banana"], ... "B": [5, 4, 3, 2, 1], ... "cars": ["beetle", "audi", "beetle", "beetle", "beetle"], ... } ... ) # embarrassingly parallel execution & very expressive query language >>> df.sort("fruits").select( ... "fruits", ... "cars", ... pl.lit("fruits").alias("literal_string_fruits"), ... pl.col("B").filter(pl.col("cars") == "beetle").sum(), ... pl.col("A").filter(pl.col("B") > 2).sum().over("cars").alias("sum_A_by_cars"), ... pl.col("A").sum().over("fruits").alias("sum_A_by_fruits"), ... pl.col("A").reverse().over("fruits").alias("rev_A_by_fruits"), ... pl.col("A").sort_by("B").over("fruits").alias("sort_A_by_B_by_fruits"), ... ) shape: (5, 8) ┌──────────┬──────────┬──────────────┬─────┬─────────────┬─────────────┬─────────────┬─────────────┐ │ fruits ┆ cars ┆ literal_stri ┆ B ┆ sum_A_by_ca ┆ sum_A_by_fr ┆ rev_A_by_fr ┆ sort_A_by_B │ │ --- ┆ --- ┆ ng_fruits ┆ --- ┆ rs ┆ uits ┆ uits ┆ _by_fruits │ │ str ┆ str ┆ --- ┆ i64 ┆ --- ┆ --- ┆ --- ┆ --- │ │ ┆ ┆ str ┆ ┆ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞══════════╪══════════╪══════════════╪═════╪═════════════╪═════════════╪═════════════╪═════════════╡ │ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 4 ┆ 4 │ │ "apple" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 7 ┆ 3 ┆ 3 │ │ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 5 ┆ 5 │ │ "banana" ┆ "audi" ┆ "fruits" ┆ 11 ┆ 2 ┆ 8 ┆ 2 ┆ 2 │ │ "banana" ┆ "beetle" ┆ "fruits" ┆ 11 ┆ 4 ┆ 8 ┆ 1 ┆ 1 │ └──────────┴──────────┴──────────────┴─────┴─────────────┴─────────────┴─────────────┴─────────────┘ ``` ## SQL ```python >>> df = pl.scan_ipc("file.arrow") >>> # create a SQL context, registering the frame as a table >>> sql = pl.SQLContext(my_table=df) >>> # create a SQL query to execute >>> query = """ ... SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM my_table ... WHERE id1 = 'id016' ... LIMIT 10 ... """ >>> ## OPTION 1 >>> # run the query, materializing as a DataFrame >>> sql.execute(query, eager=True) shape: (1, 2) ┌────────┬────────┐ │ sum_v1 ┆ min_v2 │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════╪════════╡ │ 298268 ┆ 1 │ └────────┴────────┘ >>> ## OPTION 2 >>> # run the query but don't immediately materialize the result. >>> # this returns a LazyFrame that you can continue to operate on. >>> lf = sql.execute(query) >>> (lf.join(other_table) ... .group_by("foo") ... .agg( ... pl.col("sum_v1").count() ... ).collect()) ``` SQL commands can also be run directly from your terminal using the Polars CLI: ```bash # run an inline SQL query > polars -c "SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM read_ipc('file.arrow') WHERE id1 = 'id016' LIMIT 10" # run interactively > polars Polars CLI v0.3.0 Type .help for help. > SELECT sum(v1) as sum_v1, min(v2) as min_v2 FROM read_ipc('file.arrow') WHERE id1 = 'id016' LIMIT 10; ``` Refer to the [Polars CLI repository](https://github.com/pola-rs/polars-cli) for more information. ## Performance 🚀🚀 ### Blazingly fast Polars is very fast. In fact, it is one of the best performing solutions available. See the results in [DuckDB's db-benchmark](https://duckdblabs.github.io/db-benchmark/). In the [TPC-H benchmarks](https://www.pola.rs/benchmarks.html) Polars is orders of magnitude faster than pandas, dask, modin and vaex on full queries (including IO). ### Lightweight Polars is also very lightweight. It comes with zero required dependencies, and this shows in the import times: - polars: 70ms - numpy: 104ms - pandas: 520ms ### Handles larger-than-RAM data If you have data that does not fit into memory, Polars' query engine is able to process your query (or parts of your query) in a streaming fashion. This drastically reduces memory requirements, so you might be able to process your 250GB dataset on your laptop. Collect with `collect(streaming=True)` to run the query streaming. (This might be a little slower, but it is still very fast!) ## Setup ### Python Install the latest Polars version with: ```sh pip install polars ``` We also have a conda package (`conda install -c conda-forge polars`), however pip is the preferred way to install Polars. Install Polars with all optional dependencies. ```sh pip install 'polars[all]' ``` You can also install a subset of all optional dependencies. ```sh pip install 'polars[numpy,pandas,pyarrow]' ``` | Tag | Description | | ---------- | ---------------------------------------------------------------------------- | | **all** | Install all optional dependencies (all of the following) | | pandas | Install with pandas for converting data to and from pandas DataFrames/Series | | numpy | Install with NumPy for converting data to and from NumPy arrays | | pyarrow | Reading data formats using PyArrow | | fsspec | Support for reading from remote file systems | | connectorx | Support for reading from SQL databases | | xlsx2csv | Support for reading from Excel files | | openpyxl | Support for reading from Excel files with native types | | deltalake | Support for reading and writing Delta Lake Tables | | pyiceberg | Support for reading from Apache Iceberg tables | | plot | Support for plot functions on DataFrames | | timezone | Timezone support, only needed if you are on Python<3.9 or Windows | Releases happen quite often (weekly / every few days) at the moment, so updating Polars regularly to get the latest bugfixes / features might not be a bad idea. ### Rust You can take latest release from `crates.io`, or if you want to use the latest features / performance improvements point to the `main` branch of this repo. ```toml polars = { git = "https://github.com/pola-rs/polars", rev = "
](https://www.jetbrains.com)