Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up

Plugins/extension for Polars

License

NotificationsYou must be signed in to change notification settings

pola-rs/pyo3-polars

Repository files navigation

Documentation for this functionality may also be found in thePolars User Guide.This is new functionality and should be preferred over2. as thiswill circumvent the GIL and will be the way we want to support extending polars.

Parallelism and optimizations are managed by the default polars runtime. That runtime will call into the plugin function.The plugin functions are compiled separately.

We can therefore keep polars more lean and maybe add support for apolars-distance,polars-geo,polars-ml, etc.Those can then have specialized expressions and don't have to worry as much for code bloat as they can be optionally installed.

The idea is that you define an expression in another Rust crate with a proc_macropolars_expr.

The macro may have one of the following attributes:

  • output_type -> to define the output type of that expression
  • output_type_func -> to define a function that computes the output type based on input types.
  • output_type_func_with_kwargs -> to define a function that computes the output type based on input types and keyword args.

Here is an example of aString conversion expression that converts any string topig latin:

fnpig_latin_str(value:&str,capitalize:bool,output:&mutString){ifletSome(first_char) = value.chars().next(){if capitalize{for cin value.chars().skip(1).map(|char| char.to_uppercase()){write!(output,"{c}").unwrap()}write!(output,"AY").unwrap()}else{let offset = first_char.len_utf8();write!(output,"{}{}ay",&value[offset..], first_char).unwrap()}}}#[derive(Deserialize)]structPigLatinKwargs{capitalize:bool,}#[polars_expr(output_type=String)]fnpig_latinnify(inputs:&[Series],kwargs:PigLatinKwargs) ->PolarsResult<Series>{let ca = inputs[0].str()?;let out:StringChunked =        ca.apply_into_string_amortized(|value, output|pig_latin_str(value, kwargs.capitalize, output));Ok(out.into_series())}

This can then be exposed on the Python side:

from __future__importannotationsfromtypingimportTYPE_CHECKINGimportpolarsasplfrompolars.pluginsimportregister_plugin_functionfromexpression_lib._utilsimportLIBifTYPE_CHECKING:fromexpression_lib._typingimportIntoExprColumndefpig_latinnify(expr:IntoExprColumn,capitalize:bool=False)->pl.Expr:returnregister_plugin_function(plugin_path=LIB,args=[expr],function_name="pig_latinnify",is_elementwise=True,kwargs={"capitalize":capitalize},    )

Compile/ship and then it is ready to use:

importpolarsasplfromexpression_libimportlanguagedf=pl.DataFrame({"names": ["Richard","Alice","Bob"],})out=df.with_columns(pig_latin=language.pig_latinnify("names"))

Alternatively, you canregister a custom namespace, which enables you to write:

out=df.with_columns(pig_latin=pl.col("names").language.pig_latinnify())

See the full example in [example/derive_expression]:https://github.com/pola-rs/pyo3-polars/tree/main/example/derive_expression

2. Pyo3 extensions for Polars

See theexample directory for a concrete example. Here we send a polarsDataFrame to rust and then compute ajaccard similarity in parallel usingrayon and rust hash sets.

Run example

$ cd example && make install$ venv/bin/python run.py

This will output:

shape: (2, 2)┌───────────┬───────────────┐│ list_a    ┆ list_b        ││ ---       ┆ ---           ││ list[i64] ┆ list[i64]     │╞═══════════╪═══════════════╡│ [1, 2, 3] ┆ [1, 2, ... 8] ││ [5, 5]    ┆ [5, 1, 1]     │└───────────┴───────────────┘shape: (2, 1)┌─────────┐│ jaccard ││ ---     ││ f64     │╞═════════╡│ 0.75    ││ 0.5     │└─────────┘

Compile for release

$ make install-release

What to expect

This crate offers aPySeries and aPyDataFrame which are simple wrapper aroundSeries andDataFrame. Theadvantage of these wrappers is that they can be converted to and from python as they implementFromPyObject andIntoPy.

About

Plugins/extension for Polars

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

[8]ページ先頭

©2009-2025 Movatter.jp