Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Has anyone used Apache Arrow to share python data with c#?#2670

Unanswered
dbeavon asked this question inQ&A
Discussion options

I see that there is a general purpose way to share buffer memory from python to C#
PyBuffer GetBuffer

I'm assuming that works great for numpy, pandas, and lots of other things.

Is this also suitable to Apache Arrow tables? Should we expect the Apache Arrow libraries to give us a specialized and ideomatic option, considering the fact that Arrow it is built for exchanging data and there are native implementations for just about every programming language.

Any tips would be appreciated.

You must be logged in to vote

Replies: 1 comment

Comment options

yeah in the pythonnet context Arrow still makes more sense than pushing everything through PyBuffer.
PyBuffer_GetBuffer works and it’s totally fine for simple contiguous numpy arrays. But once you move to tables, pandas data, nullable columns, strings, etc, you end up manually handling layout, dtype, nulls and lifetimes on the C# side. pythonnet doesn’t really remove that complexity.

arrow is basically built for this exact problem:

  • python produces data
  • c# consumes it
  • clear schema instead of raw pointer + length
  • zero-copy or minimal-copy options

typical setup with pythonnet:

  • python side: use pyarrow to create Table / RecordBatch
  • transfer via Arrow IPC (even in-memory stream works well)
  • c# side: read using Apache.Arrow

this integrates cleanly with pythonnet and avoids abusing the buffer protocol.

If you really need true zero-copy, Arrow C Data Interface can be used from pythonnet too, but lifetime management becomes tricky (Python GC, ownership rules, etc). For most cases IPC is simpler and safe.

so overall:

  • raw numeric buffers → PyBuffer is fine
  • structured/tabular data → Arrow is the right abstraction
    arrow isn’t extra overhead here, it’s the thing that actually makes Python ↔ C# data sharing sane.
You must be logged in to vote
0 replies
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Category
Q&A
Labels
None yet
2 participants
@dbeavon@AkhilSirvi

[8]ページ先頭

©2009-2025 Movatter.jp