- Notifications
You must be signed in to change notification settings - Fork773
Conversation
filmor left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thank you very much, this is a nice addition!
I've added a few general comments and will look into the Python-API details of this tomorrow.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
filmor commentedOct 27, 2019
By the way, the failing CI is probably since you didn't add |
thesn10 commentedOct 28, 2019
Fixed most of your comments. But i dont know why git marked almost all lines of the old csproject file as changed, even though i only added on line. I hope this is no problem :) |
filmor commentedOct 28, 2019
You probably changed the line endings |
codecov-io commentedOct 28, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Codecov Report
@@ Coverage Diff @@## master #980 +/- ##======================================= Coverage 86.71% 86.71% ======================================= Files 1 1 Lines 301 301 ======================================= Hits 261 261 Misses 40 40
Continue to review full report at Codecov.
|
lostmsu left a comment• edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This PR needs unit tests with good coverage. Currently, basic functionality is non-functional.
Uh oh!
There was an error while loading.Please reload this page.
src/runtime/pybuffer.cs Outdated
| _handle = _gchandle.AddrOfPinnedObject(); | ||
| _view = (Runtime.Py_buffer)Marshal.PtrToStructure(_handle, typeof(Runtime.Py_buffer)); | ||
| success = Runtime.PyObject_GetBuffer(obj, _handle, flags) >= 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Why return success/failure viaout, and not simply throw an exception?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
... in which case you also have to release resources.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Your right. My bad.
Uh oh!
There was an error while loading.Please reload this page.
src/runtime/pybuffer.cs Outdated
| byte[] rawData = new byte[size]; | ||
| _gchandle = GCHandle.Alloc(rawData, GCHandleType.Pinned); | ||
| _handle = _gchandle.AddrOfPinnedObject(); | ||
| _view = (Runtime.Py_buffer)Marshal.PtrToStructure(_handle, typeof(Runtime.Py_buffer)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is the only line where_view is initialized, however, at this moment the memory_handle points to is empty (e.g. all zeroes). So the_view is also always empty.
None of theifs below ever succeed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Your wrong._viewshould be empty, because PyObject_GetBuffer fills_view with data. Everyif below works as expected
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@Sngmng you might be using some funky runtime, which makes this happen.PyObject_GetBuffer does not get a pointer to_view. It gets a pointer to therawData instead. The call toMarshal.PtrToStructure simply creates a copy of memory it is pointing to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Oh my bad i messed up the order of these two lines when i was moving the code into the constructor. The PR was working before this commit tho.
src/runtime/pybuffer.cs Outdated
| } | ||
| } | ||
| public PyObject Object => new PyObject(_view.obj); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
CreatingPyObject does not increment reference counter of the object. You have to manually do it in this getter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
But PyObject_GetBuffer() does. And PyBuffer_Release decrements the reference count... Just read the docs first.https://docs.python.org/3/c-api/buffer.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
But in this case you actually get a new reference, so you have to increase the recount, or cache the instance of pyobject in constructor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Why am i getting a new refference? Isnt PyObject() just wrapping the handle that was already increfed? Whatever, im just going to add a XIncref as you say.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@Sngmng you are right, that it usually takes a pre-increfed pointer. But each instance requires its own incref. Otherwise when the instance is disposed, all other instances ofPyObject pointing to the same reference will simply crash on access.
src/runtime/pybuffer.cs Outdated
| public void Release() | ||
| { | ||
| Runtime.PyBuffer_Release(_handle); | ||
| _gchandle.Free(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It would be good to reset these handles to zero, and check, that the buffer has not been released yet.
src/runtime/runtime.cs Outdated
| public IntPtr buf; | ||
| public IntPtr obj; /* owned reference */ | ||
| public long len; | ||
| public long itemsize; /* This is Py_ssize_t so it can be |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Py_size_t is not the same asInt64 akalong. There is an ongoing discussion about sizes of Python types.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Well it works withlong and i didnt see a reason to usePy_ssize_t if it will get casted to long anyway. I looked it up in the python source and it litteralytypdefsPy_ssize_t as long. What do you recommend to do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I don't know yet, as we did not work it out. C'slong is not the same as C#'slong. The later is alwaysInt64.
This code won't work on 32 bit systems. There C'slong is almost always ==Int32
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@lostmsu What about something like this:
[StructLayout(LayoutKind.Sequential)]internal struct Py_ssize_t{ [MarshalAs(UnmanagedType.SysInt)] private IntPtr size; public long GetValue() { if (Is32Bit) return size.ToInt32(); else return size.ToInt64(); }}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
@Sngmng this struct might not work either, because on 64-bit windowsIntPtr is 64-bit, butPy_ssize_t is probably 32-bit.https://stackoverflow.com/questions/384502/what-is-the-bit-size-of-long-on-64-bit-windows
And that problem will have to be solved, before PR is accepted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
long and thus Py_ssize_t is 4 bytes on Windows (32 and 64 bit) and 8 bytes on other 64 bit systems. Very annoying.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
See my comment below, I made this mistake (assumingPy_ssize_t == long) before ;)
Py_ssize_t is a typedef forintptr_t orssize_t (on POSIX), so in all relevant cases it is equivalent in size toIntPtr.
thesn10 commentedOct 29, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Yeah it needs unit tests. But why are you claiming its non functional? Did you even test this PR? It works as expected. Just try out one of the examples i provided, you will see. Apart from that, im going to implement all of your other requested changes like implementing |
thesn10 commentedOct 29, 2019
Ive implemented most of the requested changes, but i dont know what you want me to do with the |
lostmsu commentedOct 29, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@Sngmng unfortunately, the only way I can think of so far is to have two struct types defined, and decide which one to use at runtime. It is a sad state of things, I know. |
lostmsu commentedOct 30, 2019
BTW, there appears to be a number of warnings regarding XML doc comments on the new file in the build, I guess related to the recent refactoring:https://ci.appveyor.com/project/pythonnet/pythonnet/builds/28472200/job/3otegb2hqnq9f6on |
src/runtime/pybuffer.cs Outdated
| if (disposedValue) | ||
| throw new ObjectDisposedException("PyBuffer"); | ||
| if (ReadOnly) | ||
| throw new Exception("Buffer is read-only"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
InvalidOperationException?
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
filmor commentedOct 30, 2019
Sorry for the confusion, we had this discussion already before (in#531) and the conclusion is that |
lostmsu commentedOct 30, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@filmor it might be fine for passing function parameters which will occupy an entire register anyway, but for structs incorrect size will cause the following fields to be incorrectly placed, leading to wrong memory being accessed. If our tests are run on Windows x64 (which we should do exactly for this kind of issue) they will likely catch the problem. |
filmor commentedOct 30, 2019
Hmm, it still seems to not quite work on x86, I'll try to reproduce this locally. |
lostmsu commentedOct 30, 2019
@filmor weird, should not be a problem on x86. |
filmor commentedOct 31, 2019
I'm not sure what exactly this one was the answer to, but Interestingly enough, non-xplat built Py3.7 ran through on x86... |
lostmsu commentedOct 31, 2019
thesn10 commentedNov 1, 2019
filmor commentedNov 1, 2019
Yeah, it doesn't really fail but times out. I'll try to reproduce it locally (running exactly what AppVeyor is supposed to run), but I don't use Windows as my primary dev platform, so I need a while to set everything up. |
filmor commentedNov 1, 2019
@lostmsu Could you rereview already? |
src/runtime/pybuffer.cs Outdated
| public static long SizeFromFormat(string format) | ||
| { | ||
| if (Runtime.pyversionnumber < 39) | ||
| throw new NotSupportedException("GetPointer requires at least Python 3.9"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Should the message say "SizeFromFormat ..." not "GetPointer ..."?
Uh oh!
There was an error while loading.Please reload this page.
src/runtime/pybuffer.cs Outdated
| /// Copy contiguous len bytes from buf to view. fort can be 'C' or 'F' (for C-style or Fortran-style ordering). 0 is returned on success, -1 on error. | ||
| /// </summary> | ||
| /// <returns>0 is returned on success, -1 on error.</returns> | ||
| public int FromContiguous(IntPtr buf, long len, char fort) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Since this is a public method, why not throw aPythonException on failure?
src/runtime/pybuffer.cs Outdated
| throw new ObjectDisposedException(nameof(PyBuffer)); | ||
| if (Runtime.pyversionnumber < 36) | ||
| throw new NotSupportedException("ToContiguous requires at least Python 3.6"); | ||
| return Runtime.PyBuffer_ToContiguous(buf, _handle, _view.len, order); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Same, why not throw?
src/runtime/pybuffer.cs Outdated
| { | ||
| if (disposedValue) | ||
| throw new ObjectDisposedException(nameof(PyBuffer)); | ||
| return Runtime.PyBuffer_FillInfo(_handle, exporter, buf, (IntPtr)len, _readonly, flags); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
And here
src/runtime/pybuffer.cs Outdated
| /// If this function is used as part of a getbufferproc, exporter MUST be set to the exporting object and flags must be passed unmodified.Otherwise, exporter MUST be NULL. | ||
| /// </remarks> | ||
| /// <returns>On success, set view->obj to a new reference to exporter and return 0. Otherwise, raise PyExc_BufferError, set view->obj to NULL and return -1;</returns> | ||
| public int FillInfo(IntPtr exporter, IntPtr buf, long len, int _readonly, int flags) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
_readonly should bebool
src/runtime/pybuffer.cs Outdated
| if (_view.ndim != 1) | ||
| throw new NotSupportedException("Multidimensional arrays, scalars and objects without a buffer are not supported."); | ||
| int copylen = count < (int)_view.len ? count : (int)_view.len; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This cast toint should bechecked to avoid data loss
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Also, I think it is wrong to silently write_view.len instead ofcount, whencount is> _view.len.
Return the number of bytes actually written, or throw an exception.
| if (_view.ndim != 1) | ||
| throw new NotSupportedException("Multidimensional arrays, scalars and objects without a buffer are not supported."); | ||
| int copylen = count < (int)_view.len ? count : (int)_view.len; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This cast toint should bechecked to avoid data loss
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
lostmsu commentedNov 1, 2019
@Sngmng what's your solution architecture? VS has unexpected behavior when project and solution don't match. Also try running tests from command line like CI does it, e.g. with |
thesn10 commentedNov 1, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@lostmsu I ran it through |
lostmsu commentedNov 1, 2019
@Sngmng you can try |
thesn10 commentedNov 1, 2019
If i try |
lostmsu commentedNov 2, 2019
@Sngmng build fails because of the wrong casing in Py_buffer.cs in csproj file |
lostmsu commentedNov 2, 2019
@Sngmng you can tryhttps://stackoverflow.com/questions/1507268/force-x86-clr-on-an-any-cpu-net-assembly If the binary is not AnyCPU, you could try building a x86 binary. |
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
lostmsu left a comment• edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Looks good to me, as soon as tests pass in CI.
Maybe update CHANGELOG?
| /// <summary> | ||
| /// Controls the <see cref="PyBuffer.Format"/> field. If set, this field MUST be filled in correctly. Otherwise, this field MUST be NULL. | ||
| /// </summary> | ||
| FORMATS = 0x0004, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This is originally namedFORMAT and notFORMATS but for some reason i get a "name not cls compilant" warning if i useFORMAT
lostmsu commentedNov 2, 2019
@Sngmng have you tried tests locally after removing finalizer? Do they pass? |
thesn10 commentedNov 2, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Visual studio fooled me that they pass but they didnt when i retested through |
lostmsu commentedNov 3, 2019
@Sngmng a sanity check: if you comment your tests out, do the tests pass? |
thesn10 commentedNov 6, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@lostmsu |
lostmsu commentedNov 6, 2019
@Sngmng hahaha, now I remember facing the same issue. Damn. Looks like we should add GIL checks to many entry points... |
thesn10 commentedNov 6, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@lostmsu yeah would be better hahaha. But the tests seem to fail again... seems like |
filmor commentedNov 7, 2019
At least the failure is now less erratic in that it fails exactly for all Python 3 versions on Windows running on x86 :) |
amos402 commentedNov 8, 2019
I'm not sure about that, maybe the cause is calling btw. I wonder why the |
filmor commentedNov 8, 2019
A |
lostmsu commentedMay 14, 2020
@Sngmng can you rebase/merge |
thesn10 commentedMay 14, 2020
yeah sure |
dnfclas commentedMay 14, 2020 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
thesn10 commentedMay 14, 2020
Why is appveyor configured to do the conda build? I mean it fails everytime and is ignored anyway? And it takes 6min which is a lot. Removing these checks would improve the test speed to 1min which is a 6 times speed improvement. I am waiting multiple hours for appveyor because it is so slow, just because of the unnecessary conda build... |
lostmsu commentedMay 14, 2020
Related:#1063 |
lostmsu commentedMay 14, 2020
@Sngmng the issue was just fixed in master by filmor. |
lostmsu commentedMay 22, 2020
thesn10 commentedMay 23, 2020
Be aware that the last commit to update the finalizer is still untested because i had some problems using nunit and couldnt run the tests. |
codecov-commenter commentedMay 23, 2020 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Codecov Report
@@ Coverage Diff @@## master #980 +/- ##========================================= Coverage ? 86.66% ========================================= Files ? 1 Lines ? 300 Branches ? 0 ========================================= Hits ? 260 Misses ? 40 Partials ? 0
Continue to review full report at Codecov.
|
lostmsu commentedMay 28, 2020
@Sngmng oh, your |
lostmsu commentedOct 30, 2020
Somehow this is already in o-O |

Uh oh!
There was an error while loading.Please reload this page.
What does this implement/fix? Explain your changes.
Passing data like images from managed to python will be easyer than ever!
I implemented the python buffer api which makes copying big chunks of data directly into the python object buffer possible!
You can copy strings to python object:
Or copy images to python byte arrays:
Does this close any currently open issues?
No
Any other comments?
...
Checklist
Check all those that are applicable and complete.
AUTHORSCHANGELOG