@vmuriart These two comments are updated.

@yagweb can you update docs or readme with few examples?

@denfromufa The unit tests in "src/embed_tests/pyscope.cs" are commented and can be used as examples.

Currently the relationships among PythonEngine, Py, PyScope, Py.GILState, PyScope.GILState are a little messy. The APIs may changes at any time, so I may not add these examples into docs or readme until the APIs become stable. We can use the unit tests instead.

@yagweb should be clarify the intents/differences of them before adding the feature? I agree on not adding it to the documentedAPI until we hash it out though.

@yagweb@vmuriart I'm still struggling to understand use cases for scopes, except for storing local state.
Maybe I'm missing something? That's why I asked to add something in the docs.

@vmuriart I am sorry for that, the modifications are too many last time. Last time, the PR is cleaned up by separating a new PR ‘add eval and exec’ out from it, merging two classes, renaming methods, etc. I updated all my comments and forgot to inform you. In the future, I will provide detailed comments for each commit.

@denfromufa@vmuriart Storing local state is the start point. I wanted to embedding python into a C# application using pythonnet, and found the APIs for this scene are hard to use. In my another commenthere, I concluded that

A scope class aims at managing the context of a interaction with Python and simplify the variable exchanging. It can act like a Python Module in .NET. and can be shared. In the Scope class, many python functions can be wrapped to operate with the local variables and global variables for easy-to-use and speeding up, such as the ImportAs method.

I just make a new commit, now this PR implements almost all the functions of IronPython about accessing Python code from .NET code. I still don't know whether it's enough or not until it's actually used.

@denfromufa@vmuriart The most complicated thing about using pythonnet in .NET application is the GIL. This is the cause of what I said,

Currently the relationships among PythonEngine, Py, PyScope, Py.GILState, PyScope.GILState are a little messy.

Here is a relatedproposal, but I think it's impossible in some scenes.

I think there are four scenes when embedding CPython.

1. If the application is single thread, it's no need to AcquireLock and ReleaseLock all the time, we can AcquireLock when initialize and ReleaseLock when Shutdown.

2. In a multi-thread application, if several fast execution python functions need to called, we can treat the CPython as a shared resource, and the GIL as a lock in .NET. So a solution is wrapping each function with AcquireLock and ReleaseLock, this is what the pythonnet does right now.

3. In a multi-thread application, if the python functions needs waiting IO and very time consuming. Then the GIL cannot be managed in .NET code. The python threading can be used and the .NET code should use asynchronous calls to interacting with Python. A similar use case is the Dispatcher of WPF.

4. In a multi-thread application, if multi-cores are needed. Use CPython with multi-processing, or use pypy and IronPython instead.

In the multi-thread scenes, it's better to use IronPython if possible. In the 3rd scene, I think acquire GIL by default in .NET is impossible.

@yagweb in my view the multi-threading and asynchronous execution can be outside of this PR. GIL issues are well-known in Python community. Besides multi-processing in CPython you can look at Cython with nogil contexts, and Numba with nopython mode for numerical code. PyPy-STM is not ready for production use and only available on Linux. PyPy with cpyext is currently crashing when importing pythonnet:

#330

Even for the 1st scene thegc andFinalize inclassderived.cs might throw a wrench into things.

Looks like might need to rebase this one since#389 was merged in.

@denfromufa I see, thank you.
When users need multi-core parallel in embedded CPython, they should design their code with any means including Cython, Numba etc. It’s out of pythonnet. So the 4th scene can be deleted from this list.

@vmuriart Thank you. it seems like there no "real" single-threaded application since the GC working behind and needing to call CPython. So the 1st scene can also be deleted from this list.

@denfromufa@vmuriart May it be better to make the PyScope class independent from the GIL solution?

Codecov Report

Merging#381 intomaster willdecrease coverage by0.21%.
The diff coverage is58.79%.

@@            Coverage Diff             @@##           master     #381      +/-   ##==========================================- Coverage   68.56%   68.34%   -0.22%==========================================  Files          63       64       +1       Lines        5344     5560     +216       Branches      855      894      +39     ==========================================+ Hits         3664     3800     +136- Misses       1444     1496      +52- Partials      236      264      +28

Continue to review full report at Codecov.

Legend -Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing data
Powered byCodecov. Last updatece14424...b9dcdac. Read thecomment docs.

@denfromufa@vmuriart May it be better to make the PyScope class independent from the GIL solution?

I think it is safer that GIL is held inside PyScope by default. You can maybe pass non-default argument to release the GIL.

@yagweb just finished reviewing thispr and makes sense what you're proposing. I think there's some refactoring we can do to clean up the code and there's a couple tests I want to run (like what happens if we create twoPyScope instead of aPyScope and aSubScope) (does changing a variable in the subscope that exists on the parent scope modify it).

Looks like there is agc issue lingering somewhere.

https://travis-ci.org/pythonnet/pythonnet/jobs/204901628#L1606

@vmuriart how about adding this?

#400 (comment)

That wouldn't fix it. The issue is that the engine get shutdown earlier at "some point" and looks like currently there is nothing checking the status of the engine before trying to run more commands on it.

Rebased and updated some of the internal changes

@denfromufa One of@filmor's suggestions is,

The function names for accessing the variables are a bit too verbose for my taste, Get and Set would be enough.

My response is,

I agreed the names are too verbose, but these names and the Exception messages all came from IronPython. Just to be compatible.

Personally I also prefer to the shorter namesGet,Set,Remove andContains.

In addition,Executecan be renamed toExec,AddVariablescan be renamed toImportDict. Or maybe theImportDict,ImportModule, andImportScopecan all be renamed toImport.

So, compatible with IronPython or more concise, what's your opinion?

@yagweb will the scope class address the GET, SET issues discussedhere

@JimSEOW I put my modificationshere. If your have any other questions, please talk it there.

@yagweb regarding your previouscomment I would go for more concise,@tonyroberts@vmuriart@filmor do you agree?

@yagweb @denfromufa etc.
When Xamarin create .NET binding to C++ libraries, they try their best to .dotNETfy the binding using semantics that are characteristics to the .NET ecosystem.
Perhaps we can do the same here.
=> Get and Set are more the typical Java syntax for properties (not sure how easy it is to replace them with alternatives)
=> I prefer Execute to Exec
=> We are addressing users from both .NET and python ecosystems. It will be interesting to strike a balance. However, we are talking about Python 4 .NET here. What can we learn from Ironpython to suggest some naming suggestions here?

@filmor@vmuriart I would like to use this API during my upcoming presentation on June 8th at Microsoft. Please review and comment, if not accepting!

I'll review over the weekend. I don't think this one introduced any GC bugs but im seeing a bunch of warnings showing up on the Python Tests now (though I suspect its unrelated to this but want to check).

I'll review@filmor comments to make sure requests were resolved.

Regarding doing things like IronPython: I doubt that we actually want API parity on the .NET side, for that we are too far away. I also don't see much of a benefit, pythonnet is not built on top of the scripting architecture used by IronPython but directly on Python's API, so they can not be replacements for each other anyhow.

This is completely disconnected from the discussion how thePython side should look like.

@filmor Very thank you for your detailed and valuable comments.

@denfromufa@filmor@vmuriart The last question is, should the methods 'GetVariable', 'SetVariable' etc. be renamed to shorter ones as 'Get', 'Set' etc.?

@yagweb i like the shorter names Get & Set.

The names 'Get & Set' etc. make sense, because variables are the only content a scope managed.
An example of this design is the 'Add' method of the List.

Methods are renamed and documents are added.

@vmuriart i'm planning to merge this on Wednesday, let us know if you have any feedback before that.@yagweb great work, thanks for enduring so many iterations of reviews.