Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit02ee072

Browse files
authored
Test array-api-tests and add ones_like (#23)
* Test array-api-tests and add ones_like* ruff* requirements* */* improvments* better processing of tuple* improvments* add missing argument* fix one issue* refactoring* documentation* more coverage for the functions* refactoring* documentation* fix issue with garbage collector and OrtTensor initilized from numpy array* add a unit test for bfloat16* disable one test for ones_like* lint* fix skipif* api
1 parent934a139 commit02ee072

27 files changed

+1001
-370
lines changed

‎_doc/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ well as to execute it.
3333

3434
tutorial/index
3535
api/index
36+
tech/index
3637
auto_examples/index
3738
../CHANGELOGS
3839

‎_doc/tech/aapi.rst‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
2+
Difficulty to implement an an Array API for ONNX
3+
================================================
4+
5+
Implementing the full array API is not always easy with:epkg:`onnx`.
6+
Python is not strongly typed and many different types can be used
7+
to represent a value. Argument *axis* can be an integer or a tuple
8+
(see `min from Array API
9+
<https://data-apis.org/array-api/2022.12/API_specification/
10+
generated/array_api.min.html>`
11+
for example). On the other side, `ReduceMin from ONNX
12+
<https://onnx.ai/onnx/operators/onnx__ReduceMin.html>`_
13+
is considered as a tensor.
14+
15+
Performance
16+
+++++++++++
17+
18+
The Array API must work in eager mode and for every operation,
19+
it generates an ONNX graph and executes it with a specific
20+
backend. It can be:epkg:`numpy`,:epkg:`onnxruntime` or any other
21+
backend. The generation of every graph takes a significant amount of time.
22+
It must be avoided. These graphs are cached. But a graph can be reused
23+
only if the inputs - by ONNX semantic - change. If a parameter change,
24+
a new graph must be cached. Method:meth:`JitEager.make_key`
25+
generates a unique key based on the input it receives,
26+
the signature of the function to call. If the key is the same,
27+
a cached onnx can be reused on the second call.
28+
29+
However, eager mode - use a small single onnx graph for every operation -
30+
is not the most efficient one. At the same time, the design must allow
31+
to merge every needed operation into a bigger graph.
32+
Bigger graphs can be more easily optimized by the backend.
33+
34+
Input vs parameter
35+
++++++++++++++++++
36+
37+
An input is a tensor or array, a parameter is any other type.
38+
Following onnx semantic, an input is variable, a parameter is frozen
39+
cannot be changed. It is a constant. A good design would be
40+
to considered any named input (`**kwargs`) a parameter and
41+
any input (`*args`) a tensor. But the Array API does not follow that
42+
design. Function `astype
43+
<https://data-apis.org/array-api/2022.12/API_specification/
44+
generated/array_api.astype.html>_`
45+
takes two inputs. Operator `Cast
46+
<https://onnx.ai/onnx/operators/onnx__Cast.html>_`
47+
takes one input and a frozen parameter `to`.
48+
And python allows `astype(x, dtype)` as well as `astype(x, dtype=dtype)`
49+
unless the signature enforces one call over another type.
50+
There may be ambiguities from time to time.
51+
Beside, from onnx point of view, argument dtype should be named.
52+
53+
Tensor type
54+
+++++++++++
55+
56+
An:class:`EagerTensor` must be used to represent any tensor.
57+
This class defines the backend to use as well.
58+
`EagerNumpyTensor` for:epkg:`numpy`, `EagerOrtTensor`
59+
for:epkg:`onnxruntime`. Since the Array API is new,
60+
existing packages do not fully support the API if they support it
61+
(:epkg:`scikit-learn`). Some numpy array may still be used.
62+
63+
Inplace
64+
+++++++
65+
66+
ONNX has no notion of inplace computation. Therefore something
67+
like `coefs[:, 1] = 1` is not valid unless some code is written
68+
to create another tensor. The current design supports some of these
69+
by storing every call to `__setitem__`. The user sees `coefs`
70+
but the framework sees that `coefs` holds a reference to another
71+
tensor. That's the one the framework needs to use. However, since
72+
`__setitem__` is used for efficiency, it becomes less than efficient
73+
with this design and should be avoided. This assumption may be true
74+
when the backend is relying on CPU but not on GPU.
75+
A function such as `empty
76+
<https://data-apis.org/array-api/2022.12/API_specification/
77+
generated/array_api.astype.html>`_ should be avoided as it
78+
has to be followed by calls to `__setitem__`.
79+
80+
Eager or compilation
81+
++++++++++++++++++++
82+
83+
Eager mode is what the Array API implies.
84+
Every function is converted into an ONNX graph based
85+
on its inputs without any knownledge of how these inputs
86+
were obtained. This graph is then executed before going
87+
to the next call of a function from the API.
88+
The conversion of a machine learned model
89+
into ONNX implies the gathering of all these operations
90+
into a graph. It means using a mode that records all the function
91+
calls to compile every tiny onnx graph into a unique graph.
92+
93+
Iterators and Reduction
94+
+++++++++++++++++++++++
95+
96+
An efficient implementation of function
97+
:func:`numpy.any` or:func:`numpy.all` returns
98+
as soon as the result is known.:func:`numpy.all` is
99+
false whenever the first false condition is met.
100+
Same goes for:func:`numpy.any` which is true
101+
whenever the first true condition is met.
102+
There is no such operator in ONNX (<= 20) because
103+
it is unlikely to appear in a machine learned model.
104+
However, it is highly used when two results are
105+
compared in unit tests. The ONNX implementation is
106+
not efficient due to that reason but it only impacts
107+
the unit tests.
108+
109+
Types
110+
+++++
111+
112+
:epkg:`onnx` supports more types than:epkg:`numpy` does.
113+
It is not always easy to deal with bfloat16 or float8 types.

‎_doc/tech/index.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Technical Details
2+
=================
3+
4+
..toctree::
5+
:maxdepth:2
6+
7+
aapi

‎_unittests/onnx-numpy-skips.txt‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
# API failures
22
# see https://github.com/data-apis/array-api-tests/blob/master/numpy-skips.txt
33
array_api_tests/test_creation_functions.py::test_asarray_scalars
4-
#array_api_tests/test_creation_functions.py::test_arange
4+
array_api_tests/test_creation_functions.py::test_arange
55
array_api_tests/test_creation_functions.py::test_asarray_arrays
66
array_api_tests/test_creation_functions.py::test_empty
77
array_api_tests/test_creation_functions.py::test_empty_like
88
array_api_tests/test_creation_functions.py::test_eye
99
array_api_tests/test_creation_functions.py::test_full_like
1010
array_api_tests/test_creation_functions.py::test_linspace
1111
array_api_tests/test_creation_functions.py::test_meshgrid
12-
array_api_tests/test_creation_functions.py::test_ones_like
12+
# Issue with CastLike and bfloat16 on onnx <= 1.15.0
13+
# array_api_tests/test_creation_functions.py::test_ones_like
1314
array_api_tests/test_creation_functions.py::test_zeros_like

‎_unittests/test_array_api.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
export ARRAY_API_TESTS_MODULE=onnx_array_api.array_api.onnx_numpy
2-
pytest ../array-api-tests/array_api_tests/test_creation_functions.py::test_arange||exit 1
2+
pytest-v -rxXfE../array-api-tests/array_api_tests/test_creation_functions.py::test_ones_like||exit 1
33
# pytest ../array-api-tests/array_api_tests/test_creation_functions.py --help
4-
pytest ../array-api-tests/array_api_tests/test_creation_functions.py --hypothesis-explain --skips-file=_unittests/onnx-numpy-skips.txt||exit 1
4+
pytest-v -rxXfE../array-api-tests/array_api_tests/test_creation_functions.py --hypothesis-explain --skips-file=_unittests/onnx-numpy-skips.txt||exit 1

‎_unittests/ut_array_api/test_array_apis.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ def test_zeros_numpy_1(self):
1818
deftest_zeros_ort_1(self):
1919
c=xpo.zeros(1)
2020
d=c.numpy()
21-
self.assertEqualArray(np.array([0],dtype=np.float32),d)
21+
self.assertEqualArray(np.array([0],dtype=np.float64),d)
2222

2323
deftest_ffinfo(self):
2424
dt=np.float32
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
importunittest
2+
importwarnings
3+
fromosimportgetenv
4+
fromfunctoolsimportreduce
5+
fromoperatorimportmul
6+
fromhypothesisimportgiven
7+
fromonnx_array_api.ext_test_caseimportExtTestCase
8+
fromonnx_array_api.array_apiimportonnx_numpyasonxp
9+
fromhypothesisimportstrategies
10+
fromhypothesis.extraimportarray_api
11+
12+
13+
defprod(seq):
14+
returnreduce(mul,seq,1)
15+
16+
17+
@strategies.composite
18+
defarray_api_kwargs(draw,**kw):
19+
result= {}
20+
fork,stratinkw.items():
21+
ifdraw(strategies.booleans()):
22+
result[k]=draw(strat)
23+
returnresult
24+
25+
26+
defshapes(xp,**kw):
27+
kw.setdefault("min_dims",0)
28+
kw.setdefault("min_side",0)
29+
30+
defsh(x):
31+
returnx
32+
33+
returnxp.array_shapes(**kw).filter(
34+
lambdashape:prod(iforiinsh(shape)ifi)
35+
<TestHypothesisArraysApis.MAX_ARRAY_SIZE
36+
)
37+
38+
39+
classTestHypothesisArraysApis(ExtTestCase):
40+
MAX_ARRAY_SIZE=10000
41+
VERSION="2021.12"
42+
43+
@classmethod
44+
defsetUpClass(cls):
45+
withwarnings.catch_warnings():
46+
warnings.simplefilter("ignore")
47+
fromnumpyimportarray_apiasxp
48+
49+
api_version=getenv(
50+
"ARRAY_API_TESTS_VERSION",
51+
getattr(xp,"__array_api_version__",TestHypothesisArraysApis.VERSION),
52+
)
53+
cls.xps=array_api.make_strategies_namespace(xp,api_version=api_version)
54+
api_version=getenv(
55+
"ARRAY_API_TESTS_VERSION",
56+
getattr(onxp,"__array_api_version__",TestHypothesisArraysApis.VERSION),
57+
)
58+
cls.onxps=array_api.make_strategies_namespace(onxp,api_version=api_version)
59+
60+
deftest_strategies(self):
61+
self.assertNotEmpty(self.xps)
62+
self.assertNotEmpty(self.onxps)
63+
64+
deftest_scalar_strategies(self):
65+
dtypes=dict(
66+
integer_dtypes=self.xps.integer_dtypes(),
67+
uinteger_dtypes=self.xps.unsigned_integer_dtypes(),
68+
floating_dtypes=self.xps.floating_dtypes(),
69+
numeric_dtypes=self.xps.numeric_dtypes(),
70+
boolean_dtypes=self.xps.boolean_dtypes(),
71+
scalar_dtypes=self.xps.scalar_dtypes(),
72+
)
73+
74+
dtypes_onnx=dict(
75+
integer_dtypes=self.onxps.integer_dtypes(),
76+
uinteger_dtypes=self.onxps.unsigned_integer_dtypes(),
77+
floating_dtypes=self.onxps.floating_dtypes(),
78+
numeric_dtypes=self.onxps.numeric_dtypes(),
79+
boolean_dtypes=self.onxps.boolean_dtypes(),
80+
scalar_dtypes=self.onxps.scalar_dtypes(),
81+
)
82+
83+
fork,vnpindtypes.items():
84+
vonxp=dtypes_onnx[k]
85+
anp=self.xps.arrays(dtype=vnp,shape=shapes(self.xps))
86+
aonxp=self.onxps.arrays(dtype=vonxp,shape=shapes(self.onxps))
87+
self.assertNotEmpty(anp)
88+
self.assertNotEmpty(aonxp)
89+
90+
args_np= []
91+
92+
@given(
93+
x=self.xps.arrays(dtype=dtypes["integer_dtypes"],shape=shapes(self.xps)),
94+
kw=array_api_kwargs(dtype=strategies.none()|self.xps.scalar_dtypes()),
95+
)
96+
deffct(x,kw):
97+
args_np.append((x,kw))
98+
99+
fct()
100+
self.assertEqual(len(args_np),100)
101+
102+
args_onxp= []
103+
104+
xshape=shapes(self.onxps)
105+
xx=self.onxps.arrays(dtype=dtypes_onnx["integer_dtypes"],shape=xshape)
106+
kw=array_api_kwargs(dtype=strategies.none()|self.onxps.scalar_dtypes())
107+
108+
@given(x=xx,kw=kw)
109+
deffctonx(x,kw):
110+
args_onxp.append((x,kw))
111+
112+
fctonx()
113+
self.assertEqual(len(args_onxp),len(args_np))
114+
115+
116+
if__name__=="__main__":
117+
cl=TestHypothesisArraysApis()
118+
cl.setUpClass()
119+
cl.test_scalar_strategies()
120+
unittest.main(verbosity=2)

‎_unittests/ut_array_api/test_onnx_numpy.py‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
11
importsys
22
importunittest
3+
frompackaging.versionimportVersion
34
importnumpyasnp
5+
fromonnximportTensorProto,__version__asonnx_ver
46
fromonnx_array_api.ext_test_caseimportExtTestCase
57
fromonnx_array_api.array_apiimportonnx_numpyasxp
8+
fromonnx_array_api.npx.npx_typesimportDType
69
fromonnx_array_api.npx.npx_numpy_tensorsimportEagerNumpyTensorasEagerTensor
710

811

@@ -52,6 +55,13 @@ def test_ones_none(self):
5255
self.assertNotEmpty(matnp[0,0])
5356
self.assertEqualArray(matnp,np.ones((4,5)))
5457

58+
deftest_ones_like(self):
59+
x=np.array([5,6],dtype=np.int8)
60+
y=np.ones_like(x)
61+
a=EagerTensor(x)
62+
b=xp.ones_like(a)
63+
self.assertEqualArray(y,b.numpy())
64+
5565
deftest_full(self):
5666
c=EagerTensor(np.array([4,5],dtype=np.int64))
5767
mat=xp.full(c,fill_value=5,dtype=xp.int64)
@@ -89,7 +99,25 @@ def test_arange_int00(self):
8999
expected=expected.astype(np.int64)
90100
self.assertEqualArray(matnp,expected)
91101

102+
@unittest.skipIf(
103+
Version(onnx_ver)<Version("1.15.0"),
104+
reason="Reference implementation of CastLike is bugged.",
105+
)
106+
deftest_ones_like_uint16(self):
107+
x=EagerTensor(np.array(0,dtype=np.uint16))
108+
y=np.ones_like(x.numpy())
109+
z=xp.ones_like(x)
110+
self.assertEqual(y.dtype,x.numpy().dtype)
111+
self.assertEqual(x.dtype,z.dtype)
112+
self.assertEqual(x.dtype,DType(TensorProto.UINT16))
113+
self.assertEqual(z.dtype,DType(TensorProto.UINT16))
114+
self.assertEqual(x.numpy().dtype,np.uint16)
115+
self.assertEqual(z.numpy().dtype,np.uint16)
116+
self.assertNotIn("bfloat16",str(z.numpy().dtype))
117+
expected=np.array(1,dtype=np.uint16)
118+
self.assertEqualArray(expected,z.numpy())
119+
92120

93121
if__name__=="__main__":
94-
TestOnnxNumpy().test_arange_int00()
122+
#TestOnnxNumpy().test_ones_like()
95123
unittest.main(verbosity=2)

‎_unittests/ut_array_api/test_onnx_ort.py‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
importnumpyasnp
33
fromonnx_array_api.ext_test_caseimportExtTestCase
44
fromonnx_array_api.array_apiimportonnx_ortasxp
5+
fromonnx_array_api.npx.npx_numpy_tensorsimportEagerNumpyTensor
56
fromonnx_array_api.ort.ort_tensorsimportEagerOrtTensorasEagerTensor
67

78

@@ -15,6 +16,42 @@ def test_abs(self):
1516
a=xp.absolute(mat)
1617
self.assertEqualArray(np.absolute(mat.numpy()),a.numpy())
1718

19+
deftest_matmul(self):
20+
forclsin [EagerTensor,EagerNumpyTensor]:
21+
fordtypein (np.float32,np.float64):
22+
X=cls(
23+
np.array(
24+
[[-1,-1], [-2,-1], [-3,-2], [1,1], [2,1], [3,2]],
25+
dtype=dtype,
26+
)
27+
)
28+
coef=cls(np.array([[1e-13,8]],dtype=dtype).T)
29+
self.assertEqualArray(
30+
np.array([[1e-13,8]],dtype=dtype),coef.numpy().T
31+
)
32+
expected=X.numpy() @coef.numpy()
33+
got=X @coef
34+
try:
35+
self.assertEqualArray(expected,got.numpy())
36+
exceptAssertionErrorase:
37+
raiseAssertionError(
38+
f"Discrepancies (1) with cls={cls.__name__}, dtype={dtype}"
39+
)frome
40+
41+
coef=np.array([[1e-13,8]],dtype=dtype).T
42+
expected=X.numpy() @coef
43+
got=X @coef
44+
try:
45+
self.assertEqualArray(expected,got.numpy())
46+
exceptAssertionErrorase:
47+
raiseAssertionError(
48+
f"Discrepancies (2) with cls={cls.__name__}, dtype={dtype}"
49+
)frome
50+
1851

1952
if__name__=="__main__":
53+
# import logging
54+
55+
# logging.basicConfig(level=logging.DEBUG)
56+
# TestOnnxOrt().test_matmul()
2057
unittest.main(verbosity=2)

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp