f2py script on Windows#

On Windows, the installed script for running f2py is now an.exe filerather than a*.py file and should be run from the command line asf2pywhenever theScripts directory is in the path. Runningf2py as a modulepython-mnumpy.f2py[...] will work without path modification in anyversion of NumPy.

NaT comparisons#

Consistent with the behavior of NaN, all comparisons other than inequalitychecks with datetime64 or timedelta64 NaT (“not-a-time”) values now alwaysreturnFalse, and inequality checks with NaT now always returnTrue.This includes comparisons between NaT values. For compatibility with theold behavior, usenp.isnat to explicitly check for NaT or convertdatetime64/timedelta64 arrays with.astype(np.int64) before makingcomparisons.

complex64/128 alignment has changed#

The memory alignment of complex types is now the same as a C-struct composed oftwo floating point values, while before it was equal to the size of the type.For many users (for instance on x64/unix/gcc) this means that complex64 is now4-byte aligned instead of 8-byte aligned. An important consequence is thataligned structured dtypes may now have a different size. For instance,np.dtype('c8,u1',align=True) used to have an itemsize of 16 (on x64/gcc)but now it is 12.

More in detail, the complex64 type now has the same alignment as a C-structstruct{floatr,i;}, according to the compiler used to compile numpy, andsimilarly for the complex128 and complex256 types.

nd_grid __len__ removal#

len(np.mgrid) andlen(np.ogrid) are now considered nonsensicaland raise aTypeError.

np.unravel_index now acceptsshape keyword argument#

Previously, only thedims keyword argument was acceptedfor specification of the shape of the array to be usedfor unraveling.dims remains supported, but is now deprecated.

multi-field views return a view instead of a copy#

Indexing a structured array with multiple fields, e.g.,arr[['f1','f3']],returns a view into the original array instead of a copy. The returned viewwill often have extra padding bytes corresponding to intervening fields in theoriginal array, unlike before, which will affect code such asarr[['f1','f3']].view('float64'). This change has been planned since numpy1.7. Operations hitting this path have emittedFutureWarnings since then.AdditionalFutureWarnings about this change were added in 1.12.

To help users update their code to account for these changes, a number offunctions have been added to thenumpy.lib.recfunctions module whichsafely allow such operations. For instance, the code above can be replacedwithstructured_to_unstructured(arr[['f1','f3']],dtype='float64').See the “accessing multiple fields” section of theuser guide.

Integrated squared error (ISE) estimator added tohistogram#

This method (bins='stone') for optimizing the bin number is ageneralization of the Scott’s rule. The Scott’s rule assumes the distributionis approximately Normal, while theISE is a non-parametric method based oncross-validation.

max_rows keyword added fornp.loadtxt#

New keywordmax_rows innumpy.loadtxt sets the maximum rows of thecontent to be read afterskiprows, as innumpy.genfromtxt.

modulus operator support added fornp.timedelta64 operands#

The modulus (remainder) operator is now supported for two operandsof typenp.timedelta64. The operands may have different unitsand the return value will match the type of the operands.

no-copy pickling of numpy arrays#

Up to protocol 4, numpy array pickling created 2 spurious copies of the databeing serialized. With pickle protocol 5, and thePickleBuffer API, alarge variety of numpy arrays can now be serialized without any copy usingout-of-band buffers, and with one less copy using in-band buffers. Thisresults, for large arrays, in an up to 66% drop in peak memory usage.

build shell independence#

NumPy builds should no longer interact with the host machineshell directly.exec_command has been replaced withsubprocess.check_output where appropriate.

np.polynomial.Polynomial classes render in LaTeX in Jupyter notebooks#

When used in a front-end that supports it,Polynomial instances are nowrendered through LaTeX. The current format is experimental, and is subject tochange.

randint andchoice now work on empty distributions#

Even when no elements needed to be drawn,np.random.randint andnp.random.choice raised an error when the arguments described an emptydistribution. This has been fixed so that e.g.np.random.choice([],0)==np.array([],dtype=float64).

linalg.lstsq,linalg.qr, andlinalg.svd now work with empty arrays#

Previously, aLinAlgError would be raised when an empty matrix/emptymatrices (with zero rows and/or columns) is/are passed in. Now outputs ofappropriate shapes are returned.

Chain exceptions to give better error messages for invalid PEP3118 format strings#

This should help track down problems.

Einsum optimization path updates and efficiency improvements#

Einsum was synchronized with the current upstream work.

numpy.angle andnumpy.expand_dims now work onndarray subclasses#

In particular, they now work for masked arrays.

NPY_NO_DEPRECATED_API compiler warning suppression#

SettingNPY_NO_DEPRECATED_API to a value of 0 will suppress the current compilerwarnings when the deprecated numpy API is used.

np.diff Added kwargs prepend and append#

New kwargsprepend andappend, allow for values to be inserted oneither end of the differences. Similar to options forediff1d. Now theinverse ofcumsum can be obtained easily viaprepend=0.

ARM support updated#

Support for ARM CPUs has been updated to accommodate 32 and 64 bit targets,and also big and little endian byte ordering. AARCH32 memory alignment issueshave been addressed. CI testing has been expanded to include AARCH64 targetsvia the services of shippable.com.

Appending to build flags#

numpy.distutils has always overridden rather than appended toLDFLAGS andother similar such environment variables for compiling Fortran extensions.Now, if theNPY_DISTUTILS_APPEND_FLAGS environment variable is set to 1, thebehavior will be appending. This applied to:LDFLAGS,F77FLAGS,F90FLAGS,FREEFLAGS,FOPT,FDEBUG, andFFLAGS. See gh-11525 for moredetails.

Generalized ufunc signatures now allow fixed-size dimensions#

By using a numerical value in the signature of a generalized ufunc, one canindicate that the given function requires input or output to have dimensionswith the given size. E.g., the signature of a function that converts a polarangle to a two-dimensional cartesian unit vector would be()->(2); thatfor one that converts two spherical angles to a three-dimensional unit vectorwould be(),()->(3); and that for the cross product of twothree-dimensional vectors would be(3),(3)->(3).

Note that to the elementary function these dimensions are not treated anydifferently from variable ones indicated with a name starting with a letter;the loop still is passed the corresponding size, but it can now count on thatsize being equal to the fixed one given in the signature.

Generalized ufunc signatures now allow flexible dimensions#

Some functions, in particular numpy’s implementation of@ asmatmul,are very similar to generalized ufuncs in that they operate over coredimensions, but one could not present them as such because they were able todeal with inputs in which a dimension is missing. To support this, it is nowallowed to postfix a dimension name with a question mark to indicate that thedimension does not necessarily have to be present.

With this addition, the signature formatmul can be expressed as(m?,n),(n,p?)->(m?,p?). This indicates that if, e.g., the second operandhas only one dimension, for the purposes of the elementary function it will betreated as if that input has core shape(n,1), and the output has thecorresponding core shape of(m,1). The actual output array, however, hasthe flexible dimension removed, i.e., it will have shape(...,m).Similarly, if both arguments have only a single dimension, the inputs will bepresented as having shapes(1,n) and(n,1) to the elementaryfunction, and the output as(1,1), while the actual output array returnedwill have shape(). In this way, the signature allows one to use asingle elementary function for four related but different signatures,(m,n),(n,p)->(m,p),(n),(n,p)->(p),(m,n),(n)->(m) and(n),(n)->().

np.clip and theclip method check for memory overlap#

Theout argument to these functions is now always tested for memory overlapto avoid corrupted results when memory overlap occurs.

New valueunscaled for optioncov innp.polyfit#

A further possible value has been added to thecov parameter of thenp.polyfit function. Withcov='unscaled' the scaling of the covariancematrix is disabled completely (similar to settingabsolute_sigma=True inscipy.optimize.curve_fit). This would be useful in occasions, where theweights are given by 1/sigma with sigma being the (known) standard errors of(Gaussian distributed) data points, in which case the unscaled matrix isalready a correct estimate for the covariance matrix.

Detailed docstrings for scalar numeric types#

Thehelp function, when applied to numeric types such asnumpy.intc,numpy.int_, andnumpy.longlong, now lists all of the aliased names for thattype, distinguishing between platform -dependent and -independent aliases.

__module__ attribute now points to public modules#

The__module__ attribute on most NumPy functions has been updated to referto the preferred public module from which to access a function, rather thanthe module in which the function happens to be defined. This produces moreinformative displays for functions in tools such as IPython, e.g., instead of<function'numpy.core.fromnumeric.sum'> you now see<function'numpy.sum'>.

Large allocations marked as suitable for transparent hugepages#

On systems that support transparent hugepages over the madvise system callnumpy now marks that large memory allocations can be backed by hugepages whichreduces page fault overhead and can in some fault heavy cases improveperformance significantly. On Linux the setting for huge pages to be used,/sys/kernel/mm/transparent_hugepage/enabled, must be at leastmadvise.Systems which already have it set toalways will not see much difference asthe kernel will automatically use huge pages where appropriate.

Users of very old Linux kernels (~3.x and older) should make sure that/sys/kernel/mm/transparent_hugepage/defrag is not set toalways to avoidperformance problems due concurrency issues in the memory defragmentation.

Alpine Linux (and other musl c library distros) support#

We now default to usefenv.h for floating point status error reporting.Previously we had a broken default that sometimes would not report underflow,overflow, and invalid floating point operations. Now we can support non-glibcdistributions like Alpine Linux as long as they shipfenv.h.

Speedupnp.block for large arrays#

Large arrays (greater than512*512) now use a blocking algorithm based oncopying the data directly into the appropriate slice of the resulting array.This results in significant speedups for these large arrays, particularly forarrays being blocked along more than 2 dimensions.

Speedupnp.take for read-only arrays#

The implementation ofnp.take no longer makes an unnecessary copy of thesource array when itswriteable flag is set toFalse.

Support path-like objects for more functions#

Thenp.core.records.fromfile function now supportspathlib.Pathand other path-like objects in addition to a file object. Furthermore, thenp.load function now also supports path-like objects when using memorymapping (mmap_mode keyword argument).

Better behaviour of ufunc identities during reductions#

Universal functions have an.identity which is used when.reduce iscalled on an empty axis.

As of this release, the logical binary ufuncs,logical_and,logical_or,andlogical_xor, now haveidentity s of typebool, where previously theywere of typeint. This restores the 1.14 behavior of gettingbool s whenreducing empty object arrays with these ufuncs, while also keeping the 1.15behavior of gettingint s when reducing empty object arrays with arithmeticufuncs likeadd andmultiply.

Additionally,logaddexp now has an identity of-inf, allowing it to becalled on empty sequences, where previously it could not be.

This is possible thanks to the newPyUFunc_FromFuncAndDataAndSignatureAndIdentity, which allowsarbitrary values to be used as identities now.

Improved conversion from ctypes objects#

Numpy has always supported taking a value or type fromctypes andconverting it into an array or dtype, but only behaved correctly for simplertypes. As of this release, this caveat is lifted - now:

• The_pack_ attribute ofctypes.Structure, used to emulate C’s__attribute__((packed)), is respected.

• Endianness of all ctypes objects is preserved

• ctypes.Union is supported

• Non-representable constructs raise exceptions, rather than producingdangerously incorrect results:

  • Bitfields are no longer interpreted as sub-arrays

  • Pointers are no longer replaced with the type that they point to

A newndpointer.contents member#

This matches the.contents member of normal ctypes arrays, and can be usedto construct annp.array around the pointers contents. This replacesnp.array(some_nd_pointer), which stopped working in 1.15. As a side effectof this change,ndpointer now supports dtypes with overlapping fields andpadding.

matmul is now aufunc#

numpy.matmul is now a ufunc which means that both the function and the__matmul__ operator can now be overridden by__array_ufunc__. Itsimplementation has also changed. It uses the same BLAS routines asnumpy.dot, ensuring its performance is similar for large matrices.

Start and stop arrays forlinspace,logspace andgeomspace#

These functions used to be limited to scalar stop and start values, but cannow take arrays, which will be properly broadcast and result in an outputwhich has one axis prepended. This can be used, e.g., to obtain linearlyinterpolated points between sets of points.

CI extended with additional services#

We now use additional free CI services, thanks to the companies that provide:

• Codecoverage testing via codecov.io

• Arm testing via shippable.com

• Additional test runs on azure pipelines

These are in addition to our continued use of travis, appveyor (for wheels) andLGTM

Comparison ufuncs will now error rather than return NotImplemented#

Previously, comparison ufuncs such asnp.equal would returnNotImplemented if their arguments had structured dtypes, to help comparisonoperators such as__eq__ deal with those. This is no longer needed, as therelevant logic has moved to the comparison operators proper (which thus docontinue to returnNotImplemented as needed). Hence, like all other ufuncs,the comparison ufuncs will now error on structured dtypes.

Positive will now raise a deprecation warning for non-numerical arrays#

Previously,+array unconditionally returned a copy. Now, it willraise aDeprecationWarning if the array is not numerical (i.e.,ifnp.positive(array) raises aTypeError. Forndarraysubclasses that override the default__array_ufunc__ implementation,theTypeError is passed on.

NDArrayOperatorsMixin now implements matrix multiplication#

Previously,np.lib.mixins.NDArrayOperatorsMixin did not implement thespecial methods for Python’s matrix multiplication operator (@). This haschanged now thatmatmul is a ufunc and can be overridden using__array_ufunc__.

The scaling of the covariance matrix innp.polyfit is different#

So far,np.polyfit used a non-standard factor in the scaling of the thecovariance matrix. Namely, rather than using the standardchisq/(M-N), itscaled it withchisq/(M-N-2) where M is the number of data points and N is thenumber of parameters. This scaling is inconsistent with other fitting programssuch as e.g.scipy.optimize.curve_fit and was changed tochisq/(M-N).

maximum andminimum no longer emit warnings#

As part of code introduced in 1.10,float32 andfloat64 set invalidfloat status when a Nan is encountered innumpy.maximum andnumpy.minimum,when using SSE2 semantics. This caused aRuntimeWarning to sometimes beemitted. In 1.15 we fixed the inconsistencies which caused the warnings tobecome more conspicuous. Now no warnings will be emitted.

Umath and multiarray c-extension modules merged into a single module#

The two modules were merged, according toNEP 15. Previouslynp.core.umathandnp.core.multiarray were separate c-extension modules. They are now pythonwrappers to the singlenp.core/_multiarray_math c-extension module.

getfield validity checks extended#

numpy.ndarray.getfield now checks the dtype and offset arguments to preventaccessing invalid memory locations.

NumPy functions now support overrides with__array_function__#

NumPy has a new experimental mechanism for overriding the implementation ofalmost all NumPy functions on non-NumPy arrays by defining an__array_function__ method, as described inNEP 18.

This feature is not yet been enabled by default, but has been released tofacilitate experimentation by potential users. See the NEP for details onsetting the appropriate environment variable. We expect the NumPy 1.17 releasewill enable overrides by default, which will also be more performant due to anew implementation written in C.

Arrays based off readonly buffers cannot be setwriteable#

We now disallow setting thewriteable flag True on arrays createdfromfromstring(readonly-buffer).