This PR stops installing nose, and uses pytest everywhere. Note that I did not yet delete any redundant builds. I want to make sure everything works with the different methods first and will then see about pruning stuff.

QuLogic added the topic: testing label

Jan 28, 2017

QuLogic added this to the2.1 (next point release) milestone

Jan 28, 2017

QuLogic force-pushed thepytest branch fromaa11a84 toa44514dCompare

January 28, 2017 23:52

QuLogic mentioned this pull request

Jan 29, 2017

Convert test decorators to pytest fixtures#7973

Merged

Copy link

Member

tacaswell commentedJan 29, 2017

attn@story645 I think I recognize this issue...

_________________________ TestUnitData.test_update_map _________________________[gw0] linux2 -- Python 2.7.9 /home/travis/build/matplotlib/matplotlib/venv/bin/pythonself = <matplotlib.tests.test_category.TestUnitData object at 0x7fffdfdb6910>    def test_update_map(self):        data = ['a', 'd']        oseq = ['a', 'd']        olocs = [0, 1]            data_update = ['b', 'd', 'e', np.inf]        useq = ['a', 'd', 'b', 'e', 'inf']        ulocs = [0, 1, 2, 3, -2]            unitdata = cat.UnitData(data)        assert unitdata.seq == oseq        assert unitdata.locs == olocs            unitdata.update(data_update)>       assert unitdata.seq == useqE       assert [u'a', u'd', u'b', u'e', u'i'] == ['a', 'd', 'b', 'e', 'inf']E         At index 4 diff: u'i' != u'inf'E         Use -v to get the full difflib/matplotlib/tests/test_category.py:44: AssertionError

@QuLogic Might want to turn off the fast-fail while we debug this?

Copy link

Member

story645 commentedJan 29, 2017

What version of numpy is that test using? that looks like the fun joy where numpy pre like 1.8 truncates inf and nan to the length of the largest string in the array, which in this case is 1.

Copy link

MemberAuthor

QuLogic commentedJan 29, 2017

It looks like NumPy 1.7.1.

Though I have a bigger question of why so few tests seem to be run compared to locally.

Copy link

Member

NelleV commentedJan 29, 2017

Side note: that PR is going to conflict against#7970

NelleV reviewed

Jan 29, 2017

View reviewed changes

tests.py

		@@ -4,10 +4,9 @@
		#

Copy link

Member

NelleVJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Can't we just remove this script?

Copy link

MemberAuthor

QuLogicJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Not against it, but also not going to make that decision unilaterally.

Copy link

Member

tacaswellJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I strongly prefer this script stay and remain functional.

lib/matplotlib/tests/test_ticker.py Outdated

		@@ -240,7 +237,7 @@ def _sub_labels(axis, subs=()):
		assert label_test == label_expected


		@cleanup(style='default')
		@pytest.mark.style('default')

Copy link

Member

NelleVJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why is the style classic being replaced by default?

Copy link

MemberAuthor

QuLogicJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Both lines say default here.

Copy link

Member

NelleVJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Indeed, they do… I thought I had seen some classic being converted to default, but I guess we'll see this once the tests run on travis anyways.

Copy link

MemberAuthor

QuLogicJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

There are a few that do; see#7973 for an explanation.

.travis.yml Outdated

		@@ -155,20 +151,20 @@ script:
		rm -rf ~/.cache/matplotlib
		fi
		export MPL_REPO_DIR=$PWD # needed for pep8-conformance test of the examples
		# Workaround for pytest-xdist flaky colletion order

Copy link

Member

NelleVJan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

nitpick: There is a typo here: colletion -> collection

Copy link

Member

NelleV commentedJan 29, 2017

@QuLogic We are setting the option to step testing after the first failure to speed up the tests.

QuLogic changed the title~~Switch testing to pytest completely~~[WIP] Switch testing to pytest completely

Jan 29, 2017

Copy link

MemberAuthor

QuLogic commentedJan 29, 2017

I've marked this[WIP] so that people focus on#7973 before this one; plus I have to figure out AppVeyor.

story645 mentioned this pull request

Jan 30, 2017

poor categorical support w/ numpy<1.8#7988

Closed

QuLogic force-pushed thepytest branch from7f9fa59 to3bda4ceCompare

January 31, 2017 01:41

QuLogic mentioned this pull request

Jan 31, 2017

Switch to pytest-pep8.#8001

Merged

QuLogic force-pushed thepytest branch from3bda4ce to78866d5Compare

February 1, 2017 07:17

Copy link

MemberAuthor

QuLogic commentedFeb 1, 2017•
edited
Loading

I believe this should be working now, once CI catches up. It currently includes#8001 and#8002 as well as several bug fixes that were missed due to the CI not running everything:

Changestacklevel of warning inFigure.gca because it's triggered in two tests and on Python 2.7 it doesn't get triggered properly in the second test.
Skip broken categories tests temporarily (seepoor categorical support w/ numpy<1.8 #7988)
Fix broken!= comparison withSubplotSpec
Normalize paths when looking for fonts because Windows requires it.
Remove one lastxfail that I missed in earlier PRs.

tacaswell closed this

Feb 1, 2017

tacaswell reopened this

Feb 1, 2017

Copy link

Member

tacaswell commentedFeb 1, 2017

power-cycled as the code being run on travis (from reading the errors) was not matching the code in the PR. I saw something like this on one of my day-job projects yesterday too. I think the web-hook is firing before the+refs/PRNUMBER/merge references are updating or something...

QuLogic added8 commits

February 2, 2017 18:18

Use pip to install in AppVeyor too.

6a91b9a

Rename test_only to avoid pytest collection.

ab99e65

Remove pytest collection filter.

bd91b93

It's really only needed to get the pep8 options.

Move pytest_configure into installed area.

25fdef8

This allows it to work when run via matplotlib.test().

Enable pytest-xdist on AppVeyor.

c8fc318

Change stack level of Figure.gca warning.

02d7cf8

Raising the warning on the current level breaks tests on Python 2.7which doesn't reset the warning filter correctly.

Skip broken category test with NumPy < 1.8.0.

30e40c6

Use pytest in matplotlib.test().

80281e9

QuLogic force-pushed thepytest branch from78866d5 to33a4a0eCompare

February 2, 2017 23:22

Stop installing nose in CI.

c520c2c

tacaswelland others added3 commits

February 2, 2017 18:39

FIX: implement__ne__ forSubplotSpec

ed7d353

In python3 `__ne__` automatically delegates to `__eq__`, in python2 itdoes not and falls back to the default `__ne__` test.  The NumPyversion of `assert_equal` actually checks `if a != b: raise`, whereas theversion of `assert_equal` from `unittest` (where `nose` ultimatelypulls from) checks `if not a == b: raise`

Normalize path case before checking for fonts.

ca255dd

This is a problem on Windows where file paths are case-insensitive.

Fix one last old xfail call.

3cb2752

QuLogic force-pushed thepytest branch from33a4a0e to3cb2752Compare

February 2, 2017 23:47

QuLogic changed the title~~[WIP] Switch testing to pytest completely~~[MRG] Switch testing to pytest completely

Feb 3, 2017

Copy link

MemberAuthor

QuLogic commentedFeb 3, 2017

We're good to go here now.

Fix tests.py when passed explicit filenames.

857d846

QuLogic force-pushed thepytest branch frome332caa to857d846Compare

February 3, 2017 02:19

tacaswell reviewed

Feb 3, 2017

View reviewed changes

.travis.yml Outdated

		@@ -225,7 +220,7 @@ after_success:
		else
		echo "Will only deploy docs build from matplotlib master branch"
		fi
		if [[ $NOSE_ARGS =~ "--with-coverage" \|\| $USE_PYTEST == true ]]; then
		if [[ $USE_PYTEST == true ]]; then

Copy link

Member

tacaswellFeb 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We should do this on all runs now?

Copy link

MemberAuthor

QuLogicFeb 3, 2017•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I was leaving that to#8003, but I see now it doesn't change this condition.@dopplershift Should I turn on coverage for everything here or just leave it for you?

Copy link

Contributor

dopplershiftFeb 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Go ahead and turn it on--I'll rebase on it.

tacaswell approved these changes

Feb 3, 2017

View reviewed changes

NelleV approved these changes

Feb 3, 2017

View reviewed changes

Copy link

Member

NelleV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

LGTM

NelleV changed the title~~[MRG] Switch testing to pytest completely~~[MRG+2] Switch testing to pytest completely

Feb 3, 2017

Always upload coverage on Travis.

fe20a23

QuLogic force-pushed thepytest branch from25724f7 tofe20a23Compare

February 3, 2017 06:39

dstansby approved these changes

Feb 3, 2017

View reviewed changes

dstansby merged commita517c60 intomatplotlib:master

Feb 3, 2017

dstansby changed the title~~[MRG+2] Switch testing to pytest completely~~Switch testing to pytest completely

Feb 3, 2017

dstansby mentioned this pull request

Feb 3, 2017

Document new testing procedure#8015

Closed

Copy link

Member

dstansby commentedFeb 3, 2017

The change to pytest will need documenting, but I've opened up a new issue for that at#8015.

Copy link

Member

tacaswell commentedFeb 3, 2017

1:======= 6470 passed, 10 skipped, 6 xfailed, 8 xpassed in 644.13 seconds ======== 6502
2:======= 6477 passed, 12 skipped, 2 xfailed, 3 xpassed in 609.06 seconds ======== 6497
3:======== 6485 passed, 4 skipped, 2 xfailed, 3 xpassed in 611.43 seconds ======== 6497
4: docs
5:======== 6485 passed, 4 skipped, 2 xfailed, 3 xpassed in 617.37 seconds ======== 6497
6:======= 7259 passed, 12 skipped, 2 xfailed, 3 xpassed in 610.41 seconds ======== 7276
7:====== 5256 passed, 1227 skipped, 10 xfailed, 1 xpassed in 573.52 seconds ====== 6494
8:======= 6477 passed, 12 skipped, 2 xfailed, 3 xpassed in 579.11 seconds ======== 6494