NotificationsYou must be signed in to change notification settings
Fork7.9k
Star21.4k

Commit138df72

committed

Use test cache for test result images too.

For image comparison tests in svg/pdf/ps formats, the result images areconverted to png for comparison. Previously the conversion results werecached for the baseline images, but not for the test-generated images(because of non-deterministic svg/pdf/etc. results, due tohash-salting, dict ordering, etc.).Now that the test-generated images are generally deterministic, we canenable the cache for baseline images as well. This speeds up`pytest -k '[svg]'` by ~30% (81s initially -> 55s on a seeded cache) and`pytest -k '[pdf]'` by ~10% (62s -> 55s) (there are too few (e)ps imagecomparison tests to see an effect). Also add logging regarding thecache which may help troubleshooting determinacy problems.A simple cache eviction mechanism prevents the cache from growingwithout bounds, limiting it to 2x the size of the baseline_imagesdirectory.This is a much simpler version of PR7764, which added more sophisticatedreporting of cache hits and misses and cache eviction.

1 parent34f99a9 commit138df72Copy full SHA for 138df72

File tree

1 file changed

+35

-1

lines changed

lib/matplotlib/testing
- compare.py

1 file changed

+35

-1

lines changed

`‎lib/matplotlib/testing/compare.py`

Lines changed: 35 additions & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -3,7 +3,9 @@`
`3`	`3`	`"""`
`4`	`4`
`5`	`5`	`importatexit`
	`6`	`+importfunctools`
`6`	`7`	`importhashlib`
	`8`	`+importlogging`
`7`	`9`	`importos`
`8`	`10`	`frompathlibimportPath`
`9`	`11`	`importre`
`@@ -19,6 +21,8 @@`
`19`	`21`	`frommatplotlibimportcbook`
`20`	`22`	`frommatplotlib.testing.exceptionsimportImageComparisonFailure`
`21`	`23`
	`24`	`+_log=logging.getLogger(__name__)`
	`25`	`+`
`22`	`26`	`__all__= ['compare_images','comparable_formats']`
`23`	`27`
`24`	`28`
`@@ -285,20 +289,50 @@ def convert(filename, cache):`
`285`	`289`	`cache_dir=Path(get_cache_dir())ifcacheelseNone`
`286`	`290`
`287`	`291`	`ifcache_dirisnotNone:`
	`292`	`+_register_conversion_cache_cleaner_once()`
`288`	`293`	`hash_value=get_file_hash(path)`
`289`	`294`	`cached_path=cache_dir/ (hash_value+newpath.suffix)`
`290`	`295`	`ifcached_path.exists():`
	`296`	`+_log.debug("For %s: reusing cached conversion.",filename)`
`291`	`297`	`shutil.copyfile(cached_path,newpath)`
`292`	`298`	`returnstr(newpath)`
`293`	`299`
	`300`	`+_log.debug("For %s: converting to png.",filename)`
`294`	`301`	`converter[path.suffix[1:]](path,newpath)`
`295`	`302`
`296`	`303`	`ifcache_dirisnotNone:`
	`304`	`+_log.debug("For %s: caching conversion result.",filename)`
`297`	`305`	`shutil.copyfile(newpath,cached_path)`
`298`	`306`
`299`	`307`	`returnstr(newpath)`
`300`	`308`
`301`	`309`
	`310`	`+def_clean_conversion_cache():`
	`311`	`+# This will actually ignore mpl_toolkits baseline images, but they're`
	`312`	`+# relatively small.`
	`313`	`+baseline_images_size=sum(`
	`314`	`+path.stat().st_size`
	`315`	`+forpathinPath(mpl.__file__).parent.glob("/baseline_images//*"))`
	`316`	`+# 2x: one full copy of baselines, and one full copy of test results`
	`317`	`+# (actually an overestimate: we don't convert png baselines and results).`
	`318`	`+max_cache_size=2*baseline_images_size`
	`319`	`+# Reduce cache until it fits.`
	`320`	`+cache_stat= {`
	`321`	`+path:path.stat()forpathinPath(get_cache_dir()).glob("*")}`
	`322`	`+cache_size=sum(stat.st_sizeforstatincache_stat.values())`
	`323`	`+paths_by_atime=sorted(# Oldest at the end.`
	`324`	`+cache_stat,key=lambdapath:cache_stat[path].st_atime,reverse=True)`
	`325`	`+whilecache_size>max_cache_size:`
	`326`	`+path=paths_by_atime.pop()`
	`327`	`+cache_size-=cache_stat[path].st_size`
	`328`	`+path.unlink()`
	`329`	`+`
	`330`	`+`
	`331`	`+@functools.lru_cache()# Ensure this is only registered once.`
	`332`	`+def_register_conversion_cache_cleaner_once():`
	`333`	`+atexit.register(_clean_conversion_cache)`
	`334`	`+`
	`335`	`+`
`302`	`336`	`defcrop_to_same(actual_path,actual_image,expected_path,expected_image):`
`303`	`337`	`# clip the images to the same size -- this is useful only when`
`304`	`338`	`# comparing eps to pdf`
`@@ -387,7 +421,7 @@ def compare_images(expected, actual, tol, in_decorator=False):`
`387`	`421`	`raiseIOError('Baseline image %r does not exist.'%expected)`
`388`	`422`	`extension=expected.split('.')[-1]`
`389`	`423`	`ifextension!='png':`
`390`		`-actual=convert(actual,cache=False)`
	`424`	`+actual=convert(actual,cache=True)`
`391`	`425`	`expected=convert(expected,cache=True)`
`392`	`426`
`393`	`427`	`# open the image files and remove the alpha channel (if it exists)`

0 commit comments

Comments

(0)

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Commit138df72

File tree

1 file changed

1 file changed

`‎lib/matplotlib/testing/compare.py`

0 commit comments