Due to how Agg works under the covers, this actually needs to be sqrt(INT_MAX). That's where the original limit (which goes back to the 32-bit days) comes from.

Michael,

Thanks for this quick answer.

OK. Nevertheless, from my tests, this fix allowed me to write png images
with size far bigger than sqrt(INT_MAX).

In my case, INT_MAX=2147483647 so sqrt(INT_MAX)=46340.95 and I succeed
to write images that have a 700000 pixels width...

But I understand this fix could led to portability (and memory !) issues.

Well, I hope that at least this message could help people in a similar
situation.

Le 02/09/2014 17:49, Michael Droettboom a écrit :

Due to how Agg works under the covers, this actually needs to be
sqrt(INT_MAX). That's where the original limit (which goes back to the
32-bit days) comes from.

Nicolas PASCAL

HYGEOS - ICARE UMS 2877
Université Lille 1
Polytech'Lille - Bâtiment Eydoux - Bureau E207
Avenue Paul Langevin
59650 Villeneuve d'Ascq Cedex
France

Tel : +33 (0)3 20 33 59 77
Fax : +33 (0)3 20 41 75 47

Another problem raised after releasing the tests on pixels coordinates bigger than 32768 :

while trying to draw a colormesh after the 32768 pixel position (a colorbar actually), I have seen that it wasn't correctly drawn, whereas a standard image item was.

Here is the code snippet I used to arise the problem :

importnumpyimportosimportmatplotlib# huge images can't be displayed on a screen, so directly use Aggmatplotlib.use("Agg")importmatplotlib.pyplotasplt# init the figurefig_w,fig_h=33000,500# pixelsdpi=100fig=plt.figure(figsize=(float(fig_w)/float(dpi),float(fig_h)/float(dpi)),dpi=dpi,frameon=False)# create some 2D data to plotX,Y=numpy.mgrid[-5:5:0.05,-5:5:0.05]Z=numpy.sqrt(X**2+Y**2)+numpy.sin(X**2+Y**2)# create the axe around the pixel position 32768 where appears the bugim_h,im_w=Z.shapeim_l=32768-im_w/2.ax=plt.Axes(fig, [float(im_l)/float(fig_w),0,float(im_w)/float(fig_w),1],frameon=False)ax.yaxis.set_visible(False)ax.xaxis.set_visible(False)fig.add_axes(ax)# plot the data#   image based objects based are successfully displayed : imshow, pcolorfast#   mesh based objects basedfail to be displayed : pcolor, pcolormesh#ax.imshow(Z) # Success#ax.pcolorfast(Z) # Sucessax.pcolormesh(Z)# Failure#ax.pcolor(Z) # Failure# see what's going onfname="out.png"plt.savefig(fname)os.system("gimp "+fname)

Before the fix, I obtained an image like this (32500 pixel columns have been removed on the left of the images to focus on the problem)

The image was truncated after the 32768 column. It was obviously a 16 bits issue, so I changed the type of rendering buffer in _backend_agg.h for using 32 bits display coordinates.

The rendering is good now :

I think this is an example of the problem reported in#2105

You're right, it seems to be correlated.

But, at least with a version of matplotlib >= 1.4, it already worked with the
imshow method (I didn't tried matshow)

The problem occurs only when using QuadMesh based objects, like
pcolor and pcolormesh (and so colorbars)

I know it can sound a bit strange to work with this kind of huge images,
so in some domains, like mine, the satellite imagery, it is not so rare.

Le 12/09/2014 14:34, Thomas A Caswell a écrit :

I think this is an example of the problem reported in#2105
#2105

—
Reply to this email directly or view it on GitHub
#3451 (comment).

Nicolas PASCAL

HYGEOS - ICARE UMS 2877
Université Lille 1
Polytech'Lille - Bâtiment Eydoux - Bureau E207
Avenue Paul Langevin
59650 Villeneuve d'Ascq Cedex
France

Tel : +33 (0)3 20 33 59 77
Fax : +33 (0)3 20 41 75 47

as a side notematshow just calls `imshow with a specific set of kwargs.

OK, thanks, good to know. So the display should work also with matshow...

Le 12/09/2014 14:46, Thomas A Caswell a écrit :

as a side note |matshow| just calls `imshow with a specific set of kwargs.

—
Reply to this email directly or view it on GitHub
#3451 (comment).

Nicolas PASCAL

HYGEOS - ICARE UMS 2877
Université Lille 1
Polytech'Lille - Bâtiment Eydoux - Bureau E207
Avenue Paul Langevin
59650 Villeneuve d'Ascq Cedex
France

Tel : +33 (0)3 20 33 59 77
Fax : +33 (0)3 20 41 75 47

This probably needs a good deal of work to recover after the removal of cxx.

Yes -- sorry it has languished. It needs a revamp given recent changes. At a minimum, it needs to account for the fact that Agg uses fixed-point arithmetic for coordinates, so really only has a possible image size of 24 bits, not 32 (but that should still be more than enough for most things). See this comment from Agg:

    //----------------------------------------------------poly_subpixel_scale_e    // These constants determine the subpixel accuracy, to be more precise,     // the number of bits of the fractional part of the coordinates.     // The possible coordinate capacity in bits can be calculated by formula:    // sizeof(int) * 8 - poly_subpixel_shift, i.e, for 32-bit integers and    // 8-bits fractional part the capacity is 24 bits.

Adding my voice that this is a desired feature. There is no shortage of scientific data that exceeds 32768 pixels, if one dimension is narrow enough then it doesn't even have to be large in storage size. Primary concern with this for me is allowing users to save/export (as opposed to view, which could be at least be done via axis limits for true size or subsampling for smaller size unless you had an absurdly large monitor) large images.

Feel free to re-open if someone wants to take this up again, but I think super-sized images are a bit of a specialty niche....

I came across this PR rather indirectly in trying to understand why an unrelated agg-based piece of software uses scanline_p8 rather than scanline32_p8 and was somewhat surprised to see that matplotlib does it too. To me this seems rather strange since if agg represents integer coordinates with 24 bits, why would you want to use int16 for the scanline coordinates? It's not clear to me why agg even does this in the first place.

Obviously I don't know much about agg (or the internals of matplotlib), but I don't see much of a downside to to making this change. It'd need to be tested on a 32 bit machine of course, and it'd be better to use 1<<24 - 1 rather than INT_MAX, but other than that, why not?

I would be willing to attempt to create an updated PR and see if I can try it out on my 32-bit raspberry pi if there is some interest (and if people more knowledgeable than me don't think there's anything I'm missing). Since this limitation has never actually affected me in matplotlib, I'm not very eager to. But it seems straightforward enough.

I also am not an expert at the Agg layer (it pre-dates me and (mostly) Just Works).

My understanding is that The 16 vs 32 bit here is the size of the counter used to index a scanline which it turn is the maximum width of a line of the output image (in pixels). A 32kx32k 8bit RGBA image is ~4 gigs of ram which ~20 years ago was an huge amount of ram so part of the issue may be that when this was written, getting any bigger was impractical for other reasons.

Despite the rapid increase in RAM, there has not been a comparable rise in screen pixels. To show a 32kx32k pixel image with no down-sampling you would need at ~9x15 array of 4k monitors! While I am sure that such display systems do exist, I am going to posit they are still extremely rare (and probably use a whole lot of other custom systems to actually render to 135 4k monitors!). Thus, even though wecan render much higher pixel count images, I am not sure that it is practically useful. That said, I would be happy to be proven wrong.

I do not think we would close a revived version of this PR out of hand, I suspect it would be reviewed slowly due to both a shortage of expertise in Agg and not having a clear use-case for switching to the 32 bit code path (If it aint broke don't fix it!).

To@LevN0 question: if you have images that are bigger than 32k in one dimension, you probably want to control the down sampling better than what we do inimshow.

which in turn is the maximum width of a line of the output image

Right, but the important thing is that int16 is not just used for the width of the scanline, but also the x coordinate of where it starts. As a result, there's actually nothing in agg stopping you from producing very tall images, even if you use scanline_p8 (apart from matplotlib's image size limit), but if you try to produce a very wide image, then agg will fail to start any scanlines past x=2^15 (and I believe also fail to draw anything past 2^16).
So if my understanding is correct, the real size limits if scanline_p8 is used are (32768, 16777216), and (16777216, 16777216) for scanline32_p8.

I suspect that most use cases are, as LevN0 described, much longer along one dimension than they are in the other. This is also what I was trying to do that led to issues with the other software. Such images are not really intended to be viewed all at once, but are more something that you can scroll through (so monitor resolution doesn't really matter). That said, most such images would be probably be presented better by breaking them up into multiple segments anyway.

But really my interest in this PR is more just in trying to understand why agg's "default" scanline container effectively limits the width of images to be 512x smaller than the vertex coordinates are capable of representing, since that came as a bit of a surprise to me when I encountered it. This is certainly a rather irrelevant and off-topic discussion here that would be better brought up with whoever is currently maintaining agg, but matplotlib is probably one of the biggest users of agg. My understanding is that the scanlines are typically reasonably short, but can start anywhere. So 16 bits is probably sufficient for width, but not the x coordinate. I think the more sensible thing to do would be to store the x coordinate + "coverage" (opacity) together using the same sort of 24.8 structure used for the fixed-point coordinates, with 16 bits for the widths. That'd be 8 bits extra per span than currently used (or, with word alignment, no different on either a 32 bit or 64 bit machine), as opposed to 32 bits extra for scanline32_p8. But agg does not have such a scanline container, just scanline_p8 which is too small and scanline32_p8 which is bigger than necessary (though still not unreasonably big for its usage as I understand it).