Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Type-1 font subsetting#20716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
jkseppan wants to merge3 commits intomatplotlib:main
base:main
Choose a base branch
Loading
fromjkseppan:type1-subset

Conversation

jkseppan
Copy link
Member

@jkseppanjkseppan commentedJul 22, 2021
edited
Loading

PR Summary

Type-1 subsetting

This reduces pdf file sizes when usetex is active, at the cost of
some complexity in the code. We implement a charstring bytecode
interpreter to keep track of subroutine calls in font programs.

Recommend merging to main to give people time to test this, not to
a 3.10 point release.

Give dviread.DviFont a fake filename attribute and a get_fontmap
method for character tracking.

Add type hints to the code this touches.

Closes#127.

PR Checklist

  • Has pytest style unit tests (andpytest passes).
  • IsFlake 8 compliant (runflake8 on changed files to check).
  • New features are documented, with examples if plot related.
  • Documentation is sphinx and numpydoc compliant (the docs shouldbuild without error).
  • Conforms to Matplotlib style conventions (installflake8-docstrings and runflake8 --docstring-convention=all).
  • New features have an entry indoc/users/next_whats_new/ (follow instructions in README.rst there).
  • API changes documented indoc/api/next_api_changes/ (follow instructions in README.rst there).

mpetroff reacted with hooray emoji
@anntzer
Copy link
Contributor

Is this ready for review, now that#20715 has been merged?

@jkseppan
Copy link
MemberAuthor

This is failing on Ubuntu 22.04 and Windows but passing on 24.04 and Mac. Here's one failing image (test_usetex_pdf.png, so converted from pdf to png on the test system).

test_usetex_pdf

This looks like the font is entirely broken. The expected image similarly converted looks like this:

test_usetex-expected_pdf

Strangely enough, the generated pdf file looks fine on my Mac.

@jkseppan
Copy link
MemberAuthor

I can repeat the error running on an Ubuntu 22.04 docker image:

GPL Ghostscript 9.55.0 (2021-09-27)Copyright (C) 2021 Artifex Software, Inc.  All rights reserved.This software is supplied under the GNU AGPLv3 and comes with NO WARRANTY:see the file COPYING for details.Processing pages 1 through 1.Page 1   **** Error: can't process embedded font stream,        attempting to load the font using its name.               Output may be incorrect.Querying operating system for font files...Substituting font Helvetica for MPLAAD+CMR17.Loading NimbusSans-Regular font from /usr/share/ghostscript/9.55.0/Resource/Font/NimbusSans-Regular... 4467044 2957798 6795968 5451925 4 done.   **** Error: can't process embedded font stream,        attempting to load the font using its name.               Output may be incorrect.Substituting font Helvetica for MPLAAC+CMR12.   **** Error: can't process embedded font stream,        attempting to load the font using its name.               Output may be incorrect.Substituting font Helvetica for MPLAAA+CMEX10.   **** Error: can't process embedded font stream,        attempting to load the font using its name.               Output may be incorrect.Substituting font Helvetica-Oblique for MPLAAB+CMMI12.Loading NimbusSans-Italic font from /usr/share/ghostscript/9.55.0/Resource/Font/NimbusSans-Italic... 4534308 3175939 7138400 5761600 4 done.

The subsetting is wrong in some way that breaks Ghostscript 9.55 but not the viewer in macOS or the newer Ghostscript in Ubuntu 24.04. (Ghostscript 9.56 has a completely rewritten PDF interpreter.)

@jkseppan
Copy link
MemberAuthor

I hope I found the culprit... I was writing an extra delimiter between the Subrs and the Charstrings when one was already there.

@jkseppanjkseppanforce-pushed thetype1-subset branch 2 times, most recently from9c5d971 to9a9dc05CompareMay 4, 2025 17:15
@jkseppanjkseppan marked this pull request as ready for reviewMay 4, 2025 17:16
@@ -35,10 +37,64 @@

frommatplotlib.cbookimport_format_approx
from .import_api
ifT.TYPE_CHECKING:
fromcollections.abcimportIterable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We tend not to have inline type hints (and I personally very much don't like them), but there are a few exceptions (e.g. _mathtext.py) so I guess it's up to you whether you want to leave them in.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Right – I saw that there are some files with type hints, and figured that the project might be in the process of adding them. I've found type hints pretty useful at work, but of course we should have a consistent style in the project. Has this been discussed in the past on the dev list or somewhere?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I'm probably among the most vocal opponents, but I'll defer to@QuLogic or@ksunden on this aspect.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I gave my opinion on our gitter, but I'll copy the main comment here for the sake of consolidation/potential for finding it in the future:

I would say type hints are a net positive in my opinion, though I acknowledge that there are problems (perhaps especially in a case like ours where APIs were designed well before type hints).

We went with stub files on initial implementation specifically to minimize some risk, specifically since the stub files have are not even loaded at runtime, there was no chance of them interfering.

However, I do think inline reduces a level ongoing maintenance risk, in particular the chance of the two files getting out of sync. (We have tooling/CI in place to help catch such things, but don't think it fully removes the risk)

The other factor is that stub files allow us draw the line at public APIs, and not have to worry about typing our internal logic. (Which has positives and negatives, positives being largely catching additional problems by type checker, negative largely being the surface area that needs to be covered.)

So in all, I think my personal recommendation would be a relatively slow uptake, where we keep the stub files for most public facing things in the interim, do inline type hints for internal logic when it makes sense to do so (e.g. when doing so is actively helpful for some refactor/feature addition/etc) then once the majority of internal logic has inline hints, consider inlining the hints of the more public facing APIs.

So looking at this example in particular, I think this does fall within my recommendation there. I am not personally opposed to it, but also not hard pulling for it. I do question a bit whether to advocate for "do minimal as you are working and motivated" or "the unit for adding type hints should be one file at a time". Doing the latter would carry the advantage that you get a better sense of how complete the hints are, but the disadvantage of asking people to type hint 3-4x what they were otherwise looking at (in this case, for example), which also impacts the review-ability of the PR as there are lots of changes that are actually orthogonal.

For what its worth, this particular file already has one function which got an inline typehint added in#27796

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

and@tacaswell responded

It is worth re-opening the discussion of in-line type hints
the world seems to have stabilized and in other projects I have had type hints catch actual bugs before I ran the code (but I have also had to spend 15 minutes trying to sort out how to placate the type checker for code that clearly works a couple of times)

The type hints were useful for me while working on this, since VS Code pointed out some obvious mistakes in real time. But I agree that it is not ideal to leave the file only partially annotated, since it does not pass a full type check in its current state.

I could remove the extra type hints from this PR and make a separate PR to add hints to the whole file?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Sure, let's just go for it. I'm not going to be able to resist this forever 🙄

Copy link
Contributor

@anntzeranntzer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

A few minor points to be considered, but overall this looks great.

postscript_stack:list[float],
opcode:int|str,
)->tuple[set,set,list[float],list[float]]:
"""Run one step in the charstring interpreter."""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I wonder if this may be clearer if you mutate buildchar_stack and postscript_stack in-place? this way you would write

glyphs=set();subrs=set()ifopcodein {...}:buildchar_stack[:]= []elifopcode=="seac":codes= ...;glyphs.update(...)buildchar_stack[:]= []elifopcode=="div":num2=buildchar_stack.pop()num1=buildchar_stack.pop()buildchar_stack.append(num1/num2)...returnglyphs,subrs

which feels perhaps more in the spirit of a postscript interpreter?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Could be! I'll think about this a little.

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I moved this into a separate class with the stacks and glyph/subr sets as members. I agree that it looks clearer that way.

Comment on lines 1 to 2
Type 1 fonts are now subsetted in PDF output
--------------------------------------------
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
Type 1 fonts are nowsubsetted in PDF output
--------------------------------------------
Type 1 fonts are nowsubset in PDF output
-----------------------------------------

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think I disagree here... the English verb "set" is irregular in that way, but if you search for "subsetted" in the context of fonts, it seems to be fairly common, including in fonttools and various Adobe forums. See also the accepted answer tothis question and possibly the discussion offlied out in Steven Pinker'sWords and Rules.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Maybe just be a bit more verbose?

Suggested change
Type 1 fonts are now subsetted in PDF output
--------------------------------------------
PDFs embed just the subset of Type 1 glyphs that are used
-----------------------------------------------------------

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I don't disagree that this is the correct conjugation in the past tense, rather that these sentences are not in the past tense. It is stating what is and in the (foreseeable) future shall occur.

jkseppan reacted with thumbs up emoji
Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I reworded the whole paragraph to be hopefully more understandable on its own.


When using the usetex feature with the PDF backend, Type 1 fonts are embedded
in the PDF output. These fonts used to be embedded in full, but they are now
subsetted to only include the glyphs that are actually used in the figure.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Suggested change
subsetted to only include the glyphs that are actually used in the figure.
subset to only include the glyphs that are actually used in the figure.

@jkseppan
Copy link
MemberAuthor

jkseppan commentedMay 11, 2025
edited
Loading

I was trying to improve test coverage and discovered some cases where this implementation breaks the output. Math examples seem to work fine, but there are some text fonts that use less common features that I'm clearly not handling right. Here are two cases:

@image_comparison(["subsetting-heuristica.pdf"])deftest_subsetting_heuristica():# Heuristica uses the callothersubr operator for some glyphsmpl.rcParams['text.latex.preamble']='\n'.join((r'\usepackage{heuristica}',r'\usepackage[T1]{fontenc}',r'\usepackage[utf8]{inputenc}'    ))fig,ax=plt.subplots()ax.text(0.1,0.1,r"BHTem",usetex=True,fontsize=50)ax.text(0.1,0.3,"fi",usetex=True,fontsize=50)ax.text(0.1,0.5,"ffl",usetex=True,fontsize=50)ax.set_xticks([])ax.set_yticks([])@image_comparison(["subsetting-dejavusans.pdf"])deftest_subsetting_dejavusans():# DejaVuSans uses the seac operator to compose characters with diacriticsmpl.rcParams['text.latex.preamble']='\n'.join((r'\usepackage{DejaVuSans}',r'\usepackage[T1]{fontenc}',r'\usepackage[utf8]{inputenc}'    ))fig,ax=plt.subplots()ax.text(0.1,0.1,r"\textsf{ñäö}",usetex=True,fontsize=50)ax.text(0.1,0.3,r"\textsf{fi}",usetex=True,fontsize=50)ax.text(0.1,0.5,r"\textsf{ffl}",usetex=True,fontsize=50)ax.set_xticks([])ax.set_yticks([])

The Heuristica callothersubr feature actually doesn't seem broken, but the fi and ffl ligatures are lost in both fonts.

imageimage

Both work without subsetting, although the metrics for the ligature glyphs seem to be wrong in the current code.

I'll see if I can figure out what's wrong.

@jkseppan
Copy link
MemberAuthor

The ligature problem is probably because we don't apply the encoding from TeX's font configuration to the font before subsetting. The custom encoding array is output in the PDF file but should also be used to map from character codes to glyph names. The seac issue might be a different encoding problem where we should do the lookups using Adobe Standard Encoding and not the font's own encoding.

@jkseppan
Copy link
MemberAuthor

It seems that some of the latest changes broke compatibility with older GhostScript again.

But while I debug that, a note about the new tests: they use font packages that are available on Debian or Ubuntu only by installing texlive-fonts-extra, which brings in a lot of other fonts too. Currently these tests get skipped on all runners, but would it make sense to install the extra fonts on just one of the runners to allow these tests to get run somewhere?

@jkseppanjkseppanforce-pushed thetype1-subset branch 2 times, most recently from4a8b6ff tocb204cdCompareMay 12, 2025 04:56
@jkseppan
Copy link
MemberAuthor

I added a test using Bitstream Charter, which is part of texlive-fonts-recommended, so we get at least some coverage of the full Type-1 subsetting code path.

I fixed the gs compatibility issue, which was about a broken Encoding object.

lenIV = self.prop.get('lenIV', 4)
encrypted = [
self._encrypt(charstrings[glyph], 'charstring', lenIV).decode('latin-1')
for glyph in glyphs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Should this (and _subset_subrs below) sort the glyphs (and subrs) to ensure reproducibility? (as set ordering changes over runs)

jkseppan reacted with thumbs up emoji
Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The subrs are already in order (the loop isfor i in range(n_subrs)) but sorting the glyphs is a good idea.

@jkseppan
Copy link
MemberAuthor

I removed the extra type annotations, which were incomplete in any case. I'll make a separate PR to annotate the entire file.

The fonts that get used are usually "Type 1" fonts.
They used to be embedded in full
but are now limited to the glyphs that are actually used in the figure.
This reduces the size of the resulting PDF files.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This reads well to me. Thanks!

@tacaswell
Copy link
Member

Will these new test be adjusted by#29816 ? If so we should sequence that one first.

@jkseppan
Copy link
MemberAuthor

Will these new test be adjusted by#29816 ? If so we should sequence that one first.

I don't think these depend on FreeType, since the usetex case uses TeX for layout and parses dvi files to determine the coordinates of glyphs.

@jkseppan
Copy link
MemberAuthor

The earlier CI error seems to have been caused by a newer mypy version detecting some more types in an unrelated file. The fix is also in PR#30119.

jkseppanand others added3 commitsMay 30, 2025 04:26
This reduces pdf file sizes when usetex is active, at the cost ofsome complexity in the code. We implement a charstring bytecodeinterpreter to keep track of subroutine calls in font programs.Give dviread.DviFont a fake filename attribute and a get_fontmapmethod for character tracking.In backend_pdf.py, refactor _get_subsetted_psname so it calls a method_get_subset_prefix, and reuse that to create tags for Type-1 fonts.Mark the methods static since they don't use anything from the instance.Recommend merging to main to give people time to test this, not toa 3.10 point release.Closesmatplotlib#127.Co-Authored-By: Elliott Sales de Andrade <quantum.analyst@gmail.com>
Co-authored-by: Elliott Sales de Andrade <quantum.analyst@gmail.com>
Mypy 1.16.0 flags errors here:lib/matplotlib/_mathtext.py:2531: error: "Node" has no attribute "width"  [attr-defined]lib/matplotlib/_mathtext.py:2608: error: List item 0 has incompatible type "Kern"; expected "Hlist | Vlist"  [list-item]The check for the attribute _metrics is equivalent to checking for aninstance of Char, since only Char and its subclasses set self._metrics.Mypy infers an unnecessarily tight type list[Hlist | Vlist] forspaced_nucleus so we give it a more general one.
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@QuLogicQuLogicQuLogic left review comments

@anntzeranntzeranntzer approved these changes

@jklymakjklymakjklymak left review comments

@ksundenksundenksunden left review comments

Assignees
No one assigned
Projects
None yet
Milestone
v3.11.0
Development

Successfully merging this pull request may close these issues.

When text.usetex=True with pdf backend, full subset of latex fonts is embedded into pdf file
7 participants
@jkseppan@anntzer@tacaswell@QuLogic@jklymak@ksunden@oscargus

[8]ページ先頭

©2009-2025 Movatter.jp