Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork32k
gh-79516: allow msgfmt.py to compile multiple input po files#10875
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Open
s-ball wants to merge68 commits intopython:mainChoose a base branch
base:main
Could not load branches
Branch not found:{{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated.
+276 −53
Open
Changes from1 commit
Commits
Show all changes
68 commits Select commitHold shift + click to select a range
73b5ac8 Add test for current behaviour of msgfmt.py
s-ball5fb1575 Final fix for bpo-9741, with tests proving it.
s-ballb1968e9 Update docstrings for the script and make function
s-ball0bc4ad3 Give make a simpler and more consistent interface.
s-balla9e67b4 Add a Misc/NEWS.d entry.
s-ball6c59d6c Merge branch 'multi_inputs' of https://github.com/s-ball/cpython into…
s-ball1ce22c0 Merge branch 'main' into multi_inputs
s-ball863bd97 Merge branch 'main' into multi_inputs
s-ball4390ede Fix an import error in test_i18n.py .
s-ball008ea27 Fix another import error.
s-ball8744743 Merge branch 'main' into multi_inputs
s-ball93a6eb7 Merge branch 'main' into multi_inputs
s-ballba26b80 Revert version number to 1.2
s-ball150cea1 Merge branch 'main' into multi_inputs
s-balld9afabb Merge branch 'main' into multi_inputs
s-ball9004387 Merge branch 'main' into multi_inputs
s-ball7556f79 Merge branch 'multi_inputs' of https://github.com/s-ball/cpython into…
s-ball80947d1 Merge branch 'main' into multi_inputs
s-ball12acb83 Merge branch 'main' into multi_inputs
s-ball1ecc1f3 Fix a merge error
s-balle59ba68 Merge branch 'multi_inputs' of https://github.com/s-ball/cpython into…
s-ball4ffd20a Merge branch 'main' into multi_inputs
s-ball7505f2b Merge branch 'main' into multi_inputs
s-ball2c27120 Merge branch 'main' into multi_inputs
s-ball1f4e5ac fix details
merwok4170796 Merge branch 'main' into multi_inputs
s-ball46c08c5 Move tests for the gh-79516 issue to test_msgfmt.py
s-ball24d89a6 Cosmetic improvements after review.
s-ball17b4e05 Fix an import error
s-ballbfc8a44 Merge branch 'main' into multi_inputs
s-ball106dd40 Apply suggestions from code review
s-ball9cb9395 Cosmetic improvements after review.
s-ball08bc8d7 In test_msgfmt move data files to the data folder.
s-ball9d992cd Remove duplicate tests from test_msgfmt.
s-ball31fd434 Merge branch 'main' into multi_inputs
s-ball916aec7 Merge branch 'main' into multi_inputs
s-ball51fcf09 Rename data files for test_msgfmt.Test_multi_input
s-balld51ad50 Apply suggestions from code review
s-ball677f720 Cosmetic improvements after review.
s-ballb4ea80a compile_messages now accepts several input files
s-ball3120add whitespace nit
merwokd642923 Merge branch 'main' into multi_inputs
s-ball9d91f12 Make explicit how mo files can be re-generated.
s-ball4d83cb7 Merge branch 'multi_inputs' of https://github.com/s-ball/cpython into…
s-ball421272b Apply suggestions from code review
s-ball09b97d9 Update Lib/test/test_tools/test_msgfmt.py
s-ball12cae51 Merge branch 'main' into multi_inputs
s-ball3760851 Merge branch 'main' into multi_inputs
s-balld45039c Generate json files for MultiInputTest data files.
s-balldde5ef1 Apply suggestions from code review
s-ballbb6e0c5 Cosmetic improvements (long lines...)
s-ball797990a Merge branch 'multi_inputs' of https://github.com/s-ball/cpython into…
s-ballc95af16 Merge branch 'main' into multi_inputs
s-ball905ec70 Revert an unwanted change.
s-ballb7a7c48 Simplifies the general logic of msgfmt.py
s-ball1b3b73e Update Tools/i18n/msgfmt.py
s-ball1db2c66 Changes per review.
s-ball3a6e1ef Merge branch 'multi_inputs' of https://github.com/s-ball/cpython into…
s-ball743bdc5 Removes the now unused get_names function.
s-ball50e7145 Tests for extension corner cases for msgfmt.py
s-ballcaa8955 Merge branch 'main' into multi_inputs
s-balla4f1769 Fix a parameter inversion in test_msgfmt.py
s-ball11f6e69 Merge branch 'main' into multi_inputs
s-ball0ed3107 Merge branch 'main' into multi_inputs
s-balld1e0a26 Merge branch 'main' into multi_inputs
s-ball213afcb Merge branch 'main' into multi_inputs
s-ball4d54e50 Merge main into multi_inputs
s-ballab97edd Merge branch 'main' into multi_inputs
s-ballFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
NextNext commit
Add test for current behaviour of msgfmt.py
Test option processing, and conversion of one single file with orwithout the -o option.Also test the little documented behaviour of merging two input fileswith -o option.
- Loading branch information
Uh oh!
There was an error while loading.Please reload this page.
commit73b5ac88202e6560b01cf37c82f03d51996746a8
There are no files selected for viewing
185 changes: 183 additions & 2 deletionsLib/test/test_tools/test_i18n.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -5,9 +5,9 @@ | ||
| import unittest | ||
| from textwrap import dedent | ||
| from test.support.script_helper import assert_python_ok, assert_python_failure | ||
| from test.test_tools import skip_if_missing, toolsdir | ||
| from test.support import temp_cwd, temp_dir, unload, change_cwd | ||
| skip_if_missing() | ||
| @@ -243,3 +243,184 @@ def test_files_list(self): | ||
| self.assertIn(f'msgid "{text1}"', data) | ||
| self.assertIn(f'msgid "{text2}"', data) | ||
| self.assertNotIn(text3, data) | ||
| class Test_msgfmt(unittest.TestCase): | ||
| """Tests for the msgfmt.py tool | ||
| bpo-35335 - bpo-9741 | ||
merwok marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| """ | ||
| script = os.path.join(toolsdir,'i18n', 'msgfmt.py') | ||
merwok marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| # binary images of tiny po files | ||
| # windows end of lines for first one | ||
| file1_fr_po = b'''# French translations for python package.\r | ||
| # Copyright (C) 2018 THE python\'S COPYRIGHT HOLDER\r | ||
| # This file is distributed under the same license as the python package.\r | ||
| # s-ball <s-ball@laposte.net>, 2018.\r | ||
| #\r | ||
| msgid ""\r | ||
| msgstr ""\r | ||
| "Project-Id-Version: python 3.8\\n"\r | ||
| "Report-Msgid-Bugs-To: \\n"\r | ||
| "POT-Creation-Date: 2018-11-30 23:46+0100\\n"\r | ||
| "PO-Revision-Date: 2018-11-30 23:47+0100\\n"\r | ||
| "Last-Translator: s-ball <s-ball@laposte.net>\\n"\r | ||
| "Language-Team: French\\n"\r\n"Language: fr\\n"\r | ||
| "MIME-Version: 1.0\\n"\r | ||
| "Content-Type: text/plain; charset=UTF-8\\n"\r | ||
| "Content-Transfer-Encoding: 8bit\\n"\r | ||
| "Plural-Forms: nplurals=2; plural=(n > 1);\\n"\r | ||
| \r | ||
| #: file1.py:6\r | ||
| msgid "Hello!"\r | ||
| msgstr "Bonjour !"\r | ||
| \r | ||
| #: file1.py:7\r | ||
| #, python-brace-format\r | ||
| msgid "{n} horse"\r | ||
| msgid_plural "{n} horses"\r | ||
| msgstr[0] "{n} cheval"\r | ||
| msgstr[1] "{n} chevaux"\r | ||
| ''' | ||
| # Unix end of file and a non ascii character for second one | ||
| file2_fr_po = rb'''# French translations for python package. | ||
| # Copyright (C) 2018 THE python'S COPYRIGHT HOLDER | ||
| # This file is distributed under the same license as the python package. | ||
| # s-ball <s-ball@laposte.net>, 2018. | ||
| # | ||
| msgid "" | ||
| msgstr "" | ||
| "Project-Id-Version: python 3.8\n" | ||
| "Report-Msgid-Bugs-To: \n" | ||
| "POT-Creation-Date: 2018-11-30 23:57+0100\n" | ||
| "PO-Revision-Date: 2018-11-30 23:57+0100\n" | ||
| "Last-Translator: s-ball <s-ball@laposte.net>\n" | ||
| "Language-Team: French\n" | ||
| "Language: fr\n" | ||
| "MIME-Version: 1.0\n" | ||
| "Content-Type: text/plain; charset=UTF-8\n" | ||
| "Content-Transfer-Encoding: 8bit\n" | ||
| "Plural-Forms: nplurals=2; plural=(n > 1);\n" | ||
| #: file2.py:6 | ||
| msgid "It's over." | ||
| msgstr "C'est termin\xc3\xa9." | ||
| #: file2.py:7 | ||
| msgid "Bye..." | ||
| msgstr "Au revoir ..." | ||
| ''' | ||
| # binary images of corresponding compiled mo files | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. images? | ||
| file1_fr_mo = b'\xde\x12\x04\x95\x00\x00\x00\x00\x03\x00\x00\x00\x1c' \ | ||
| b'\x00\x00\x004\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. Not very important: escaping newlines with backslashes is not recommended, better use parens. This also gives more room per line: file1_fr_mo= (b'bytes bytes bytes bytes etc to 80 columns') | ||
| b'\x00\x00\x00\x00\x00L\x00\x00\x00\x06\x00\x00\x00M' \ | ||
| b'\x00\x00\x00\x14\x00\x00\x00T\x00\x00\x00[\x01\x00' \ | ||
| b'\x00i\x00\x00\x00\t\x00\x00\x00\xc5\x01\x00\x00\x16' \ | ||
| b'\x00\x00\x00\xcf\x01\x00\x00\x00Hello!\x00{n} horse' \ | ||
| b'\x00{n} horses\x00Project-Id-Version: python 3.8\n' \ | ||
| b'Report-Msgid-Bugs-To: \nPOT-Creation-Date: 2018-11-30 ' \ | ||
| b'23:46+0100\nPO-Revision-Date: 2018-11-30 23:47+0100\n'\ | ||
| b'Last-Translator: s-ball <s-ball@laposte.net>\n' \ | ||
| b'Language-Team: French\nLanguage: fr\nMIME-Version: 1.0' \ | ||
| b'\nContent-Type: text/plain; charset=UTF-8\nContent-' \ | ||
| b'Transfer-Encoding: 8bit\nPlural-Forms: nplurals=2' \ | ||
| b'; plural=(n > 1);\n\x00Bonjour !\x00{n} cheval\x00{n' \ | ||
| b'} chevaux\x00' | ||
| file2_fr_mo = b"\xde\x12\x04\x95\x00\x00\x00\x00\x03\x00\x00\x00" \ | ||
| b"\x1c\x00\x00\x004\x00\x00\x00\x00\x00\x00\x00\x00" \ | ||
| b"\x00\x00\x00\x00\x00\x00\x00L\x00\x00\x00\x06\x00" \ | ||
| b"\x00\x00M\x00\x00\x00\n\x00\x00\x00T\x00\x00\x00[" \ | ||
| b"\x01\x00\x00_\x00\x00\x00\r\x00\x00\x00\xbb\x01" \ | ||
| b"\x00\x00\x0f\x00\x00\x00\xc9\x01\x00\x00\x00Bye.." \ | ||
| b".\x00It's over.\x00Project-Id-Version: python 3.8" \ | ||
| b"\nReport-Msgid-Bugs-To: \nPOT-Creation-Date: 2018" \ | ||
| b"-11-30 23:57+0100\nPO-Revision-Date: 2018-11-30 2" \ | ||
| b"3:57+0100\nLast-Translator: s-ball <s-ball@lapost" \ | ||
| b"e.net>\nLanguage-Team: French\nLanguage: fr\nMIME" \ | ||
| b"-Version: 1.0\nContent-Type: text/plain; charset=" \ | ||
| b"UTF-8\nContent-Transfer-Encoding: 8bit\nPlural-Fo" \ | ||
| b"rms: nplurals=2; plural=(n > 1);\n\x00Au revoir ." \ | ||
| b"..\x00C'est termin\xc3\xa9.\x00" | ||
| # image of merging both po files keeping second header | ||
| file12_fr_mo = b"\xde\x12\x04\x95\x00\x00\x00\x00\x05\x00\x00\x00" \ | ||
| b"\x1c\x00\x00\x00D\x00\x00\x00\x00\x00\x00\x00\x00" \ | ||
| b"\x00\x00\x00\x00\x00\x00\x00l\x00\x00\x00\x06\x00" \ | ||
| b"\x00\x00m\x00\x00\x00\x06\x00\x00\x00t\x00\x00\x00" \ | ||
| b"\n\x00\x00\x00{\x00\x00\x00\x14\x00\x00\x00\x86" \ | ||
| b"\x00\x00\x00[\x01\x00\x00\x9b\x00\x00\x00\r\x00" \ | ||
| b"\x00\x00\xf7\x01\x00\x00\t\x00\x00\x00\x05\x02\x00" \ | ||
| b"\x00\x0f\x00\x00\x00\x0f\x02\x00\x00\x16\x00\x00" \ | ||
| b"\x00\x1f\x02\x00\x00\x00Bye...\x00Hello!\x00It's " \ | ||
| b"over.\x00{n} horse\x00{n} horses\x00Project-Id-Ver" \ | ||
| b"sion: python 3.8\nReport-Msgid-Bugs-To: \nPOT-Crea" \ | ||
| b"tion-Date: 2018-11-30 23:57+0100\nPO-Revision-Date" \ | ||
| b": 2018-11-30 23:57+0100\nLast-Translator: s-ball <" \ | ||
| b"s-ball@laposte.net>\nLanguage-Team: French\nLangua" \ | ||
| b"ge: fr\nMIME-Version: 1.0\nContent-Type: text/plai" \ | ||
| b"n; charset=UTF-8\nContent-Transfer-Encoding: 8bit" \ | ||
| b"\nPlural-Forms: nplurals=2; plural=(n > 1);\n\x00A" \ | ||
| b"u revoir ...\x00Bonjour !\x00C'est termin\xc3\xa9." \ | ||
| b"\x00{n} cheval\x00{n} chevaux\x00" | ||
| def imp(self): | ||
| i18ndir = os.path.join(toolsdir, 'i18n') | ||
| sys.path.append(i18ndir) | ||
| import msgfmt | ||
| sys.path.remove(i18ndir) | ||
| return msgfmt | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. Is it really necessary to import the script as a module? I thought it was only a command-line tool. | ||
| def test_help(self): | ||
| """Test option -h""" | ||
| rc, stdout, stderr = assert_python_ok(self.script, '-h') | ||
| self.assertEqual(0, rc) | ||
| self.assertTrue(stderr.startswith( | ||
| b'Generate binary message catalog from textual' | ||
| b' translation description.' | ||
| ) | ||
| ) | ||
merwok marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| def test_wrong(self): | ||
| """Test wrong option""" | ||
| rc, stdout, stderr = assert_python_failure(self.script, '-x') | ||
| self.assertEqual(1, rc) | ||
| self.assertTrue(stderr.startswith( | ||
| b'Generate binary message catalog from textual' | ||
| b' translation description.' | ||
| ) | ||
| ) | ||
merwok marked this conversation as resolved. Show resolvedHide resolvedUh oh!There was an error while loading.Please reload this page. | ||
| def test_outputfile(self): | ||
| """Test script with -o option - 1 single file, Windows EOL""" | ||
| with temp_cwd(None): | ||
| with open("file1_fr.po", "wb") as out: | ||
| out.write(self.file1_fr_po) | ||
| assert_python_ok(self.script, '-o', 'file1.mo', 'file1_fr.po') | ||
| with open('file1.mo', 'rb') as fin: | ||
| self.assertEqual(self.file1_fr_mo, fin.read()) | ||
| def test_no_outputfile(self): | ||
| """Test script without -o option - 1 single file, Unix EOL""" | ||
| with temp_cwd(None): | ||
| with open("file2_fr.po", "wb") as out: | ||
| out.write(self.file2_fr_po) | ||
| assert_python_ok(self.script, 'file2_fr.po') | ||
| with open('file2_fr.mo', 'rb') as fin: | ||
| self.assertEqual(self.file2_fr_mo, fin.read()) | ||
| def test_both_with_outputfile(self): | ||
| """Test script with -o option and 2 input files""" | ||
| # msgfmt.py version 1.2 behaviour is to correctly merge the input | ||
| # files and to keep last entry when same entry occurs in more than | ||
| # one file | ||
| with temp_cwd(None): | ||
| with open("file1_fr.po", "wb") as out: | ||
| out.write(self.file1_fr_po) | ||
| with open("file2_fr.po", "wb") as out: | ||
| out.write(self.file2_fr_po) | ||
| assert_python_ok(self.script, '-o', 'file1.mo', | ||
| 'file1_fr.po', 'file2_fr.po') | ||
| with open('file1.mo', 'rb') as fin: | ||
| self.assertEqual(self.file12_fr_mo, fin.read()) | ||
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.