Movatterモバイル変換

bedevere-appbot mentioned this pull request

python -m base64 do not handle stdin correct like ast.py or json.tool#138775

Closed

yihong0618 changed the title~~gh-138775: fix handlepython -m base64 stdin correct with EOF single.~~gh-138775: fix handlepython -m base64 stdin correct with EOF signal.

picnixz changed the title~~gh-138775: fix handlepython -m base64 stdin correct with EOF signal.~~gh-138775: fix handlepython -m base64 stdin correct with EOF signal

Copy link

Contributor

aisk commentedSep 11, 2025•
edited
Loading

I’m concerned that the current implementation reads the input in chunks, this allows the function to process the input as a stream via a pipe and avoids using too much memory with large inputs:

cpython/Lib/base64.py

Lines 531 to 537 in4978bfc

	defencode(input,output):
	"""Encode a file; input and output are binary files."""
	whiles:=input.read(MAXBINSIZE):
	whilelen(s)<MAXBINSIZEand (ns:=input.read(MAXBINSIZE-len(s))):
	s+=ns
	line=binascii.b2a_base64(s)
	output.write(line)

With this change, however, all input is read into memory, which breaks the chunked behavior.

Copy link

ContributorAuthor

yihong0618 commentedSep 11, 2025

I’m concerned that the current implementation reads the input in chunks, this allows the function to process the input as a stream via a pipe and avoids using too much memory with large inputs. :
cpython/Lib/base64.py
Lines 531 to 537 in4978bfc
defencode(input,output):
"""Encode a file; input and output are binary files."""
whiles:=input.read(MAXBINSIZE):
whilelen(s)<MAXBINSIZEand (ns:=input.read(MAXBINSIZE-len(s))):
s+=ns
line=binascii.b2a_base64(s)
output.write(line)
With this change, however, all input is read into memory, which breaks the chunked behavior.

Thanks for the comments

I will checked ifisatty to check
if
echo 1234 | ./python.exe -m base64
or
./python.exe -m base64
and do not break the old method

and in stdin old behavior is wrong, so I think is fine.
will check isatty if can check or not.

fix: flollow the comments when pipe stdin use buffer

22f8939

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

Copy link

ContributorAuthor

yihong0618 commentedSep 11, 2025

keep the old behavior and useisatty to check
as dochttps://docs.python.org/3/library/sys.html#sys.stdin

picnixz reviewed

Misc/NEWS.d/next/Library/2025-09-11-15-03-37.gh-issue-138775.w7rnSx.rst OutdatedShow resolvedHide resolved

Lib/base64.py OutdatedShow resolvedHide resolved

Copy link

ContributorAuthor

yihong0618 commentedSep 11, 2025

Thank you very much for the review again, I will try to learn how to express the change more clearly.
and do better next time, thanks again

Apply suggestions from code review

94c716d

Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>

Copy link

Member

picnixz commentedSep 11, 2025

I will try to learn how to express the change more clearly.
and do better next time, thanks again

Don't worry. Considering English isn't your first language, you can just leave formulation matters to others.

aisk approved these changes

bedevere-appbot added awaiting core review and removed awaiting review labels

ZeroIntensity reviewed

Sep 16, 2025

Misc/NEWS.d/next/Library/2025-09-11-15-03-37.gh-issue-138775.w7rnSx.rst OutdatedShow resolvedHide resolved

Lib/base64.py OutdatedShow resolvedHide resolved

fix: apply review comments in Lib/base64.py

8c9c499

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>

This comment was marked as resolved.

fix: address comments

8d6208b

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

picnixz mentioned this pull request

Oct 3, 2025

^D may printD upon exit inpython -m json#139541

Closed

Reword comment and NEWS entry.

0cbc440

gpshead approved these changes

bedevere-appbot added awaiting merge and removed awaiting core review labels

gpshead added needs backport to 3.13

bugs and security fixes

needs backport to 3.14bugs and security fixes labels

gpsheadenabled auto-merge (squash)

November 11, 2025 23:22

Copy link

Member

I went with this PR after playing around with the BufferedReader connected to a terminal - it really is difficult to tell when you have an EOF in that scenario vs what thebase64.encode() code wants to do when reading fixed sized blocks at a time in order to produce the fixed sized output lines. subsequent .read() calls after a short read or even after a 0 length read (the normal EOF indicator) still block and could later return more data (ugh, terminals!). So doing the isatty check does seem most practical here given that theio stack.

gpshead merged commitf5c2a41 intopython:main

54 checks passed

Copy link

miss-islington-appbot commentedNov 11, 2025

Thanks@yihong0618 for the PR, and@gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

bedevere-appbot removed the awaiting merge label

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

pythongh-138775: fix handlepython -m base64stdin correct with EOF…

4958344

… signal (pythonGH-138776)* fix: handle  stdin correct with EOF single.* fix: flollow the comments when pipe stdin use buffer* Apply suggestions from code review* fix: apply review comments in Lib/base64.py* fix: address comments* Reword comment and NEWS entry.---------(cherry picked from commit f5c2a41f9a6b3be95c5be9dbae0a4a3342d356dc)Co-authored-by: yihong <zouzou0208@gmail.com>Signed-off-by: yihong0618 <zouzou0208@gmail.com>Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>Co-authored-by: Peter Bierma <zintensitydev@gmail.com>Co-authored-by: Gregory P. Smith <greg@krypto.org>

Copy link

bedevere-appbot commentedNov 11, 2025

GH-141432 is a backport of this pull request to the3.14 branch.

bedevere-appbot removed the needs backport to 3.14bugs and security fixes label

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request

pythongh-138775: fix handlepython -m base64stdin correct with EOF…

c29c595

… signal (pythonGH-138776)* fix: handle  stdin correct with EOF single.* fix: flollow the comments when pipe stdin use buffer* Apply suggestions from code review* fix: apply review comments in Lib/base64.py* fix: address comments* Reword comment and NEWS entry.---------(cherry picked from commit f5c2a41f9a6b3be95c5be9dbae0a4a3342d356dc)Co-authored-by: yihong <zouzou0208@gmail.com>Signed-off-by: yihong0618 <zouzou0208@gmail.com>Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>Co-authored-by: Peter Bierma <zintensitydev@gmail.com>Co-authored-by: Gregory P. Smith <greg@krypto.org>

Copy link

bedevere-appbot commentedNov 11, 2025

GH-141433 is a backport of this pull request to the3.13 branch.

bedevere-appbot removed the needs backport to 3.13bugs and security fixes label