Image (C) Tai Kedzierski

We just had a use case where we needed to POST a file over to a server. The naive implementation for posting withrequests is to do

withopen("my_file.bin",'rb')asfh:requests.post(url,data={"bytes":fh.read()})

Job done! Well. If the file isreeeally big, that.read() operation will attempt to load the entire file into memory,before passing the loaded bytes torequests.post(...)

Clearly, this is going to hurt. A lot.

Usemmap

A quick searchyielded a solution usingmmap to create a "memory mapped" object, which would behave like a string, whilst being backed by a file that only gets read in chunks as needed.

As ever, I like making things re-usable, and easy to slot-in. I adapted the example into a contextual object that can be used in-place of a normal call toopen()

# It's a tiny snippet, but go on.# Delviered to You under MIT Expat License, aka "Do what you want"# I'm not even fussy about attribution.importmmapclassStringyFileReader:def__init__(self,file_name,mode):ifmodenotin("r","rb"):raiseValueError(f"Invalid mode'{mode}'. Only read-modes are supported")self._fh=open(file_name,mode)# A file size of 0 means "whatever the size of the file actually is" on non-Windows# On Windows, you'll need to obtain the actual size, thoughfsize=0self._mmap=mmap.mmap(self._fh.fileno(),fsize,access=mmap.ACCESS_READ)def__enter__(self):returnselfdefread(self):returnself._mmapdef__exit__(self,*args):self._mmap.close()self._fh.close()

Which then lets us simply tweak the original naive example to:

withStringyFileReader("my_file.bin",'rb')asfh:requests.post(url,data={"bytes":fh.read()})

Job. Done.

EDIT: we've discovered through further use thatrequests is pretty stupid. It sill tries to readthe entire file into memory - possibly by doing a copy of the "string" it receives during one of its internal operations. So this solution seems to only stand in limited cases...