File Paths

When you access a file on an operating system, a file path is required. The file path is a string that represents the location of a file. It’s broken up into three major parts:

1. Folder Path: the file folder location on the file system where subsequent folders are separated by a forward slash/ (Unix) or backslash\ (Windows)
2. File Name: the actual name of the file
3. Extension: the end of the file path pre-pended with a period (.) used to indicate the file type

Here’s a quick example. Let’s say you have a file located within a file structure like this:

/│├── path/|   ││   ├── to/│   │   └── cats.gif│   ││   └── dog_breeds.txt|└── animals.csv

Let’s say you wanted to access thecats.gif file, and your current location was in the same folder aspath. In order to access the file, you need to go through thepath folder and then theto folder, finally arriving at thecats.gif file. The Folder Path ispath/to/. The File Name iscats. The File Extension is.gif. So the full path ispath/to/cats.gif.

Now let’s say that your current location or current working directory (cwd) is in theto folder of our example folder structure. Instead of referring to thecats.gif by the full path ofpath/to/cats.gif, the file can be simply referenced by the file name and extensioncats.gif.

/│├── path/|   │|   ├── to/  ← Your current working directory (cwd) is here|   │   └── cats.gif  ← Accessing this file|   │|   └── dog_breeds.txt|└── animals.csv

But what aboutdog_breeds.txt? How would you access that without using the full path? You can use the special characters double-dot (..) to move one directory up. This means that../dog_breeds.txt will reference thedog_breeds.txt file from the directory ofto:

/│├── path/  ← Referencing this parent folder|   │|   ├── to/  ← Current working directory (cwd)|   │   └── cats.gif|   │|   └── dog_breeds.txt  ← Accessing this file|└── animals.csv

The double-dot (..) can be chained together to traverse multiple directories above the current directory. For example, to accessanimals.csv from theto folder, you would use../../animals.csv.

Line Endings

One problem often encountered whenworking with file data is the representation of a new line or line ending. The line ending has its roots from back in the Morse Code era,when a specific pro-sign was used to communicate the end of a transmission or the end of a line.

Later, thiswas standardized for teleprinters by both the International Organization for Standardization (ISO) and the American Standards Association (ASA). ASA standard states that line endings should use the sequence of the Carriage Return (CR or\r)and the Line Feed (LF or\n) characters (CR+LF or\r\n). The ISO standard however allowed for either theCR+LF characters or just theLF character.

Windows uses theCR+LF characters to indicate a new line, while Unix and the newer Mac versions use just theLF character. This can cause some complications when you’re processing files on an operating system that is different than the file’s source. Here’s a quick example. Let’s say that we examine the filedog_breeds.txt that was created on a Windows system:

Pug\r\nJack Russell Terrier\r\nEnglish Springer Spaniel\r\nGerman Shepherd\r\nStaffordshire Bull Terrier\r\nCavalier King Charles Spaniel\r\nGolden Retriever\r\nWest Highland White Terrier\r\nBoxer\r\nBorder Terrier\r\n

This same output will be interpreted on a Unix device differently:

Pug\r\nJack Russell Terrier\r\nEnglish Springer Spaniel\r\nGerman Shepherd\r\nStaffordshire Bull Terrier\r\nCavalier King Charles Spaniel\r\nGolden Retriever\r\nWest Highland White Terrier\r\nBoxer\r\nBorder Terrier\r\n

This can make iterating over each line problematic, and you may need to account for situations like this.

Character Encodings

Another common problem that you may face is the encoding of the byte data. An encoding is a translation from byte data to human readable characters. This is typically done by assigning a numerical value to represent a character. The two most common encodings are theASCII andUNICODE Formats.ASCII can only store 128 characters, whileUnicode can contain up to 1,114,112 characters.

ASCII is actually a subset ofUnicode (UTF-8), meaning that ASCII and Unicode share the same numerical to character values. It’s important to note that parsing a file with the incorrect character encoding can lead to failures or misrepresentation of the character. For example, if a file was created using the UTF-8 encoding, and you try to parse it using the ASCII encoding, if there is a character that is outside of those 128 values, then an error will be thrown.

Text File Types

A text file is the most common file that you’ll encounter. Here are some examples of how these files are opened:

open('abc.txt')open('abc.txt','r')open('abc.txt','w')

With these types of files,open() will return aTextIOWrapper file object:

>>>file=open('dog_breeds.txt')>>>type(file)<class '_io.TextIOWrapper'>

This is the default file object returned byopen().

Buffered Binary File Types

A buffered binary file type is used for reading and writing binary files. Here are some examples of how these files are opened:

open('abc.txt','rb')open('abc.txt','wb')

With these types of files,open() will return either aBufferedReader orBufferedWriter file object:

>>>file=open('dog_breeds.txt','rb')>>>type(file)<class '_io.BufferedReader'>>>>file=open('dog_breeds.txt','wb')>>>type(file)<class '_io.BufferedWriter'>

Raw File Types

A raw file type is:

“generally used as a low-level building-block for binary and text streams.” (Source)

It is therefore not typically used.

Here’s an example of how these files are opened:

open('abc.txt','rb',buffering=0)

With these types of files,open() will return aFileIO file object:

>>>file=open('dog_breeds.txt','rb',buffering=0)>>>type(file)<class '_io.FileIO'>

Iterating Over Each Line in the File

A common thing to do while reading a file is to iterate over each line. Here’s an example of how to use the Python.readline() method to perform that iteration:

>>>withopen('dog_breeds.txt','r')asreader:>>># Read and print the entire file line by line>>>line=reader.readline()>>>whileline!='':# The EOF char is an empty string>>>print(line,end='')>>>line=reader.readline()PugJack Russell TerrierEnglish Springer SpanielGerman ShepherdStaffordshire Bull TerrierCavalier King Charles SpanielGolden RetrieverWest Highland White TerrierBoxerBorder Terrier

Another way you could iterate over each line in the file is to use the Python.readlines() method of the file object. Remember,.readlines() returns a list where each element in the list represents a line in the file:

>>>withopen('dog_breeds.txt','r')asreader:>>>forlineinreader.readlines():>>>print(line,end='')PugJack Russell TerrierEnglish Springer SpanielGerman ShepherdStaffordshire Bull TerrierCavalier King Charles SpanielGolden RetrieverWest Highland White TerrierBoxerBorder Terrier

However, the above examples can be further simplified by iterating over the file object itself:

>>>withopen('dog_breeds.txt','r')asreader:>>># Read and print the entire file line by line>>>forlineinreader:>>>print(line,end='')PugJack Russell TerrierEnglish Springer SpanielGerman ShepherdStaffordshire Bull TerrierCavalier King Charles SpanielGolden RetrieverWest Highland White TerrierBoxerBorder Terrier

This final approach is more Pythonic and can be quicker and more memory efficient. Therefore, it is suggested you use this instead.

Now let’s dive into writing files. As with reading files, file objects have multiple methods that are useful for writing to a file:

Here’s a quick example of using.write() and.writelines():

withopen('dog_breeds.txt','r')asreader:# Note: readlines doesn't trim the line endingsdog_breeds=reader.readlines()withopen('dog_breeds_reversed.txt','w')aswriter:# Alternatively you could use# writer.writelines(reversed(dog_breeds))# Write the dog breeds to the file in reversed orderforbreedinreversed(dog_breeds):writer.write(breed)

Working With Bytes

Sometimes, you may need to work with files usingbyte strings. This is done by adding the'b' character to themode argument. All of the same methods for the file object apply. However, each of the methods expect and return abytes object instead:

>>>withopen('dog_breeds.txt','rb')asreader:>>>print(reader.readline())b'Pug\n'

Opening a text file using theb flag isn’t that interesting. Let’s say we have this cute picture of a Jack Russell Terrier (jack_russell.png):

You can actually open that file in Python and examine the contents! Since the.png file format is well defined, the header of the file is 8 bytes broken up like this:

Sure enough, when you open the file and read these bytes individually, you can see that this is indeed a.png header file:

>>>withopen('jack_russell.png','rb')asbyte_reader:>>>print(byte_reader.read(1))>>>print(byte_reader.read(3))>>>print(byte_reader.read(2))>>>print(byte_reader.read(1))>>>print(byte_reader.read(1))b'\x89'b'PNG'b'\r\n'b'\x1a'b'\n'

A Full Example:dos2unix.py

Let’s bring this whole thing home and look at a full example of how to read and write to a file. The following is ados2unix like tool that will convert a file that contains line endings of\r\n to\n.

This tool is broken up into three major sections. The first isstr2unix(), which converts a string from\r\n line endings to\n. The second isdos2unix(), which converts a string that contains\r\n characters into\n.dos2unix() callsstr2unix() internally. Finally, there’s the__main__ block, which is called only when the file is executed as a script. Think of it as themain function found in other programming languages.

"""A simple script and library to convert files or strings from dos likeline endings with Unix like line endings."""importargparseimportosdefstr2unix(input_str:str)->str:r"""    Converts the string from \r\n line endings to \n    Parameters    ----------    input_str        The string whose line endings will be converted    Returns    -------        The converted string    """r_str=input_str.replace('\r\n','\n')returnr_strdefdos2unix(source_file:str,dest_file:str):"""    Converts a file that contains Dos like line endings into Unix like    Parameters    ----------    source_file        The path to the source file to be converted    dest_file        The path to the converted file for output    """# NOTE: Could add file existence checking and file overwriting# protectionwithopen(source_file,'r')asreader:dos_content=reader.read()unix_content=str2unix(dos_content)withopen(dest_file,'w')aswriter:writer.write(unix_content)if__name__=="__main__":# Create our Argument parser and set its descriptionparser=argparse.ArgumentParser(description="Script that converts a DOS like file to an Unix like file",)# Add the arguments:#   - source_file: the source file we want to convert#   - dest_file: the destination where the output should go# Note: the use of the argument type of argparse.FileType could# streamline some thingsparser.add_argument('source_file',help='The location of the source ')parser.add_argument('--dest_file',help='Location of dest file (default: source_file appended with `_unix`',default=None)# Parse the args (argparse automatically grabs the values from# sys.argv)args=parser.parse_args()s_file=args.source_filed_file=args.dest_file# If the destination file wasn't passed, then assume we want to# create a new file based on the old oneifd_fileisNone:file_path,file_extension=os.path.splitext(s_file)d_file=f'{file_path}_unix{file_extension}'dos2unix(s_file,d_file)

__file__

The__file__ attribute is aspecial attribute of modules, similar to__name__. It is:

“the pathname of the file from which the module was loaded, if it was loaded from a file.” (Source

Here’s a real world example. In one of my past jobs, I did multiple tests for a hardware device. Each test was written using a Python script with the test script file name used as a title. These scripts would then be executed and could print their status using the__file__ special attribute. Here’s an example folder structure:

project/|├── tests/|   ├── test_commanding.py|   ├── test_power.py|   ├── test_wireHousing.py|   └── test_leds.py|└── main.py

Runningmain.py produces the following:

>>>pythonmain.pytests/test_commanding.pyStarted:tests/test_commanding.pyPassed!tests/test_power.pyStarted:tests/test_power.pyPassed!tests/test_wireHousing.pyStarted:tests/test_wireHousing.pyFailed!tests/test_leds.pyStarted:tests/test_leds.pyPassed!

I was able to run and get the status of all my tests dynamically through use of the__file__ special attribute.

Appending to a File

Sometimes, you may want to append to a file or start writing at the end of an already populated file. This is easily done by using the'a' character for themode argument:

withopen('dog_breeds.txt','a')asa_writer:a_writer.write('\nBeagle')

When you examinedog_breeds.txt again, you’ll see that the beginning of the file is unchanged andBeagle is now added to the end of the file:

>>>withopen('dog_breeds.txt','r')asreader:>>>print(reader.read())PugJack Russell TerrierEnglish Springer SpanielGerman ShepherdStaffordshire Bull TerrierCavalier King Charles SpanielGolden RetrieverWest Highland White TerrierBoxerBorder TerrierBeagle

Working With Two Files at the Same Time

There are times when you may want to read a file and write to another file at the same time. If you use the example that was shown when you were learning how to write to a file, it can actually be combined into the following:

d_path='dog_breeds.txt'd_r_path='dog_breeds_reversed.txt'withopen(d_path,'r')asreader,open(d_r_path,'w')aswriter:dog_breeds=reader.readlines()writer.writelines(reversed(dog_breeds))

Creating Your Own Context Manager

There may come a time when you’ll need finer control of the file object by placing it inside a custom class. When you do this, using thewith statement can no longer be used unless you add a few magic methods:__enter__ and__exit__. By adding these, you’ll have created what’s called acontext manager.

__enter__() is invoked when calling thewith statement.__exit__() is called upon exiting from thewith statement block.

Here’s a template that you can use to make your custom class:

classmy_file_reader():def__init__(self,file_path):self.__path=file_pathself.__file_object=Nonedef__enter__(self):self.__file_object=open(self.__path)returnselfdef__exit__(self,type,val,tb):self.__file_object.close()# Additional methods implemented below

Now that you’ve got your custom class that is now a context manager, you can use it similarly to theopen() built-in:

withmy_file_reader('dog_breeds.txt')asreader:# Perform custom class operationspass

Here’s a good example. Remember the cute Jack Russell image we had? Perhaps you want to open other.png files but don’t want to parse the header file each time. Here’s an example of how to do this. This example also uses custom iterators. If you’re not familiar with them, check outPython Iterators:

classPngReader():# Every .png file contains this in the header.  Use it to verify# the file is indeed a .png._expected_magic=b'\x89PNG\r\n\x1a\n'def__init__(self,file_path):# Ensure the file has the right extensionifnotfile_path.endswith('.png'):raiseNameError("File must be a '.png' extension")self.__path=file_pathself.__file_object=Nonedef__enter__(self):self.__file_object=open(self.__path,'rb')magic=self.__file_object.read(8)ifmagic!=self._expected_magic:raiseTypeError("The File is not a properly formatted .png file!")returnselfdef__exit__(self,type,val,tb):self.__file_object.close()def__iter__(self):# This and __next__() are used to create a custom iterator# See https://dbader.org/blog/python-iteratorsreturnselfdef__next__(self):# Read the file in "Chunks"# See https://en.wikipedia.org/wiki/Portable_Network_Graphics#%22Chunks%22_within_the_fileinitial_data=self.__file_object.read(4)# The file hasn't been opened or reached EOF.  This means we# can't go any further so stop the iteration by raising the# StopIteration.ifself.__file_objectisNoneorinitial_data==b'':raiseStopIterationelse:# Each chunk has a len, type, data (based on len) and crc# Grab these values and return them as a tuplechunk_len=int.from_bytes(initial_data,byteorder='big')chunk_type=self.__file_object.read(4)chunk_data=self.__file_object.read(chunk_len)chunk_crc=self.__file_object.read(4)returnchunk_len,chunk_type,chunk_data,chunk_crc

You can now open.png files and properly parse them using your custom context manager:

>>>withPngReader('jack_russell.png')asreader:>>>forl,t,d,cinreader:>>>print(f"{l:05},{t},{c}")00013, b'IHDR', b'v\x121k'00001, b'sRGB', b'\xae\xce\x1c\xe9'00009, b'pHYs', b'(<]\x19'00345, b'iTXt', b"L\xc2'Y"16384, b'IDAT', b'i\x99\x0c('16384, b'IDAT', b'\xb3\xfa\x9a$'16384, b'IDAT', b'\xff\xbf\xd1\n'16384, b'IDAT', b'\xc3\x9c\xb1}'16384, b'IDAT', b'\xe3\x02\xba\x91'16384, b'IDAT', b'\xa0\xa99='16384, b'IDAT', b'\xf4\x8b.\x92'16384, b'IDAT', b'\x17i\xfc\xde'16384, b'IDAT', b'\x8fb\x0e\xe4'16384, b'IDAT', b')3={'01040, b'IDAT', b'\xd6\xb8\xc1\x9f'00000, b'IEND', b'\xaeB`\x82'