How to load documents from a directory
LangChain'sDirectoryLoader implements functionality for reading files from disk into LangChainDocument objects. Here we demonstrate:
- How to load from a filesystem, including use of wildcard patterns;
- How to use multithreading for file I/O;
- How to use custom loader classes to parse specific file types (e.g., code);
- How to handle errors, such as those due to decoding.
from langchain_community.document_loadersimport DirectoryLoader
DirectoryLoader accepts aloader_cls kwarg, which defaults toUnstructuredLoader.Unstructured supports parsing for a number of formats, such as PDF and HTML. Here we use it to read in a markdown (.md) file.
We can use theglob parameter to control which files to load. Note that here it doesn't load the.rst file or the.html files.
loader= DirectoryLoader("../", glob="**/*.md")
docs= loader.load()
len(docs)
20
print(docs[0].page_content[:100])
Security
LangChain has a large ecosystem of integrations with various external resources like local
Show a progress bar
By default a progress bar will not be shown. To show a progress bar, install thetqdm library (e.g.pip install tqdm), and set theshow_progress parameter toTrue.
loader= DirectoryLoader("../", glob="**/*.md", show_progress=True)
docs= loader.load()
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 54.56it/s]
Use multithreading
By default the loading happens in one thread. In order to utilize several threads set theuse_multithreading flag to true.
loader= DirectoryLoader("../", glob="**/*.md", use_multithreading=True)
docs= loader.load()
Change loader class
By default this uses theUnstructuredLoader class. To customize the loader, specify the loader class in theloader_cls kwarg. Below we show an example usingTextLoader:
from langchain_community.document_loadersimport TextLoader
loader= DirectoryLoader("../", glob="**/*.md", loader_cls=TextLoader)
docs= loader.load()
print(docs[0].page_content[:100])
# Security
LangChain has a large ecosystem of integrations with various external resources like loc
Notice that while theUnstructuredLoader parses Markdown headers,TextLoader does not.
If you need to load Python source code files, use thePythonLoader:
from langchain_community.document_loadersimport PythonLoader
loader= DirectoryLoader("../../../../../", glob="**/*.py", loader_cls=PythonLoader)
Auto-detect file encodings with TextLoader
DirectoryLoader can help manage errors due to variations in file encodings. Below we will attempt to load in a collection of files, one of which includes non-UTF8 encodings.
path="../../../libs/langchain/tests/unit_tests/examples/"
loader= DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader)
A. Default Behavior
By default we raise an error:
loader.load()
Error loading file ../../../../libs/langchain/tests/unit_tests/examples/example-non-utf8.txt
---------------------------------------------------------------------------
``````output
UnicodeDecodeError Traceback (most recent call last)
``````output
File ~/repos/langchain/libs/community/langchain_community/document_loaders/text.py:43, in TextLoader.lazy_load(self)
42 with open(self.file_path, encoding=self.encoding) as f:
---> 43 text = f.read()
44 except UnicodeDecodeError as e:
``````output
File ~/.pyenv/versions/3.10.4/lib/python3.10/codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
321 data = self.buffer + input
--> 322 (result, consumed) = self._buffer_decode(data, self.errors, final)
323 # keep undecoded input until the next call
``````output
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 0: invalid continuation byte
``````output
The above exception was the direct cause of the following exception:
``````output
RuntimeError Traceback (most recent call last)
``````output
Cell In[10], line 1
----> 1 loader.load()
``````output
File ~/repos/langchain/libs/community/langchain_community/document_loaders/directory.py:117, in DirectoryLoader.load(self)
115 def load(self) -> List[Document]:
116 """Load documents."""
--> 117 return list(self.lazy_load())
``````output
File ~/repos/langchain/libs/community/langchain_community/document_loaders/directory.py:182, in DirectoryLoader.lazy_load(self)
180 else:
181 for i in items:
--> 182 yield from self._lazy_load_file(i, p, pbar)
184 if pbar:
185 pbar.close()
``````output
File ~/repos/langchain/libs/community/langchain_community/document_loaders/directory.py:220, in DirectoryLoader._lazy_load_file(self, item, path, pbar)
218 else:
219 logger.error(f"Error loading file {str(item)}")
--> 220 raise e
221 finally:
222 if pbar:
``````output
File ~/repos/langchain/libs/community/langchain_community/document_loaders/directory.py:210, in DirectoryLoader._lazy_load_file(self, item, path, pbar)
208 loader = self.loader_cls(str(item), **self.loader_kwargs)
209 try:
--> 210 for subdoc in loader.lazy_load():
211 yield subdoc
212 except NotImplementedError:
``````output
File ~/repos/langchain/libs/community/langchain_community/document_loaders/text.py:56, in TextLoader.lazy_load(self)
54 continue
55 else:
---> 56 raise RuntimeError(f"Error loading {self.file_path}") from e
57 except Exception as e:
58 raise RuntimeError(f"Error loading {self.file_path}") from e
``````output
RuntimeError: Error loading ../../../../libs/langchain/tests/unit_tests/examples/example-non-utf8.txt
The fileexample-non-utf8.txt uses a different encoding, so theload() function fails with a helpful message indicating which file failed decoding.
With the default behavior ofTextLoader any failure to load any of the documents will fail the whole loading process and no documents are loaded.
B. Silent fail
We can pass the parametersilent_errors to theDirectoryLoader to skip the files which could not be loaded and continue the load process.
loader= DirectoryLoader(
path, glob="**/*.txt", loader_cls=TextLoader, silent_errors=True
)
docs= loader.load()
Error loading file ../../../../libs/langchain/tests/unit_tests/examples/example-non-utf8.txt: Error loading ../../../../libs/langchain/tests/unit_tests/examples/example-non-utf8.txt
doc_sources=[doc.metadata["source"]for docin docs]
doc_sources
['../../../../libs/langchain/tests/unit_tests/examples/example-utf8.txt']
C. Auto detect encodings
We can also askTextLoader to auto detect the file encoding before failing, by passing theautodetect_encoding to the loader class.
text_loader_kwargs={"autodetect_encoding":True}
loader= DirectoryLoader(
path, glob="**/*.txt", loader_cls=TextLoader, loader_kwargs=text_loader_kwargs
)
docs= loader.load()
doc_sources=[doc.metadata["source"]for docin docs]
doc_sources
['../../../../libs/langchain/tests/unit_tests/examples/example-utf8.txt',
'../../../../libs/langchain/tests/unit_tests/examples/example-non-utf8.txt']