Repository files navigation

This is a collection of common utility functions and classes that we at the Caltech Library have found useful in our other Python projects.

• Introduction
• Installation
• Usage
• Getting help
• Contributing
• License
• Authors and history
• Acknowledgments

This repository does not constitute a single program; instead, it contains a collection of modules with utility functions and classes that we have found ourselves using repeatedly in other Python projects.

The instructions below assume you have a Python interpreter installed on your computer; if that's not the case, please firstinstall Python version 3 and familiarize yourself with running Python programs on your system.

OnLinux,macOS, andWindows operating systems, you should be able to installcommonpy withpip. To installcommonpy from thePython package repository (PyPI), run the following command:

python3 -m pip install commonpy

As an alternative to getting it fromPyPI, you can usepip to installcommonpy directly from GitHub, like this:

python3 -m pip install git+https://github.com/caltechlibrary/commonpy.git

The basic approach to using this package is to import the modules and functions you need. For example:

fromcommonpy.file_utilsimportreadableifreadable('/path/to/some/file'):# do something

The following subsections describe the different modules available.

Thedata_structures module provides miscellaneous data classes.

Thedata_utils module provides a number of miscellaneous simple functions for some common operations on data of various kinds.

Thefile_utils module provides a number of miscellaneous simple functions for some common operations on files and directories.

Theinterrupt module includeswait(...), a replacement forsleep(...) that is interruptible and works with multiple threads. It also provides methods to cause an interruption (including doing it by issuing a^C to the program), check whether an interruption occurred, and other related operations.

Themodule_utils collection of functions is useful for working with paths related to a running module, for example to find internal data files that might be needed for normal operation.

Functionconfig_path(...) is useful to use in conjunction with Python'sconfigparser module. It returns~/.config/modulename/ on Unix-like systems.

Thenetwork_utils module provides several functions that are useful when performing network operations.

Thenetwork andnet functions in thenetwork_utils module implements a fairly high-level network operation interface that internally handles timeouts, rate limits, polling, HTTP/2, and more. The function signatures are identical to this:

network(method,url,client=None,handle_rate=True,polling=False,**kwargs)

The difference between the two functions is their behavior with respect to exceptions. The functionnetwork returns only aresponse object, and raises an exception if any error occurs. Thenet function returns two values:response, error and does not raise exceptions except in the case of bad arguments; instead, any exceptions are returned as theerror value in the list of return values. This allows the caller to inspect theresponse object even in cases where exceptions are raised.

Theresponse object returned by bothnet andnetwork is the response from the get or post HTTP call. This object comes from the PythonHTTPX module used by

Themethod parameter value must be a string chosen from the list of known HTTP methods:"get","post","head","options","put","delete", or"patch".

Theurl parameter value must be the URL to which the HTTP method will be applied.

If keywordclient is notNone, it's assumed to be aHTTPX Client object to use for the network call. Settings such as timeouts should be done by the caller creating appropriately-configuredClient objects.

If keywordhandle_rate isTrue, both functions will automatically pause and retry if it receives an HTTP code 429 ("too many requests") from the server. IfFalse, it will return the exceptionCommonPy.exceptions.RateLimitExceeded instead.

If keywordpolling isTrue, certain statuses like 404 are ignored and the response is returned; otherwise, they are considered errors. The behavior whenTrue is useful in situations where a URL does not exist until something is ready at the server, and the caller is repeatedly checking the URL. It is up to the caller to implement the polling schedule and call this function (withpolling = True) as needed.

Additional keyword arguments understood byHTTPX can be passed to bothnetwork andnet.

Both methods always pass the argumentallow_redirects = True to the underlying Python HTTPX library network calls.

The functionsdownload(url, local_destination) anddownload_file(url, local_destination) download a file at the givenurl, writing it to the file specified by the parameterlocal_destination. The former version of the function will raise exceptions in case of problems; the latter version simply returnTrue orFalse depending on the success of the download.

The CommonPy module defines a number of exceptions that it may return. (Most of the exceptions are potentially thrown bynet, discussed above.)

If you find an issue, please submit it inthe GitHub issue tracker for this repository.

We would be happy to receive your help and participation with enhancing CommonPy! Please visit theguidelines for contributing for some tips on getting started.

If you plan on doing any development on CommonPy, you may want to install the package dependencies listed inrequirements-dev.txt, e.g., using a command such as the following. This will install dependencies necessary to runpytest.

python3 -m pip install -r requirements-dev.txt

Software produced by the Caltech Library is Copyright (C) 2020-2023, Caltech. This software is freely distributed under a BSD/MIT type license. Please see theLICENSE file for more information.

Mike Hucka started this collection of utilities sometime in 2018.

This work was funded by the California Institute of Technology Library.

Thevector artwork of a toolbox, used as the icon for this repository, was created bypriyanka from the Noun Project. It is licensed under the Creative CommonsCC-BY 3.0 license.

CommonPy makes use of numerous open-source packages, without which it would have been effectively impossible to develop CommonPy with the resources we had. I want to acknowledge this debt. In alphabetical order, the packages are:

• boltons – package of miscellaneous Python utilities
• dateparser – parse dates in almost any string format
• deprecation – a library to handle deprecation of code in your Python packages
• h2 – HTTP/2 support library used byHTTPX
• httpx – Python HTTP client library that supports HTTP/2
• humanize – make numbers more easily readable by humans
• ipdb – the IPython debugger
• pytest – testing framework for Python
• pytest-mock – wrapper around themock package for use withpytest
• python_dateutil – date utilities
• PyYAML – Python YAML parser
• pywin32 – Windows APIs for Python
• sidetrack – simple debug logging/tracing package
• tldextract – module to parse domains from URLs
• twine – package for publishing Python packages to PyPI
• validator-collection – collection of Python functions for validating data