Repository files navigation

pyuvdata defines a pythonic interface to interferometric data sets.Currently pyuvdata supports reading and writing of miriad, uvfits, CASA measurement setsand uvh5 files and reading of FHD(Fast Holographic Deconvolution) visibilitysave files, SMA Mir files and MWA correlator FITS files.

Documentation of the API, conventions used throughout the package, and a tutorialis hosted onReadTheDocs.

The main goals are:

1. To provide a high quality, well documented path to convert between file formats.
2. Support the direct use of interferometric datasets from python with minimal software.
3. Provide precise data definitions and convention details via both humanreadable code and high quality documentation.

pyuvdata has four major user classes:

• UVData: supports interferometric data (visibilities) and associated metadata
• UVCal: supports interferometric calibration solutions (antenna-based) andassociated metadata.
• UVBeam: supports primary beams (E-field or power) and associated metadata. Anumber of analytic beam objects are also available and the BeamInterface objectserves as a unified interface for UVBeam and analytic beam objects.
• UVFlag: A class to handle the manipulation and combination of flags for data sets.Also can convert raw data quality metrics into flags using thresholding.(This object is somewhat new and experimental. Consider it to be a beta version)

• Miriad has been thoroughly tested with aipy-style Miriad files and minimallytested with ATCA files. Reading/writing Miriad files is not supported on Windows.
• UVFITS conforms to AIPS memo 117 (as of March 2020). It is tested againstFHD, CASA, and AIPS. However AIPS is limited to <80 antennas and CASA'simportuvfitstask does not seem to support >255 antennas. Because of this and other limitationsto CASA'simportuvfits task, we reccommend that users planning to work in CASAavoid usingimportuvfits and use the measurement set writer instead.
• CASA measurement sets, primarily conforming toCASA Memo 229, with some elements takenfrom the proposed v3.0 format documented inCASA Memo 264. Measurementsets are tested against datasets from VLA, MWA (filled viacotter),ALMA, and SMA (filled using theimportuvfits task). Extensive loopback testinghas been done to verify that pyuvdata written measurement sets are compatiblewith CASA.
• UVH5 is an HDF5-based file format defined by the HERA collaboration,details in theuvh5 memo. Note that this is asomewhat new format, so it may evolve a bit but we will strive to make futureversions backwards compatible with the current format.It is probably not compatible with other interferometric HDF5 files defined by other groups.
• FHD (read-only support, tested against MWA and PAPER data)
• MIR (read-only support, though experimental write functions are available, tested against SMA data)
• MWA correlator FITS files (read-only support, tested against Cotter outputs and FHD)

• CalH5: a new an HDF5-based file format defined in pyuvdata, details in thecalh5 memo.Note that this format is a somewhat new format, so it may evolve a bit but wewill strive to make future versions backwards compatible with the current format.
• Measurement Set calibration files (read and write, gains/delay/bandpass supported,beta version). Tested against a limited set of SMA, LWA, and VLA calibrationfiles generated in CASA.
• CalFITS: a custom format defined in pyuvdata, details in thecalfits memo.Note that this format does not support all possible UVCal objects, so we preferCalH5 which has full support for all variations of UVCal objects.
• FHD calibration files (read-only support)

• BeamFITS: a custom format defined in pyuvdata that supports both regularlygridded beams and beams on a HEALPix grid for both E-field and power beams,details in thebeamfits memo.
• CST beam text files (read only support) with a defined yaml file format formetadata, details here:cst settings file
• MWA Beams (read only support)

• Incorporating numba to alleviate bottlenecks and to replace some existingcython extensions as appropriate.

See ourissue logfor a full list.

Contributions to this package to add new file formats or address any of theissues in theissue logare very welcome, as are bug reports and feature requests.Please see ourguide on contributing

pyuvdata has been used with data from the following telescopes. If you use it ondata from a telescope we don't have listed here please let us know how it wentvia an issue! We would like to make pyuvdata generally useful to the community foras many telescopes as possible.

• MWA
• HERA
• PAPER
• LWA
• ALMA
• VLA
• ATCA
• SMA
• ATA
• VLBA

We use ageneration.major.minor version number format. We use thegenerationnumber for very significant improvements or major rewrites, themajor numberto indicate substantial package changes (intended to be released every 3-4 months)and theminor number to release smaller incremental updates (intended to bereleased approximately monthly and which usually do not include breaking APIchanges). We do our best to provide a significant period (usually 2 majorgenerations) of deprecation warnings for all breaking changes to the API.We track all changes in ourchangelog.

pyuvdata was originally developed in the low frequency 21cm community to supportthe development of and interchange of data between calibration and foregroundsubtraction pipelines. Particular focus has been paid to supporting drift andphased array modes.

Please cite both of our JOSS papers:

Keating et al., (2025). pyuvdata v3: an interface for astronomical interferometric data sets in Python. Journal of Open Source Software, 10(109), 7482,https://doi.org/10.21105/joss.07482 [ADS Link]

Hazelton et al, (2017), pyuvdata: an interface for astronomical interferometeric datasets in python, Journal of Open Source Software, 2(10), 140, doi:10.21105/joss.00140 [ADS Link]

Simple installation via conda is available for users, developers should followthe directions underDeveloper Installation below.

For simple installation, the latest stable version is available via conda(preferred:conda install -c conda-forge pyuvdata) or pip (pip install pyuvdata).

There are some optional dependencies that are required for specific functionality,which will not be installed automatically by conda or pip.SeeDependencies for details on installing optional dependencies.

Required:

• astropy >= 6.0
• docstring_parser>=0.15
• h5py >= 3.7
• numba >= 0.57.0
• numpy >= 1.23
• pyerfa >= 2.0.1.1
• python >= 3.11
• pyyaml >= 5.4.1
• scipy >= 1.9
• setuptools_scm >= 8.1

Optional:

• astropy-healpix >= 1.0.2 (for working with beams in HEALPix formats)
• astroquery >= 0.4.4 (for enabling phasing to ephemeris objects using JPL-Horizons)
• hdf5plugin >= 3.3.1 (for enabling bitshuffle and other hdf5 compression filters in uvh5 files)
• lunarsky >=0.2.5 (for working with simulated datasets for lunar telescopes)
• novas and novas_de405 (for using the NOVAS library for astrometry)
• python-casacore >= 3.5.2 (for working with CASA measurement sets)

The numpy and astropy versions are important, so make sure these are up to date.

We suggest using conda to install all the dependencies. If you want to installpython-casacore and astropy-healpix, you'll need to add conda-forge as a channel(conda config --add channels conda-forge).

If you do not want to use conda, the packages are also available on PyPI(python-casacore may require more effort, see details for that package below).You can install the optional dependencies via pip by specifying an optionwhen you install pyuvdata, as inpip install pyuvdata[healpix]which will install all the required packages for using the HEALPix functionalityin pyuvdata. The options that can be passed in this way are:

• astroquery
• casa
• hdf5_compression
• healpix
• lunar
• novas
• all
• test
• doc
• dev

The first set (astroquery,casa,hdf5_compression,healpix,lunar,andnovas) enable various specific functionality whileall will installall of the above to enable all functionality. The last three (test,docanddev) include everything installed withall plus packages for testingand building the docs which may be useful for developers of pyuvdata.

python-casacore requires the casacore c++ libraries. It can be installed easilyusing conda (python-casacore on conda-forge).

If you do not want to use conda, the casacore c++ libraries are available forubuntu through thekern suite. On OSX, casacore isavailable through theska-sa brew tap.The python-casacore library (with manual install instructions) is available athttps://github.com/casacore/python-casacore

Clone the repository usinggit clone https://github.com/RadioAstronomySoftwareGroup/pyuvdata.git

Navigate into the pyuvdata directory and runpip install .(note thatpython setup.py install does not work).Note that this will attempt to automatically install any missing dependencies.If you use conda or another package manager you might prefer to first installthe dependencies as described inDependencies.

To install without dependencies, runpip install --no-deps .

To compile the binary extension modules such that you can successfully runimport pyuvdata from the top-level directory of your Git checkout, run:python setup.py build_ext --inplace

If you want to do development on pyuvdata, in addition to the other dependenciesyou will need the following packages:

• pytest >= 8.2
• pytest-cases >= 3.9.1
• pytest-cov
• cython == 3.0
• coverage
• pre-commit
• matplotlib
• sphinx
• pypandoc

One other package, pytest-xdist, is not required, but can be used to speed up runningthe test suite by running tests in parallel. To use it call pytest with the-n auto option.

One way to ensure you have all the needed packages is to use the includedenvironment.yaml file to create a new environment that willcontain all the optional dependencies along with dependencies required fortesting and development (conda env create -f environment.yaml).Alternatively, you can specifytest,doc, ordev when installing pyuvdata(as inpip install pyuvdata[dev]) to install the packages needed for testing(including coverage and linting) and documentation development;dev includes everything intest anddoc.

To use pre-commit to prevent committing code that does not follow our style, you'llneed to runpre-commit install in the top levelpyuvdata directory.

Uses thepytest package to execute test suite.From the source pyuvdata directory runpytest orpython -m pytest.

Testing ofUVFlag module requires thepytest-cases plug-in.

pyuvdata is organized around objects that contain all the data and metadata requiredto represent and work with interferometric data, calibration solutions, flags,antenna beams and telescope layouts. Each object has the data and metadata asattributes along with many useful methods for importing and exporting files andmanipulating and transforming the data in useful ways. Please see our extensivedocumentation onReadTheDocs for tutorials andcomplete API details.

pyuvdata is maintained by the RASG Managers, which currently include:

• Adam Beardsley (Winona State University)
• Bryna Hazelton (University of Washington)
• Garrett "Karto" Keating (Smithsonian Astrophysical Observatory)
• Daniel Jacobs (Arizona State University)
• Matt Kolopanis (Arizona State University)
• Paul La Plante (University of Nevada, Las Vegas)
• Jonathan Pober (Brown University)

Please use the channels discussed in theguide on contributingfor code-related discussions. You can contact us privately if needed atrasgmanagers@gmail.com.

Support for pyuvdata was provided by NSF awards #1835421 and #1835120.