Repository files navigation

The missing GoPro workflow and data manager for macOS.

For those with one or more GoPro cameras. When using these action cameras regularly,the limitations of the GoPro ecosystem quickly become very obvious.GoPro is focusing its efforts on its mobile app experience, and not much developmentis directed toward macOS.GoProX is geared toward (semi-) professionals and prosumers that simply needmore from their devices.GoProX is based on a data-first approach to importing, processing and maintainingmedia generated by various GoPro devices. At this moment the tool is activelydeveloped and tested with GoPro Hero8, Hero9, Hero10, Hero11 and GoPro Max.

The most common way to installgoprox is via home-brew.

To installgoprox simply type

brew install fxstein/fxstein/goprox

Alternatively, you can add the fxstein tap manually before installing goprox

brew tap fxstein/fxsteinbrew install goprox

Once installed, you can upgradegoprox by simply running

brew upgrade goprox

Alternatively, you can also uninstall and reinstall goprox in case of issues with theupgrade

brew uninstall goproxbrew install goprox

To simplify repeat usage ofgoprox it supports saving its configuration in~/.goprox.To create this configurations file, simply executegoprox with its--setup option:

goprox --setup --library "/mylibrary/dir" --source "." --copyright "My Name"

You can also create alternate configuration files by specifying the--config option:

goprox --config "myotherconfig" --setup --library "/myotherlibrary" --source "." --copyright "My Other Name"

All subsequent runs ofgoprox will by default leverage the setting stored in~/.goproxunless the--config option is specified with an alternate configuration file.

The--setup option can be rerun as often as desired to change the settings stored in~/.goprox.Whenever--setup detects a prior configuration it creates a backup copy named like.goprox.bak.1657302965. The configuration file is a simple text file that is sourced ingoprox.

As part of the setup,goprox will create a library skeleton unless the library already exists.The default location is~/goprox, but placing the library on a dedicated storage device is highlyencouraged as the data volumes will be very significant.

This is the hierarchy of thegoprox library:

goprox/ (named and placed as required)├── archive/├── imported/├── processed/└── deleted/

All subsequent runs ofgoprox will validate this storage hierarchy. See the storage hierarchy sectionfor more details

Once installed and set up GoProX maintains a file system based media library thatis organized by time, camera types and media file types. The main goal is tomaintain all file details from the original to the processed items, even if mediafiles are further being processed in various apps including e.g.Apple Photos orBlackmagic DaVinci Resolve.Unfortunately, many commonly used apps ignore most of the important metadata,making it very hard for users to filter, sort or search for common things inside ever-growinglibraries of media.GoProX performs a few simple tasks that will make your life with GoPromedia on Apple platforms a lot easier.First, it renames the files as part of--import and--process tasks. This is anoften overlooked step that will lead to the loss of some of the most basic metadataover time. Especially when files get exported, copied or moved, things like the original date& time, and the source of the original image can get lost easily, creating issues down the line.

It then adds additional tags and keywords into the processed media files to makesearching, filtering and corrections a lot easier.

GoProX supports--geonames lookups for GPS-based timezone information,--firmware checks and upgrades,--archive of raw media into compressedarchives before--import and--process tasks are being applied, a--cleantask to remove processed media from storage devices as well as a--timeshifttask that makes it possible to bulk change the date and time information ofmedia files. It also makes it easy to apply default--copyright tags to allprocessed media.Default options can be set using the--setup tasks and settings are beingstored in the user's home directory in~.goprox.The output can be timestamped via the--time option to log long-running tasks.For developers, GoProX comes with a small set of test data from various camerasthat allows for testing and validation of changes with the--test option.

goprox --import

The--import option will read fromsource (default to the current directory.) andimport all image and video media files intolibrary\imported.This is best used for importing media files from the path of a mounted media card. To doso, insert your camera's microSD card into a reader attached to your Mac. Open Finderto see the mounted card, right-click it and selectNew Terminal at Folder. This willopen a new zsh terminal at the mounted card folder.

Alternatively--import can process atar.gz archive directly. It will first untarthe contents into a temp directory and then import from there. This is helpful whenre-importing previously archived sd card data

goprox --archive

The--archive option will create a full archive of thesource folder as a tarball insidelibrary\archive named like20220802215621_GoPro_Hero10_2442.tar.gz. This is useful topreserve the full content of the SD card before making any changes to it.

goprox --archive --import --clean

All the options can be performed in a single pass. In this examplegoprox will firstcreate a newarchive of thesource, then import all the media files contained on thesource and finally remove all media files from thesource. The--clean option willonly execute when combined with either--archive or--import or both to avoidaccidental deletion of media files.

goprox --firmware

The--firmware option will check the camera model and current firmware version of the sd cardcurrently mounted and will upgrade it to the latest GoPro firmware for your model if a newerone exists. Simply put the sd card back into the camera and on the next boot up the camera willupgrade itself.

Thelabs modifier:goprox --firmware labs will perform the same check for the latest GoProLabs firmware version. Omitting thelabs modifier in a later firmware check will return thecamera to the latest official firmware.

goprox --import --time

Adding the--time option creates a timestamped output of thegoprox run to aid in loggingfor long-running tasks that import or process thousands of media files.All output ofgoprox is getting timestamped like

[2022-08-04 12:15:09] Info: goprox v00.08.08

goprox --archive --import --clean --time --firmware

This example combines the most commonly used features for any import of media files directlyfrom the camera's sd card.--archive creates a full tarball of the content--import imports all media files into thelibrary--clean removes all media files from the sd card upon successful completion of the--archive and--import tasks--time timestamps allgoprox output--fimware checks the sd-card for the current firmware version of the camera and if necessaryinstalls the latest firmware onto the card for an automatic upgrade next time the camera isbooted up. For this to function properly sd-cards of different cameras should not be mixed.

It is recommended to perform as manyimport tasks as you have cameras with new footage.Multiple camera sd-card can be imported simultaneously if a multi-card reader is available.

goprox --process --time

The--process option takes unmodified imported media files and rewrites them with enhancedmetadata. This is wheregoprox inserts tags and flags into the media files that are then pickedup by the likes of Apple Photos. For GoPro video media (mp4 & 360)--process also performsa UTC timeshift of all the Quicktime tags to allow downstream tools like Apple or Google Photosto display the correct date and time.

By default,process will look for newly imported media filessince the lastprocess run. Alternatively,all or any valid time window can be specified*[0-9](y|m|w|d|H|M|S).For example,goprox --process 30d will process the past 30d whilegoprox --process all willprocess all imported media files. Caution should be used when reprocessing older media files as thecontent of a file will change with a newer version ofgoprox, as any change in metadata will leadto a modified file.

GoPro uses various file-naming conventions in its cameras. There is noclear structure and in most cases, you get some sort of prefix followed by arunning number and a file extension.

Here are some examples of GoPro Filenames:

GOPR0001.JPGGH010008.MP4GX010408.MP4GS__3305.JPGGS013331.360

If you happen to leverage the GoPro+ Cloud it gets a lot weirder with thingslike

GPTempDownload.jpg

for anything GoPro Quik touches and forwards to e.g. Apple Photos.

There are many problems with these filenames - many not unique to GoPro. First of all,they are very non-descriptive. You might be able to deduct the cameramodel to some extent, but that's about it. No date/time in case that informationgets lost somewhere along the way. But also the fact that these names arecreated for a world where users only ever use a single camera at the same time.As all cameras start with0001 for the first media and continuously count up, it isonly a question of time when you will run into naming conflicts with two ormore cameras. That usually means one file from one camera will overwrite anotherfile from another camera. And depending on usage each camera willeventually, in some cases even regularly, restart at0001.

So instead of very complicated folder structures to keep images from collidingwith each other, GoProX creates more sophisticated filenames that allow you tosort and act on the filenames themselves.

Let's take a look at the names of the files GoProX creates on--import:

20211010090947_GoPro_Hero10_7678_GOPR0768.JPG20160130223238_GoPro_Max_6013_GS__1596.JPG20210615110514_GoPro_Hero9_9650_GOPR1353.JPG20201231054457_GoPro_Hero8_1659_GOPR0129.JPG20210731003746_GoPro_Hero9_4139_GH013340.MP420211011072108_GoPro_Max_6013_GS012830.360

The first 8 digits are the original date when the photo was taken, followed by 6digits of the time, followed by the model of the camera. The for digits afterthat are the last four digits of the camera's serial number - to allow you tofind all the media from a particular camera. Especially important if you findout after the fact that a setting was wrong in one of your cameras (Like whenyou find a 2016 date for GoPro Max media). Finally, the original filename andextension are added.

Right there is a wealth of information that helps with a lot of issues and evensoftware bugs in the likes of Apple Photos that for example messes up the date &time for imported MP4 files. As the filename is kept along with the images andvideos it gives you a very simple way to double-check what is going on.

.360 videos from the GoPro Max listed here cannot be imported directly into thelikes of Apple Photos. You will need GoPro Player on the Mac or GoPro Quik on iOSto process them for further consumption. Alternatively, they can be used as mp4files and processed in platforms like DaVinci Resolve or Adobe Premier with properplugins installed.

The--process option further refines the filenames and also adds and rewritesembedded metadata. All file extensions are normalized to lowercase extensionsand the files are sorted by main file type.As part of that process,.360 files become.mp4 files that are otherwiseidentical to the GoPro.360 format but can now be handled by most downstreamapplications.

Processed files get an additionalP_ prefix to delineate from the unmodifiedimported files.

P_20210606094917_GoPro_Hero9_4139_GOPR0182.jpgP_20210806114935_GoPro_Hero9_4139_GOPR3422.jpgP_20220206144720_GoPro_Hero10_8034_GOPR2313.jpgP_20220206145556_GoPro_Hero10_8034_GOPR2320.jpg...P_20210627102316_GoPro_Hero9_0021_GH013156.mp4P_20211015084940_GoPro_Max_6013_GS013292.mp4

In addition to the names of the files getting supercharged, there is a ton ofinformation available inside the Exif metadata that macOS or Apple Photos keepintact but ignores and does not even display unless you use Preview on a particularfile to inspect the metadata.Apple Photos only gives you Title, Caption and Tags to work with to storeadditional information aside from the most basic metadata.

This is where GoProX performs a little metadata shuffle to make at least someattributes searchable from within Apple Photos.

Several of the low cardinality Exif fields are getting converted to a list oftags. That list Apple Photos converts to its tags when importing the media.

Here is a list of tags that are being created by GoProX:

Make: ...Camera: ...Camera: ... ....Software: ...AutoRotation: ...Orientation: ...SceneCaptureType: ...ProTune: ...Sharpness: ...MeteringMode: ...GainControl: ...Contrast: ...Saturation: ...WhiteBalance: ...HDRSetting: ...ExposureLockUsed: ...ProjectionType: ...ImageStabilization: ...

These tags can be used inside of Apple Photos for various tasks, mostimportantly to create Smart Albums that key off one or more of them.

As you will capture more and more media with multiple GoPro cameras, it isalways a good idea to set the Artist/Author/Copyright fields in Exif.That way you keep a record of the files you have created.

All of this needs to be part of a simple workflow that captures and stores theoriginal media as well as the processed files.GoProX will most likely only be one step in your process and as been designedto recursively walk the source tree and process all media files in itsstructure. It does not matter if the source images are in a single folder ororganized by date, camera or whatever subject you choose.

On the output side, a very simple data-based file structure is created with onefolder per original media creation date.

The tool can be run on a single subfolder of your source files or at the root ofit. The output will always be the date folder hierarchy at the destinationfolder.

When re-run, existing target files are skipped but logged in the error log. Thatway large or incremental processing jobs can be restarted as often as youlike. This comes with some processing overhead for existing files but isgenerally very fast.

If you want to reprocess certain files, simply delete them at the destinationfolder and rerun the process.

goprox is based on a simple file-based library structure for arching, importing,and processing media files over time. By default, it creates the following structure ondisk:

goprox/ (named and placed as required)├── archive/├── imported/├── processed/└── deleted/

Any of these first-level storage subtrees can be kept as directories or replaced bylinks to different storage devices. This helps distribute storage requirements acrossmultiple devices but can be equally beneficial for processing performance.

The following example keepsarchive anddeleted on the main drive while pointingimported andprocessed to different locations.

goprox/ (named and placed as required)├── archive/├── imported -> /Volumes/Office G-RAID/goprox/imported/├── processed -> /Volumes/Office G-RAID/goprox/processed/└── deleted/

To create this distributed structure, simply create anothergoprox library skeletonon any storage device of your choice (in this case/Volumes/Office G-RAID/) (onlythe portions of the structure you want to link to are required). For example:

/Volumes/Office G-RAID/goprox/ ├── imported├── processed

You could also consider puttingarchive on a dedicated low-cost storage device:

/Volumes/Office Dock/goprox/ ├── archive

Once created, head over to the main library, remove the empty directories (if you areperforming this as a migration see below) you would like to point to differentlocations, and simple run:

ln -s /Volumes/Office\ G-RAID/goprox/imported/ importedln -s /Volumes/Office\ G-RAID/goprox/processed/ processed

Once set up as required,goprox will be fully aware of the distributed nature of thelibrary and will validate the structure every time you run it. As part of thevalidation, it will check and warn of broken links and, if necessary, stop the executionof operations that would require the portions of the library that are unreachable.

This is necessary to avoid situations where external storage devices are not mountedor simply not available when on the road.

Simple rungoprox --verbose with no other parameters to get the storage validation summary:

...Info: Validating storage hierarchy... Info: goprox library: /Users/xxxxxxx/goprox directory validated Info: goprox archive: /Users/xxxxxxx/goprox/archive directory validated Warning: goprox imported: /Users/xxxxxxx/goprox/imported is a broken link to /Volumes/Office G-RAID/goprox/imported/Warning: Make sure the storage device is mounted and the directory has not been moved. Warning: goprox processed: /Users/xxxxxxx/goprox/processed is a broken link to /Volumes/Office G-RAID/goprox/processed/ Warning: Make sure the storage device is mounted and the directory has not been moved. Info: goprox deleted: /Users/xxxxxxx/goprox/deleted directory validated Info: Finished storage hierarchy validation. Info: GoProx processing finished.

In this particular example, you can still perform--archive,--firmware, or--clean tasksbut will see an error if you attempt--import or--process.

Once you get started, you will seegoprox fill the library's content with yourGoPro media data.

Here is a sample summary of howgoprox builds the tree inside the various components:

goprox/├── archive/│   ├── 20221015125658_GoPro_Hero10_8034.tar.gz│   ├── 20221015131037_GoPro Max_6013.tar.gz│   ├── 20221015103421_GoPro_Hero11_4632.tar.gz│   └── 20220713183316_GoPro_Hero9_0021.tar.gz├── imported/│   ├── 2021│   └── 2022│       ├── 20220520│       └── 20221013│           ├── 20221013084759_GoPro_Hero11_5131_G0294305.JPG│           ├── 20221013094220_GoPro_Hero10_4299_G0019484.JPG│           ├── 20221013153428_GoPro_Max_6013_GS016167.360│           └── geonames.json└── processed/    ├── JPEG    │   ├── 2021    │   └── 2022    │       ├── 20220520    │       └── 20221013    │           ├── P_20221013084759_GoPro_Hero11_5131_G0294305.JPG    │           └── P_20221013094220_GoPro_Hero10_4299_G0019484.JPG    ├── MP4    └── 360        ├── 2021        └── 2022            ├── 20220520            └── 20221013                └── P_20221013153428_GoPro_Max_6013_GS016167.360

archive is a flat directory with the individual sdcard image backups astar.gz.These archives usually get migrated to long term storage (eg: AWS Glacier) or simplydeleted after a while. When you get started withgoprox, it is stronglyrecommended to perform archives as they are a simple, foolproof way to undo anythingthat could go wrong as you experiment with features and your workflow.goprox supports imports straight from an archive file. This allows you to rerunany process from scratch without ever losing any data. The cost is an extra copyof all the media.

imported contains a structure by year and then by date to keep the hierarchymanageable. Within a particular day, you will find all mediagoprox has imported.These dates are derived from when the media was created/shot and NOT when it wasimported. That information you can find in the modification dates of each file.goprox has a strict DO-NOT-OVERRIDE existing files policy. It only adds, neverreplaces. If you want to rerun a particular day, delete those files within the treeand re-run the import from your archives. Be aware that the archive dates do notnecessarily correspond with the individual file date. The archives are labeled bythe day and time the archive has been created. All media files inside ofimportedare unmodified but renamed. It is the exact media that came off your sdcard butintelligently named according to the files EXIF data.

processed contains a further refined structure by file type, then year and date.This is done because, in many cases, JPEG, MP4, and 360 files go through differentworkflows in post-production. For example, JPEGs might get imported into Apple Photos,whereas MP4s get used by Blackmagic DaVinci or similar. 360s usually haveto be processed by GoPro Player first before they can be used in cutting and editing.processed media files are rewritten with additional metadata that allows Appsto search, sort, and filter these files in new ways. To be able to distinguishimported fromprocessed media, aP_ prefix is added to all processedmedia.

deleted is currently a placeholder for future functionality to allow upstreamdeletion propagation back into the goprox library. It will containlists of files that have been deleted in upstream Apps like Apple Photos to removethose files retroactively from thegoprox library.

For extra performance, you could consider placingimported andprocessed ondifferent SSDs. Internal NVME drives can get up to 1.3GB/s transfer rates, whileexternal devices like Samsung's T7s max out at 600-700MB/s. The G-Raid can doabout 400MB/s in RAID 0 and 200MB/s in RAID 1.

If you need to migrate file structures from one device to another, it is highlyrecommended to bypass Finder and usecp in archive mode to maximize copy performancebut also preserve owner, date & time as well as other attributes in your library.

cp -RpPvn source targetcp -RpPvn /Volumes/Original/goprox/imported /Volumes/Office\ G-RAID/goprox

This will safely copy all the contents of the source to the new target. It will skipexisting files (so you can re-run it in case of a failure) but it is important tocapture and review its output, particularly the error out. For long-running migrationswrap in nohup:

nohup cp -RpPvn source target 1>>cp-progress.log 2>>cp-errors.lognohup cp -RpPvn /Volumes/Original/goprox/imported /Volumes/Office\ G-RAID/goprox 1>>goprox-cp-progress.log 2>>goprox-cp-errors.log

goprox is actively being developed and tested with the following cameras:

• GoPro Hero 8 (HD8.01.02.51.00)
• GoPro Hero 9 (HD9.01.01.72.00)
• GoPro Hero 10 (H21.01.01.46.00)
• GoPro Hero 11 (H22.01.01.10.00)
• GoPro Max (H19.03.02.00.00)
• GoPro The Remote (GP.REMOTE.FW.01.02.00)

Most functionality should work with older GoPro models as well, just the--firmware and--firmware labs options are limited to these models.

Theexiftool has phenomenal capabilities and if you letit do the vast majority of work by itself, it is really fast.

Testing on 20,000 images on a 2017 iMac with a quad-core i5 and one 2TB SSD(source: original files folder) plus one 14TB HDD (target: processed filesfolder) have continuously resulted in 200MB+/sec read and write speeds at lessthan 20% CPU utilization.

I tried to keep the tool as simple as possible. This undoubtedly will come withlimitations for certain use-cases. Feel free to drop me a line or a PR withenhancements.

GoProX is not related to GoPro Inc or its products. It is not supported norendorsed by GoPro Inc.GoProX is not related to Apple Inc, Blackmagic Design Pty Ltd or any otherproducts or platforms referred to as part of the description or documentation ofthe tool unless explicitly stated.

GoProX leverages theexiftool by Phil Harvey to extractcommon GoPro metadata to name and tag images and videos for further processing.

GoProX leverages theGeoNames database by Marc Wickto perform GPS-based geocode lookups of timezones and related data.

Portions of code and documentation created with the help ofChatGPT.

Homebrew tap to enable the installation ofgoprox:homebrew-fxstein