This PEP proposes setting up a new PSF provided resource, forge.python.org,as a location for maintaining various supporting repositories(such as the repository for Python Enhancement Proposals) in a way that ismore accessible to new contributors, and easier to manage for coredevelopers.
This PEP doesnot propose any changes to the core development workflowfor CPython itself (seePEP 462 in relation to that).
This PEP has beenwithdrawn by the authorin favour of the GitLab based proposal inPEP 507.
If anyone else would like to take over championing this PEP, contact thecore-workflow mailing list
This PEP proposes that an instance of the self-hosted Kallithea coderepository management system be deployed as “forge.python.org”.
Individual repositories (such as the developer guide or the PEPs repository)may then be migrated from the existing hg.python.org infrastructure to thenew forge.python.org infrastructure on a case-by-case basis. Each migrationwill need to decide whether to retain a read-only mirror on hg.python.org,or whether to just migrate wholesale to the new location.
In addition to supporting read-only mirrors on hg.python.org,forge.python.org will also aim to support hosting mirrors on popularproprietary hosting sites like GitHub and BitBucket. The aim will be toallow users familiar with these sites to submit and discuss pull requestsusing their preferred workflow, with forge.python.org automatically bringingthose contributions over to the master repository.
Given the availability and popularity of commercially backed “free for opensource projects” repository hosting services, this would not be a generalpurpose hosting site for arbitrary Python projects. The initial focus will bespecifically on CPython and other repositories currently hosted onhg.python.org. In the future, this could potentially be expanded toconsolidating other PSF managed repositories that are currently externallyhosted to gain access to a pull request based workflow, such as therepository for the python.org Django application. As with the initialmigrations, any such future migrations would be considered on a case-by-casebasis, taking into account the preferences of the primary users of eachrepository.
Currently, hg.python.org hosts more than just the core CPython repository,it also hosts other repositories such as those for the CPython developerguide and for Python Enhancement Proposals, along with various “sandbox”repositories for core developer experimentation.
While the simple “pull request” style workflow made popular by code hostingsites like GitHub and BitBucket isn’t adequate for the complex branchingmodel needed for parallel maintenance and development of the variousCPython releases, it’s a good fit for several of the ancillary projectsthat surround CPython that we don’t wish to move to a proprietary hostingsite.
The key requirements proposed for a PSF provided software forge are:
Additional recommended requirements that are satisfied by this proposal,but may be negotiable if a sufficiently compelling alternative is presented:
The preference for self-hosting without ongoing fees rules out thefree-as-in-beer providers like GitHub and BitBucket, in addition to thevarious proprietary source code management offerings.
The preference for Mercurial support not only rules out GitHub, but alsoother Git-only solutions like GitLab and Gitorious.
The hard requirement for online editing support rules out the ApacheAllura/HgForge combination.
The preference for a fully open source solution rules out RhodeCode.
Of the various options considered by the author of this proposal, thatleavesKallithea SCM as the proposedfoundation for a forge.python.org service.
Kallithea is a full GPLv3 application (derived from the clearlyand unambiguously GPLv3 licensed components of RhodeCode) that is beingdeveloped under the auspices of theSoftware Freedom Conservancy. TheConservancy hasaffirmed that theKallithea codebase is completely and validly licensed under GPLv3. Inaddition to their role in building the initial Kallithea community, theConservancy is also the legal home of both the Mercurial and Git projects.Other SFC member projects that may be familiar to Python users includeTwisted, Gevent, BuildBot and PyPy.
The primary benefit of deploying Kallithea as forge.python.org is thatsupporting repositories such as the developer guide and the PEP repo couldpotentially be managed using pull requests and online editing. This would bemuch simpler than the current workflow which requires PEP editors andother core developers to act as intermediaries to apply updates suggestedby other users.
The richer administrative functionality would also make it substantiallyeasier to grant users access to particular repositories for collaborationpurposes, without having to grant them general access to the entireinstallation. This helps lower barriers to entry, as trust can morereadily be granted and earned incrementally, rather than being anall-or-nothing decision around granting core developer access.
Even with its current workflow, CPython itself remains one of the largestopen source projects in the world (in thetop 2%of projects tracked on OpenHub). Unfortunately, we have been significantlyless effective at encouraging contributions to the projects that make upCPython’s workflow infrastructure, including ensuring that our installationstrack upstream, and that wherever feasible, our own customisations arecontributed back to the original project.
As such, a core component of this proposal is to actively engage with theupstream Kallithea community to lower the barriers to working with and onthe Kallithea SCM, as well as with the PSF Infrastructure team to ensurethe forge.python.org service integrates cleanly with the PSF’s infrastructureautomation.
This approach aims to provide a number of key benefits:
Some initial steps have already been taken to address these sustainingengineering concerns:
The next major step to be undertaken is to come up with a local developmentworkflow that allows contributors on Windows, Mac OS X and Linux to runthe Kallithea tests locally, without interfering with the operation oftheir own system. The currently planned approach for this is to focus onVagrant, which is a popular automated virtual machine management systemspecifically aimed at developers running local VMs for testing purposes.TheVagrant based development guidelinesfor OpenShift Origin provide an extended example of the kind of workflow thisapproach enables. It’s also worth noting that Vagrant is one of the optionsfor working with a local build of themain python.org website.
If these workflow proposals end up working well for Kallithea, they may alsobe worth proposing for use by the upstream projects backing other PSF andCPython infrastructure services, including Roundup, BuildBot, and the mainpython.org web site.
As of July 2015, I now work for Red Hat as a software development workflowdesigner and process architect, focusing on the upstream developer experiencein Fedora. Two of the key pieces of that experience will be familiar to manyweb service developers: Docker for local container management, and Vagrant forcross-platform local development VM management. Spending time applying thesetechnologies in multiple upstream contexts helps provide additional insightinto what works well and what still needs further improvement to provide a goodsoftware development experience that is well integrated on Fedora, but alsoreadily available on other Linux distributions, Windows, Mac OS X.
In relation to code review workflows in particular, the primary code reviewworkflow management tools I’ve used in my career areGerrit (for multi-step code review with fine-grained access control), GitHuband BitBucket (for basic pull request based workflows), and Rietveld (forCPython’s optional pre-commit reviews).
Kallithea is interesting as a base project to build, as it’s currently acombined repo hosting and code review management platform, but doesn’tdirectly integrate the two by offering online merges. This creates theopportunity to blend the low barrier to entry benefits of the GitHub/BitBucketpull request model with the mentoring and task hand-off benefits of Gerritin defining an online code merging model for Kallithea in collaboration withthe upstream Kallithea developers.
Introducing a new service into the CPython infrastructure presents a numberof interesting technical concerns and challenges. This section covers severalof the most significant ones.
The default position of this PEP is that the new forge.python.org servicewill be integrated into the existing PSF Salt infrastructure and hosted onthe PSF’s Rackspace cloud infrastructure.
However, other hosting options will also be considered, in particular,possible deployment as aKubernetes hosted webservice on eitherGoogle Container Engine orthe next generation of Red Hat’sOpenShift Online service, by using eitherGCEPersistentDisk or the open sourceGlusterFS distributed filesystemto hold the source code repositories.
Ongoing infrastructure maintenance is an area of concern within the PSF,as we currently lack a system administrator mentorship program equivalent totheFedora Infrastructure Apprentice orGNOME Infrastructure Apprenticeprograms.
Instead, systems tend to be maintained largely by developers as a part-timeactivity on top of their development related contributions, rather thanseeking to recruit folks that are more interested in operations (i.e.keeping existing systems running well) than they are in development (i.e.making changes to the services to provide new features or a better userexperience, or to address existing issues).
While I’d personally like to see the PSF operating such a program at somepoint in the future, I don’t consider setting one up to be afeasible near term goal. However, I do consider it feasible to continuelaying the groundwork for such a program by extending the PSF’s existingusage of modern infrastructure technologies like OpenStack and Salt tocover more services, as well as starting to explore the potential benefits ofcontainers and container platforms when it comes to maintaining and enhancingPSF provided services.
I also plan to look into the question of whether or not an open source cloudmanagement platform likeManageIQ may help usbring our emerging “cloud sprawl” problem across Rackspace, Google, Amazonand other services more under control.
Ideally we’d like to be able to offer a single account that spans allpython.org services, including Kallithea, Roundup/Rietveld, PyPI and theback end for the new python.org site, but actually implementing that wouldbe a distinct infrastructure project, independent of this PEP. (It’s alsoworth noting that the fine-grained control of ACLs offered by such acapability is a prerequisite for setting up aneffective system administrator mentorship program)
For the initial rollout of forge.python.org, we will likely create yet anotheridentity silo within the PSF infrastructure. A potentially superioralternative would be to add support forpython-social-auth to Kallithea, but actuallydoing so would not be a requirement for the initial rollout of the service(the main technical concern there is that Kallithea is a Pylons applicationthat has not yet been ported to Pyramid, so integration will require eitheradding a Pylons backend to python-social-auth, or else embarking on thePyramid migration in Kallithea).
This PEP proposes leaving the existing hg.python.org installation alone,and setting up Kallithea on a new host. This approach minimises the riskof interfering with the development of CPython itself (and any otherprojects that don’t migrate to the new software forge), but does make anymigrations of existing repos more disruptive (since existing checkouts willbreak).
Kallithea provides configurable issue tracker integration. This will needto be set up appropriately to integrate with the Roundup issue tracker atbugs.python.org before the initial rollout of the forge.python.org service.
The initial rollout of forge.python.org would support publication of read-onlymirrors, both on hg.python.org and other services, as that is a relativelystraightforward operation that can be implemented in a commit hook.
While a highly desirable feature, accepting pull requests on externalservices, and mirroring them as submissions to the master repositories onforge.python.org is a more complex problem, and would likely not be includedas part of the initial rollout of the forge.python.org service.
Kallithea’s native support for both Git and Mercurial offers an opportunityto make it relatively straightforward for developers to use the clientof their choice to interact with repositories hosted on forge.python.org.
This transparent interoperability doesnot exist yet, but running our ownmulti-VCS repository hosting service provides the opportunity to make thiscapability a reality, rather than passively waiting for a proprietaryprovider to deign to provide a feature that likely isn’t in their commercialinterest. There’s a significant misalignment of incentives between opensource communities and commercial providers in this particular area, as eventhough offering VCS client choice can significantly reduce community frictionby eliminating the need for projects to make autocratic decisions that forceparticular tooling choices on potential contributors, top down enforcementof tool selection (regardless of developer preference) is currently stillthe norm in the corporate and other organisational environments that produceGitHub and Atlassian’s paying customers.
Prior to acceptance, in the absence of transparent interoperability, this PEPshould propose specific recommendations for inclusion in the CPythondeveloper’s guide section forgit users for creatingpull requests against forge.python.org hosted Mercurial repositories.
[TODO: Update this section for Brett’s revised timeline, which aims to havea CPython demo repository online by October 31st, to get a better indicationoffuture capabilities once CPython itself migrates over to the newsystem, rather than just the support repos]
This proposal is part of Brett Cannon’scurrent evaluationof improvement proposals for various aspects of the CPython developmentworkflow. Key dates in that timeline are:
If this proposal is selected for further development, it is proposed to startwith the rollout of the following pilot deployment:
The following items would be needed for a production migration, but theredoesn’t appear to be an obvious way to trial an updated implementation aspart of the pilot:
The following items would be objectives of the overall workflow improvementprocess, but are considered “desirable, but not essential” for the initialadoption of the new service in September (if this proposal is the oneselected and the proposed pilot deployment is successful):
The workflow requirements for the main CPython development repository aresignificantly more complex than those for the repositories being discussedin this PEP. These concerns are covered in more detail inPEP 462.
Given Guido’s recommendation to replace Rietveld with a more activelymaintained code review system, my current plan is to rewrite that PEP touse Kallithea as the proposed glue layer, with enhanced Kallithea pullrequests eventually replacing the current practice of uploading patche filesdirectly to the issue tracker.
I’ve also started working with Pierre Yves-David on acustom Mercurialextensionthat automates some aspects of the CPython core development workflow.
This document has been placed in the public domain.
Source:https://github.com/python/peps/blob/main/peps/pep-0474.rst
Last modified:2025-02-01 08:59:27 GMT