Phabricator is an open-source software development platform. In Wikimedia, Phabricator is used for project management, software bug reporting, and feature requests. Seemw:Phabricator for more details on end user usage.

phabricator.wikimedia.org runs onphab1004 ineqiad.

The Phabricator install relies on db1250 sincephab:T398818 (m3 eqiad master), with several replicas. Databases access is routed through dbproxy1003, a.k.a. m3-master.

A disaster recovery plan for phabricator.wikimedia.org is atPhabricator/Disaster Recovery.

Metrics are onhttps://grafana.wikimedia.org/d/000000587/phabricator.

Since 2023-08-23, we actually use the Phorge fork of Phabricator (T333885), but we have not (yet?) started to update references to the old software name.

The operations specific projects on Phabricator^[1] include:

• User requests hardware viaOperations requests#Hardware Requests
• Rob reviews thehardware-requests project for all tasks that are assigned to him, or unassigned.

• Tasks assigned to others are not reviewed as often, as they are awaiting input from the assignee. If they are left neglected by the assignee long term, they will likely be rejected, or have thehardware-requests project removed from the task.

• If the system specification meets an on-site spare, system allocation may proceed.

• This allocation step is typically processed by Rob and approved by Mark. (It involves a general overview of the roadmap and system procurement planning.)

• If the system specifications require an order of hardware, the following occurs:

• A RT procurement queue ticket is created for each set of vendor quotes.

• Example: A caching system at this time could be Dell or HP, we create two RT tickets. One for each vendor to provide quotes for the system specification in question.

• Quotes are generated and reviewed by Rob, Mark, and the requestors for the hardware.
• Quotes are approved for purchase by Mark/Damon/Lila (escalation dependent on overall cost) and are typically placed by Rob (for US ordering) or Mark (for EU ordering).

• The hardware-requests task will have the system details noted (hostname/asset tag) and the task will be linked to the system setup task.

• These are kept separate for easy future search history on hardware allocations; thus its nice to leave a task with the hardware-request in said project.

• A new phabricator task is created in theoperations project.

• This task is the primary tracking task for the setup and deployment of the server.

• Task should include the following (base template):
• System Deployment Steps:

   [] - mgmt dns entries created/updated (both asset tag & hostname) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]   [] - system bios and mgmt setup and tested [link sub-task for on-site work here, sub-task should include the ops-datacenter project]   [] - network switch setup (port description & vlan) [link sub-task for network configuration here, sub-task should include the network project]   [] - production dns entries created/updated (just hostname, no asset tag entry) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]   [] - install-server module updated (dhcp and netboot/partitioning) [done via this task when on-site subtasks complete]   [] - install OS (note jessie or trusty) [done via this task when network sub-task(s) complete]   [] - service implementation [done via this task post puppet acceptance]

• The main task is basically for all the software setup, and the sub-tasks are for the specific on-site or networking tasks.

• Many times, the network task isn't created, as the person doing the software work can also do the network configuration.

• User (usually an SRE, unless you really know what you are doing) requests hardware viaOperations_requests#Virtual_Machine_Requests_.28Production.29
• SRE reviews thevm-requests project the task for obvious mistakes for all tasks that are tagged.

• Tasks assigned to others are not reviewed as often, as they are awaiting input from the assignee. If they are left neglected by the assignee long term, they will likely be rejected, or have thevm-requests project removed from the task.
• If you are reading this as the SRE that is reviewing the request or are evaluating your own request the docs for what to look for are at:Ganeti#Verify cluster resource availability

• If the system specifications meet all requirements for approval/allocation of a production virtual machine, the machine can be created. The creation should be undertaken by the SRE that filed the request to increase familiarity with the platform.

• All Phabricator documentation refers to scripts in the phabricator bin directory. On our setup, that is:/srv/phab/phabricator/bin/

Write the entire contents of phabricator's databases to disk, compressed:

cd /srv/phab/phabricatorsudo ./bin/storage dump --output /srv/dumps/phabricator_db_$(date +%Y%m%d%H%M%S).sql.gz --compress

First you need the repo's callsign. This is an all-uppercase identifier with 'r' prefixed that is used in urls and such in Phabricator for the repo. For example, Puppet's isOPUP. First SSH to phab100N. Then:

cd /srv/phab/phabricatorsudo ./bin/remove destroy rFOO

First you need the file's ID prefixed with 'F'. First SSH to phab100N. Then:

cd /srv/phab/phabricatorsudo ./bin/remove destroy Fxxxxxxxx

Members of the#acl*userdisable Phabricator project can ban a user viahttps://phab-ban.toolforge.org/

This is not recommended if the account has already been active! Deleting a user can be needed when a user entered a wrong email address in the registration form and now cannot verify their address to finish account creation. First SSH to phab100N. Then:

cd /srv/phab/phabricatorsudo ./bin/remove destroy @AccountNameOfThatUser

In case of a Legal request to delete data associated with a user account,

• SSH to phab100N and remove their email address from the database via
  UPDATE phabricator_user.user_email SET address = 'deleted-user-1234@example.com', dateModified = UNIX_TIMESTAMP() WHERE userPHID = 'PHID-USER-....';
  (useConduit to get their User PHID)
• Rename the user account to something like `DeletedUser1234` (number could be their user ID),
• Disable the user account,
• Potentiallyrestrict view access to objects created by the user if object data contains PII.

Phabricator has no built-in concept of Ghost Users like GitLab.

• Please note that removal of 2FA is a serious request, and all too easily socially engineered. All requests of this nature should be treated with the same degree of security and confirmation as ssh key changes. Theuser guidelines require one month between the paste of the user committed identity hash on the wiki user page and the reset request, or verification via a video call.
• When copying the text phrase from a Phabricator Paste, make sure to useView Raw File and save the file, to avoid issues with line breaks via copy&paste. (Potentially also check with a hex editor that no additional byte such as 0x0A has been added.) Afterwards, runcat file | sha512sum (or whatever algorithm was used, e.g. could also besha3sum -a 512 or such).
• Once confirmed, the actual command is quite simple, run on the phabricator host:

  sudo /srv/phab/phabricator/bin/auth strip --all-types --user <username>

• You will be prompted with a yes or no to remove the multi-authentication types on the user.

Users can do this themselves with the big red "Terminate Tokens" button in Settings > Conduit API Tokens. If it needs to be forced for some reason, you can do it from a phabricator server:

ssh phab1004.eqiad.wmnet    sudo /srv/phab/phabricator/bin/auth revoke --type conduit --from @<username>

This invalidates any active sessions and forces the user to log in again.

ssh phab1004    sudo /srv/phab/phabricator/bin/auth revoke --type session --from @<username>

This invalidates any authorized ssh keys that the user has configured in phabricator.

ssh phab1004    sudo /srv/phab/phabricator/bin/auth revoke --type ssh --from @<username>

Warning: This takes a really long time, probably more than 8 hours. Service will be online during the reindex, however, search quality will be degraded.

ssh phab1004   sudo /srv/phab/phabricator/bin/search init   sudo /srv/phab/phabricator/bin/search index --all --force --background

Caution: This removes most of the user's activity from Phabricator and it is adestructive operation. This should only be done when cleaning up vandalism from an account which has no legitimate activity. If the account had real contributions prior to being compromised, then another solution is needed to avoid deleting the legitimate contributions along with the spam.

This procedure will attempt to undo all edits made by a given user. If you add the--delete argument it will also remove all traces of the corresponding transactions from the phabricator activity log. This should be successful in all cases except for 1 limitation: Any field which has been edited by someone after the vandal's edit will be treated as an edit conflict and the field will be left alone to avoid potentially overwriting useful edits by other users.

How it works: Therollback script simply replays the edit transactions in reverse, from newest to oldest. Each transaction in Phabricator stores the field name, the old value and the new value. To revert a user's activity, the script will do is as follows:

1. For each task edited by the vandal user:
   1. For each transaction made by the vandal user (newest to oldest):
      1. If the transaction's "new" value matches the field's current value, then the transaction's "old" value is applied to the field.
   2. After all transactions have been replayed, if any field was changed then the record is saved back to the database.
   3. Finally, if--delete was also specified, then all the replayed transactions are also deleted to clean up the history of activity.

ssh phab1004    sudo /srv/phab/libext/misc/bin/rollback execute --delete --user <username>

There is no such script anymore as it led to database corruption; seephab:T342275. Thus this is manual work now.

First set up a bulk job in phabricator's GUI, then get the bulk job id and run the make-silent command below, specifying your bulk job id. Finally, start the job in the GUI and it will run without sending notifications.

ssh phab1004    sudo /srv/phab/phabricator/bin/bulk make-silent  --id <bulkid>

See alsomw:Phabricator/Help#Batch edits for more information and guidance.

To put phabricator into read-only mode, which allows it to continue serving requests during a master database restart, do the following on the active phabricator server:

ssh phab1004    sudo /srv/phab/phabricator/bin/config set cluster.read-only true    # restart database server    sudo /srv/phab/phabricator/bin/config set cluster.read-only false

Herald rules can be disabled via

ssh phab1004    sudo /srv/phab/phabricator/bin/herald rule --disable --rule<rulenumber>

To check if a Phabricator user is who they say they are there is a script to get their email address and whether it's verified from the SQL database:

chk_phuser <Phabricator username>

ssh phab1004    sudo chk_phuser Dzahn

ssh phab1004    sudo /srv/phab/phabricator/bin/policy unlock --edit YourPhabUserName T12345678

First get the internal PHID of the object to unlock, for example viaConduit by passing{"ids":[12345678]} asconstraints.

ssh phab1004    sudo /srv/phab/phabricator/bin/policy unlock --edit YourPhabUserName PHIDofObject

SeePhabricator/Mail debugging.

There is a test instance athttps://phabricator.wmcloud.org/ running inCloudVPS in thedevtools project.

It is mentioned to users athttps://www.mediawiki.org/wiki/Phabricator#Get_started

As of today, October 2025, the full instance name isphabricator-bullseye.devtools.eqiad1.wikimedia.cloud.

To get shell access you will have to request project membership indevtools. Then seeHelp:Accessing Cloud VPS instances how to ssh to it via a bastion host.

The test instance has no SLA and may be down for maintenance at any time.

Sometimes users request access to the Phabricator test instance but due toT388022 the test instance can not send email. Therefore users need to ask an existing shell admin to manually verify their email address to get access. Users are told atW1 to create a ticket for the collab-services team. (example:T406698)

How to:

- ssh to the instance:

ssh phabricator-bullseye.devtools(assuming an appropriate ssh client config, see section above)

- if you don't know the email address and only have the user name, get it from the database:

sudo mysql -u rootMariaDB [(none)]> use phabricator_user;MariaDB [phabricator_user]> SELECT userName,address,userPHID from user_email INNER JOIN user on user_email.userPHID=user.phid and userName LIKE "<USERNAME>";

- once you have the email address:

cd /srv/phab/phabricator/binsudo ./auth verify <email address>

SeePhabricator/Ban IP address

Access to Phabricator is restricted by rate limiting rules inrequestctl. This rate limiting was enabled in May 2024 due to a high level of scraping and abusive traffic (seeT362401). Users affected by the rate limiting will see a "http 429 - too many requests" temporarily.

As of 09/2025 there are two requestctl actions:

• action/cache-text/phabricator_throttle_clouds to throttle clouds
• action/cache-text/phabricator_block_scrapers to block smaller cloud providers

Normal traffic from legitimate users shouldn't be affected in most cases. To avoid triggering the rate limit, the following can help:

• Keep the number of requests per second low, especially when using the API, scripts, or curl.
• Use a unique, non-shared IP address (avoid cloud networks, VPNs or proxies).
• Set a proper, application-specific User-Agent header.
• Try again later once the limit resets.

SREs can adjust the rate limiting settings in private-puppet/requestctl. The relevant configuration for Phabricator can be found undercache-text/phabricator*.

Phabricator is currently hosted on phab1004.eqiad.wmnet / phab2002.codfw.wmnet.

The full path of traffic from the public internet through to the database is as follows:

cache_text esams -> cache_text codfw -> cache_text eqiad -> phab1004 -> dbproxy1003 -> db1043

Some Phabricator applications throwing exceptions likeFailed to load class or interface "Phabricator*" - this can sometimes be resolved by runningarc liberate inside of/srv/phab/phabricator which will update the library map as inthis commit.

Phabricator Dashboard on Grafana

Check the logs on/var/log/apache2/phabricator_error.log (or inLogstash applicaion logs andLogstash apache logs for a more readable format)

Check the host in Icinga for more failed checks (eg. PHD should be running).

Check the status of the phd process (sudo systemctl status phd).

Do not run aphlict server using websockets and proxy through Apache also running main Phabricator.

SeePhabricator/Slowness for more info.

If a Phabricator server had to be rebooted for any reason you might get Icinga pybal alerts. Pybal will alert that backends for the git-ssh.wikimedia.org service are down but pooled. Example:

<+icinga-wm> PROBLEM - PyBal backends health check on lvs1018 is CRITICAL: PYBAL CRITICAL - CRITICAL - git-ssh6_22: Servers phab1001-vcs.eqiad.wmnet are marked down but pooled

The reason for this is a race condition where the ssh-phab service is started before the additional IPv6 IP is added to the interface. The service WILL be running so it might not be obvious why it's considered down. This is because it will be listening only on IPv4 while pybal / lvs servers are going to try and use IPv6.

The server has multiple IPs, 2 on loopback for LVS and 4 on en01.

The fix in this case is to manually restart the service with

systemctl restart ssh-phab

You'll have to wait a few minutes and then the pybal Icinga alerts will recover.

cat <phabricator_root>phabricator/conf/local/local.json | python -m json.tool

or

<phabricator_root>phabricator/bin/config list

All possible values can be seen on a current install:

https://<fqdn>/config/all/

A simple failure of the phabricator server, e.g. a disk failure or other hardware failure onphab1001.

Take a look at a previous fail-over ticket atT238956.

Code changes needed for the actual fail-over can be seen at thetopic branch phab-buster. Decommissioning of the previous server can be seed at thetopic branch phab1003-decom.

Additionally theetherpad Phabricator-migration-20191203 was used.

If there are 2 existing servers, just follow the steps. If the existing prod server died, assume "old_server" means the warm standby in the other data center. If the standby server died see the section below.

1. install a new server and add the role::phabricator puppet class on it, run puppet agent
2. rsync /srv/repos from old_server to new_server, run it with --delete as well and ensure both sides have the same size. (rsyncd / ferm rules for this are already puppetized on all servers)
3. verify code in /srv/phab is up to date and both servers are on the same git tag (if not use scap to deploy to new server / run 'scap pull' on it)
4. switch the "phabricator dumps host" to the new server.code change
5. (optional) put phab on new_server in maintenance mode (phab admin action)
6. set downtimes for both servers in Icinga
7. change the "phabricator_server" setting to the new server name.code change
8. (changing the "active server" setting is not needed anymore, setup has been simplified)
9. switch the discovery record in DNS to the new server. The TTL is 300 seconds by default for all discovery records. It does not need to be changed but be aware there might be a 5 minute window where clients could get the old server.code change
10. switch the config for varnish to the new servercode change
11. switch the mail destination on mx to the new servercode change
12. using systemctl, restart the "ssh-phab" service on the new server to make it listen on IPv6
13. usingconftool, depool the "vcs" service on the old server, change conftool data to use the new servercode change and pool it
14. (if reimage script failed in the past and you have ongoing Icinga alerts about pybal and the vcs server): delete stale confd files on puppetmaster to clear Icinga alerts about confd template compilation failing
15. make the "phd" service run on the new server to avoid breakage of reposcode change
16. verify things work and remove Icinga downtimes
17. (a few days later) decom the old server following the usual decom steps and as outlined in the phab1003-decom branch linked above

If the non-active server died and you want to re-create it under a new host name:

1. install a new server and add the role::phabricator puppet class on it, run puppet agent
2. rsync /srv/repos from the prod server to the new_server, run it with --delete as well and ensure both sides have the same size. (rsyncd / ferm rules for this are already puppetized on all servers)
3. verify code in /srv/phab is up to date and both servers are on the same git tag (if not use scap to deploy to new server / run 'scap pull' on it)
4. Add the new host name to the list of "phabricator_servers" in Hiera in hieradata/role/common/phabricator.yaml.
5. using systemctl, restart the "ssh-phab" service on the new server to make it listen on IPv6
6. usingconftool, depool the "vcs" service on the old server, change conftool data to use the new servercode change and pool it
7. You do NOT have to worry about the phd service running, it's only needed on the active server.

Complete data center failover, e.g. some major event takes down eqiad and we need to fail over to codfw.

How to make codfw master writable

root@cumin1001:~# mysql --skip-ssl -hm3-master.codfw.wmnet

Master database fails, we need to fail over to a slave and swap the slave to become a master

If the master goes down, the proxy would automatically failover to the existing slave (which is read-only) and would need to be set up as read_only=OFF by an admin.

• T190572 - Prepare a disaster recovery plan for failing over Phabricator
• Phabricator/Disaster Recovery

Mr-Widget is the integration panel that shows GitLab changes on a Phabricator task.^[2][3]

1. ↑The Operations specific Phabricator projects were discussed inT119944 in early 2016.
2. ↑https://phabricator.wikimedia.org/T324149
3. ↑https://phabricator.wikimedia.org/T385930

• https://phabricator.wikimedia.org

• Services
• Release Engineering