Admin tasks and tools
Toolforge components
Packaging
Runbooks and emergency guides
Other
[edit]
This runbook explains how to debug and fix common issues withToolforge Redis.
checker.tools.wmflabs.org/toolschecker: Redis set/get is apaging alert that triggers where toolschecked is unable to talk to Redis.SSH to Redis hosts attools-redis-X.tools.eqiad1.wikimedia.cloud and check the output ofredis-cli info replication.
The Redis service is using a non-standard unit name in Systemd:redis-instance-tcp_6379.service
When this happens, the following message will be logged:ERR max number of clients reached. Check in the logs of all servers withsudo journalctl -g "max number of clients". If you find this message appearing repeatedly, restart the Systemd unit on the hosts where the message is logged:
$sudosystemctlrestartredis-instance-tcp_6379.service#wikimedia-cloud-admin is the main communication channel for Toolforge admins.
If Redis is down, you should follow theWikimedia Cloud Services team/Incident Response Process