Syzygy Bases

From Chessprogramming wiki

Home *Knowledge *Endgame Tablebases * Syzygy Bases

Syzygy - Three Planets Dance OverLa Silla^[1]

Syzygy Bases,^[2]
a compact six piece endgame database developed byRonald de Man, published on April 01, 2013. Since August 2018,seven piece Syzygy Bases are available after an effort byBojun Guo started in March 2018^[3]. The generator is released under theGNU General Public License Version 2, thethread safe probing code is released without restrictions^[4].

Data

File Types

Syzygy Bases consist of two sets of files,WDL files (extension .rtbw) storing win/draw/loss information considering thefifty-move rule for access duringsearch, andDTZ files (extension .rtbz) withdistance-to-zero information for access at theroot. WDL has full data for two sides but DTZ50 omitted data of one side to save space. Each endgame has a pair of those types.

File Sizes

Men	WDL	DTZ	Total
3-5	378.1MiB	561.9 MiB	939.0 MiB
6	67.8GiB	81.4 GiB	149.2 GiB
7	8.5TiB	8.3 TiB	16.7 TiB

Comparision

Syzygy EGTB is significantly smaller than any existentDTM EGTB. It is 7 times as small as Gaviota for 5 men, 8 times as small as Nalimov for 6 men, and 8 times as small as Lomonosov for 7 men. However, when all DTM EGTBs have full data from two sides, Syzygy EGTB omits data from one side for DTZ data to save space.Ronald de Man estimated if we keep them all, Syzygy's 6 men size may increase by 158 GB, become 307 GB in total, double in size, and be 4 times as small as Nalimov's 6 men.

Generation

Up to 6-man

On the first release (Apr 01, 2013) the generator was ready to generate all endgames up to 6 men. It is multithreading and processes completely in RAM. Generating all 6 men requires a system with at least 32 GB of RAM and may run in 5 days (the period was measured with a computer 6-core i3930K @ 4.2Ghz, 64 GB).

7-man

Ronald de Man wasn't initially interested in the creation of 7-men Syzygy Based^[5] since the generation would require about 1 TB of RAM, too expensive at that time. Generation time would be about 64 x per table, which means around 175 x time total^[6]. His original generator could not create them. But in 2018, he supportedBojun Guo in his 5-month attempt to generated them^[7]. His hardware was estimated at over US$ 90K. In August of 2018 their creation was completed^[8]^[9].

8-man

After the completion of the 7-man, some people were curious about the feasibility of building the 8-man. Ronald de Man estimated that without modifying much the generator, the task requires computers with 64 TB RAM and 2000 TB hard disks^[10] (cost about $640K and $40K respectively in 2020^[11]). The generator can be modified to work on much cheaper computers with 64 GB RAM but that may need a few thousand years of computing^[12].

Checksums

Syzygy endgame files may contain 128-bit checksum keys at the end of those files. It also has its own code for checksums (based on Google's cityhash library).

Search

During the Search

During thesearch, with the WDL tables stored onSSD^[13], it is possible to probe the tables at alldepths without much slowdown. They have been tested in Ronald de Man's engineSjaak (playing onFICS as TrojanKnight(C)) for a couple of months quite successfully, don't probe inquiescence search.

At the Root

At theroot, since pure DTZ50-optimal play (i.e. minimaxing the number of moves to the nextcapture orpawn move by either side) can be very unnatural, it might be desirable to let the engine search on the winning moves until it becomes clear that insufficient progress is being made and only then switch to DTZ-optimal play (e.g. by detectingrepetitions and monitoring thehalfmove clock)^[14].

Pros & Cons

Pros

Small sizes. It is about 8 times as small as the second-best EGTBs. Having small sizes is the main success key of Syzygy Bases since it is much easier to create, store, provide, and download than other EGTBs
Free and more popular (than other EGTBs) to find on the Internet
SupportDTZ50 metric. That metric can help engines to have better results thanDTM which is supported widely by other EGTBs

Cons

Hard to integrate with chess engines.Ronald de Man has not provided probing code as an independent library but as a part ofStockfish chess engines. The probe code has integrated too deeply with that chess engine code and it requires a lot of effort to de-integrate, modify and integrate with other chess projects
Hard to understand and contribute to the project. ETGB itself is a hard topic. Syzygy EGTB has also integrated many advanced techniques/tricks. It is written in old-style C language. All make it very hard to understand and/or modify to improve or for other purposes
DTZ50 metric may lead the engines to win in much longer and ugly ways, compared withDTM one

Data publish

Ronald de Man has provided only open-source code for generators/probers but not endgame files themselves. Using his tools some people have generated endgames and published them via DVD or online.

DVD

As of February 2015, all 3-5 and most important 6-men Syzygy Bases are commercially available on 4DVDs byChessBase asEndgame Turbo 4 for their productsDeep Fritz 14,Komodo Chess 8,Houdini 4 orChessBase 12/13^[15]^[16].

Free Download

Some free FTP servers, such asBojun Guo andLichess servers, allow downloading.

3-6 men

The sizes of those men are small enough to download and store on modern computers. Users should download them in full sets (3, 4, 5, 6 men).

7 men

All 7 men files' size is over 16.7 TiB, over the storage of typical modern computers. They also require a long time to download. Thus some users choose to download one or a few endgames only, based on their statistics of use in endgames. Below are the top 20 of those endgames by their order. The first one, KRPPvKRP, has a significantly higher frequency of use than the others and should always be downloaded:

Rank	Name	Rank	Name	Rank	Name	Rank	Name
1	KRPPvKRP	6	KNPPvKNP	11	KBPPvKNP	16	KRBPPvKR
2	KBPPvKBP	7	KNPPvKBP	12	KRPPvKRB	17	KBPPvKPP
3	KPPPvKPP	8	KRBPvKRP	13	KRPPvKPP	18	KRPPPvKP
4	KRPPPvKR	9	KQPPvKPP	14	KBPPvKRP	19	KRBPvKRB
5	KQPPvKQP	10	KQPPPvKP	15	KRNPvKRP	20	KRPPvKRN

Probe Code and Tools

Stockfish

Ronald de Man did not provide the probe code as an independent library. Instead, he published it firstly as an already integrated code forStockfish chess engines. It is C++ code and it has been rewritten and updated several times by theStockfish team.

Fathom

Fathom is a stand-alone Syzygy based probing tool andAPI byBasil Falcinelli, introduced in November 2015 along with hisGull 3 release^[17] . Unlike the original tbprobe code, Fathom does not necessarily require the callee to providemove generation functionality. The new modifications and extensions to Ronald de Man's original code which can be "redistributed and/or modified without restrictions", are released under the permissiveMIT License. The API consists of three functions^[18] :

tb_init initializes the tablebase
tb_probe_wdl probes the Win-Draw-Loss (WDL) table for a given position
tb_probe_root probes the Distance-To-Zero (DTZ) table for the given position.

Jon Dart has a fork of Fathom with some bug fixes and enhancements^[19], also supporting 7-man^[20].

Pyrrhic

Pyrrhic is a cleaned up Fathom byAndrew Grant, introduced in August 2020^[21]^[22].

Elo gained by Stockfish

Stockfish with ClassicalEvaluation

Fishtest team revealed a test withStockfish (SF10dev) at 10+0.1, with all Syzygy WDL files on RAM, testing using none(0), 4, 5, and 6 men TB in a round-robin tournament^[23].

Rank	Name	Elo	+/-	Games	Score	Draws
1	syzygy6	13	2	82591	51.8%	59.5%
2	syzygy5	2	2	82590	50.3%	59.4%
3	syzygy4	-7	2	82591	49.0%	59.3%
4	syzygy0	-7	2	82592	48.9%	59.4%

Stockfish NNUE

Elo gains by Stockfish NNUE with Sygyzy 6 men have decreased^[24]. With version 15, it gains only 2.7 Elo.

Support other chess variants

With the request by Yunus Emre Ayhan on September 2024,Ronald de Man modified Syzygy to support Shatranj, a chess variant. Using that code,Bojun Guo has generated all 6-man EGTB^[25].

Quotes

byRonald de Man in a reply toGuy Haworth, April 06, 2013^[26] :

I create both WDL and DTZ in one go, so I don't use WDL in the creation of DTZ. The algorithm used is thegrandfather algorithm with 2 plies per iteration (I thinkHGM calls this leapfrogging, but I might be wrong). I tried the outcounting method, but it didn't seem to be competitive (and it makes things more complicated).^[27]^[28]A pure WDL/DTZ pair is not of much use for creating WDL50+/DTZ50+. I create tables in RAM that have all the information necessary for WDL50+ and DTZ50+, then permute them to different indexing schemes and compress them. I do test runs on subsets of the data to find good permutations. (The idea to try permutations is fromJesper Torp Kristensen's master thesis.)^[29]^[30]

Ronald de Man explained some advantages of Syzyzy overNalimov Tablebases, July 06, 2019^[31] :

They are smaller, so more is cached in RAM (via the OS's system/page cache), and they are compressed in small blocks which allows decompression on the fly (so no need to use RAM for caching large decompressed blocks). This makes them much faster on the same hardware, which allows probing much closer to the leaves. They also initialise a lot faster (initialising 6-men Nalimov on startup of the engine seems to have taken about half an hour in the past, probably it is faster on modern hardware but still).In addition, 7-man syzygy TBs are now available. With SSDs becoming cheaper and cheaper, having 7-men WDL TBs on SSD starts to become a practical possibility (unfortunately RAM prices are lagging behind).

Ronald de Man in a reply toNguyen Pham, April 15, 2020^[32] :

Syzygy WDL is double-sided, DTZ is single-sided.So to know whether a 7-piece position is winning, losing or drawn (or cursed), the engine needs to do only a single probe of a 7-piece WDL table. (It may in addition have to do some probes of 6-piece WDL tables if any direct captures are available.)If the engine needs to know the DTZ value (which is only necessary when a TB root position has been reached), the probing code may have to do a 1-ply search to get to the "right" side of the DTZ table.For 6-piece TBs, DTZ is 81.9GB when storing only the smaller side of each table. Storing both sides might require perhaps 240GB.

Morgan Houppin explained why his chess engineStash doesn't have Syzygy probe code, March 25, 2021^[33] :

Syzygy probing is a hell of a mess, and I don't want to plug two thousand lines of foreign code that I don't understand, nor do I have the motivation to fully understand how Syzygy files are stored, and then write the additional two thousand lines of code to read them for a mere 5 Elo gain at TCEC.

Endgame News

In his 2014Chess Endgame News inICGA Journal, Vol. 37, No. 2^[34] ,Guy Haworth classified Syzygy Bases asnew data inthree ways:

5-valued scale for evaluating positions in the context of theFIDE50-move rule (50mr) which constrains the length of phases of play
1. +2 ≡ unconditional win for the side to move
2. +1 ≡ ‘win’ which can be frustrated by best play and a 50mr draw-claim
3. _0 ≡ unconditional draw
4. -1 ≡ ‘loss’ saved by a 50mr draw-claim
5. -2 ≡ unconditional loss
depths for ‘50mr draw’ positions with value ±1
depths in symmetric, information-preserving ply ‘p’

and further gives some news about early software bugs and glitches concerningChessBase products, and the importance ofMD5 to check the EGT integrity.