Repository files navigation

yst is a tool for generating a static website by fillingstringtemplates with data taken fromYAML orCSV text files orSQLite3 file based databases. This approachcombines the speed, security, and ease of deployment of a staticwebsite with the flexibility and maintainability of a dynamic site thatseparates presentation and data.

yst is written in Haskell. The easiest way to installyst is by usingHaskell'scabal install tool. The best way to get this tool is toinstall theHaskell platform, which includes a complete installationof theGHC compiler and thecabal executable.

Once you havecabal, you can install yst with two commands:

cabal updatecabal install yst

(Note that by default,cabal installs theyst executable intoa special directory:~/.cabal/bin on unix systems. You will needto make sure that directory is in your system path.)

If you get the error "yst-0.2.3 depends on template-haskell-2.4.0.0which failed to install," then try the following:

cabal install syb-with-class-0.6cabal install yst

To get started withyst, use the command:

yst create mysite

This will create a directorymysite and populate it with the filesneeded for a sample site. Change to this directory and runyst withno arguments to create the site:

cd mysiteyst

The site will be created in thesite directory. Open upsite/index.htmlto take a look.

The configuration fileindex.yaml tellsyst which pages to build,and from which templates and data files. Let's take a look, so we can seewhat went intoindex.html. The file is a YAML list of YAML hashes(name/value pairs). The first item is

- url      : index.html  title    : Home  template : index.st  requires : event.st  data     :    recentevents : FROM events.yaml ORDER BY date DESC LIMIT 2

This says: build the pageindex.html from the string templateindex.st(and subtemplateevent.st) and data fromevents.yaml. Sort this data(a list of events) by date in descending order, and discard all but thefirst two items. Put the resulting data in the string template attributerecentevents. Give the page the title "Home."

Now take a look atevents.yaml, the data source. Again it is a YAMLlist of YAML hashes, each item in the list representing one event.The first is:

- date: 2009-06-28  speaker:  Sam Smith  title: Building a static website

Pretty self-explanatory! Try adding a new event, then rebuild thesite by typingyst and see how it looks.

If you have problems, make sureevents.yaml is a valid YAML file.Structure is shown through indentation, so make sure things lineup right. And occasionally you may need to use quotation marksaround string values---for example, when a title contains a colon.

There's one more ingredient in our recipe---the string templates.Take a look atindex.st, the template from whichindex.htmlis built:

# WelcomeHere's our website. Have a look around.Our last two events:$recentevents:event()$For a complete list of events, see the [events](events.html) page.

The first thing to notice is that this is inmarkdown format (or,to be precise, markdown withpandoc extensions). So, for example,the first line is a level-one header, and there is a hyperlinkto the events page on the last line.

The second thing to notice is the bit between dollar signs.This is a placeholder for some formatted data. The renderedtemplate will include the listrecentevents (remember, thiswas populated fromevents.yaml after some transformations---seeabove). And each element of this list will be formatted bya subtemplate calledevent.st:

- $if(it.speaker)$$it.speaker; separator=" and "$, $endif$ *$it.title$*.

Let's break this down. The whole line will print a bulleted listitem.it here refers to the event that is being processed by thetemplate. So the first part says: if this event has aspeaker field,print the speaker, or, if thespeaker field is a list, print all thespeakers separated by the word "and", followed by a comma. And thesecond part says, print the contents of thetitle field, surroundingit with asterisks (which is the markdown way of making itemphasized).

(See thestring template documentation for details of template syntax,and examples.)

If you look atindex.html, you'll see this rough structure, but inan HTML file, not a markdown file.yst converts the rendered markdowntemplate to HTML (using pandoc), and inserts it into a "layout" filecalledlayout.html.st. If you look at this file, you'll see that it'san HTML file with blanks for$contents$ and$nav$. The$contents$blank gets filled by the rendered template, converted to HTML, andthe$nav$ blank gets filled by an HTML navigation menu (an unorderedlist with links).

To summarize our example, then:yst sorts and filters the data inevents.yaml according to our instructions; inserts this data intotheevents.st template, formatting each item using theevent.sttemplate; uses pandoc to convert the rendered template to HTML;constructs a navigation menu; and puts the contents and navigationmenu in the layout templatelayout.html.st. The result is our page,index.html.

Synopsis:

yst                    # rebuilds site, using default config.yamlyst -f myconf.yaml     # rebuilds site, using myconf.yaml as configyst create newsite     # creates a starter (demo) site in newsite directory

When run without arguments,yst looks atindex.yaml to determinethe dependencies of each page, and rebuilds only the pages whosedependencies have changed since the last build.

In order for this to work properly, you must be sure to list allsubtemplates included recursively in the main page template usingtherequires field. This field takes a single filename or aYAML list, which may be in either of two formats:

requires: [event.st, date.st]

or

requires:  - event.st  - date.st

If you don't list all the subtemplates needed to render a pageunderrequires,yst will still work, but it might sometimesfail to rebuild a page when one of these subtemplates has been changed.

The configuration file specifies the following:

• indexfile: the filename of the index file (default:index.yaml)
• title: the title of the whole site
• sourcedir: list of directories containing templates and page sources(default:.)
• datadir: list of directories containing yaml data files (default:.)
• filesdir: list of directories containing static files (default:files)
• layout: the default layout template for the site, relative tosourcedir (default:layout.html.st)
• navstyle: eithertop for a top menu orside for a sidemenu

The directories specified bysourcedir anddatadir are searched inorder to find source/template or data files, respectively. This allowsfor a../templates directory to be shared among multiple sites, forexample. Static files are merged from the contents of all directoriesinfilesdir. All of these accept a string as a singleton list.

The index file is a YAML list of pages. Each page may have the followingfields:

• url: the relative URL of the page to be built
• title: the title of the page
• template: the main string template from which the page will be built
• source: the markdown source from which the page will be built
• requires: other files changes to which should trigger a page rebuild(primarily subtemplates of the main page template)
• data: string template attributes, data sources, and transformations(see below)
• layout: a layout template to use, if other than the site default
• inmenu: if 'no', the page will not be included in the site navigationmenu

Each page must have at leasturl,title, and eithertemplate orsource. Values fortemplate,source, andlayout are relative tosourcedir specified inconfig.yaml.

The pages may be organized into a tree-like hierarchy, which will bereflected in the site navigation menu. It is easiest to see how thisworks by example:

- Rooms:  - url      : room101.html    title    : Room 101    source   : room101.txt  - url      : room203.html    title    : Room 203    source   : room203.txt

Here we have a subtree called "Rooms" with two pages under it.Subtrees can contain other subtrees. Just be consistent about indentation.

Thedata field inindex.yaml can populate any number ofstringtemplate attributes with data from YAML or CSV files or SQLite3databases. The syntax is easiest to explain by example (note that thekeywords do not have to be in ALL CAPS, although they may, and thequery doesn't have to end with a semicolon, though it may):

data:    events:  from events.yaml order by date desc group by title then location    people:  from people.csv order by birthday then lastname where              birthstate = 'CA' limit 5    beststudents: from students.sqlite               query "select * from students where grade > 5"              order by name

First we have the name of the stringtemplate attribute to be populated(say,events). Then, after the colon, we have the data source(events.yaml). If the data source is an SQLite3 database, it should befollowed by a query that is a quoted string.

The data source is followed by one or moretransformations, which willbe applied in order. Here are the possible transformations. Inwhat follows, brackets denote an optional component,| denotesalternatives, and* indicates that the component may be repeatedseveral times:

ORDER BY field [ASC|DESC] [THEN field [ASC|DESC]]*

Sorts a list by comparing the value offield.ASC(the default) means "ascending", andDESC means "descending".The keywordTHEN is used to separate fields that will becompared in order. So, if we are ordering bybirthday then lastname,we will compare birthdays, and if these are equal, we will breakthe tie by comparing last names.

GROUP BY field [THEN field]*

Converts a list into a list of lists, where each sublist containsonly items with the same value forfield. So, for example,group by date takes a list of events and produces a list oflists of items, where each sublist contains events occuring ata single date.GROUP BY date THEN venue would produce a listof lists of lists, and so on.

LIMIT n

Removes all but then top items from a list.n must be a number.

WHERE condition

Selects only items that meet a condition.

Acondition in aWHERE statement is a Boolean combination (usingNOT,AND,OR, and parentheses for disambiguation) ofbasicconditions. Abasic condition is either of the formvalue op value, wherevalue may be either a fieldname or a constant, or ofthe formHAS fieldname' wherefieldname' is a string constant. Notethat all constants must be enclosed in quotes.op may be one of thefollowing:=,>=,<=,>,< and 'contains'.

The basic conditionarg1 contains arg2 succeeds if and only ifarg1 is a fieldname whose value is a list containing the value ofarg2.

The basic conditionHAS fieldname' succeeds if and only if the item has the fieldname as a field. For example,HAS "data"'.

Note that the order of transformations is significant. You can getdifferent results if you useLIMIT before or afterORDER BY,for example.

If you want to specify an attribute's value directly, rather thanreading it from a file, just omit the "FROM":

data:  deadline: 11/20/2009

Any YAML value can be given to an attribute in this way.

Any file or subdirectory in thefiles directory (or whatever isthe value offilesdir inconfig.yaml) will be copied verbatim tothe site. So this is the place to put javascripts, css files, images,PDFs, and the like.

yst will recognize date fields in data files automatically, if thedates are in one of the following formats:

• the locale's standard date format
• MM/DD/YYYY (e.g. 04/28/1953)
• MM/DD/YY (e.g. 04/28/53)
• YYYY-MM-DD (e.g. 1953-04-28)
• DD MON YYYY (e.g. 28 Apr 1953)

Dates may be formatted in templates using a stringtemplate "format"directive. There's an example in the demo filedate.st:

$it; format="%B %d, %Y"$

The following codes may be used in format strings (taken fromHaskell'sDate.Time.Format documentation):

• %D : same as%m/%d/%y
• %F : same as%Y-%m-%d
• %x : as dateFmt locale (e.g.%m/%d/%y)
• %Y : year
• %y : last two digits of year, 00 - 99
• %C : century (being the first two digits of the year), 00 - 99
• %B : month name, long form (fst from months locale), January - December
• %b, %h : month name, short form (snd from months locale), Jan - Dec
• %m : month of year, leading 0 as needed, 01 - 12
• %d : day of month, leading 0 as needed, 01 - 31
• %e : day of month, leading space as needed, 1 - 31
• %j : day of year for Ordinal Date format, 001 - 366
• %G : year for Week Date format
• %g : last two digits of year for Week Date format, 00 - 99
• %f : century (first two digits of year) for Week Date format, 00 - 99
• %V : week for Week Date format, 01 - 53
• %u : day for Week Date format, 1 - 7
• %a : day of week, short form (snd from wDays locale), Sun - Sat
• %A : day of week, long form (fst from wDays locale), Sunday - Saturday
• %U : week number of year, where weeks start on Sunday (as sundayStartWeek), 00 - 53
• %w : day of week number, 0 (= Sunday) - 6 (= Saturday)
• %W : week number of year, where weeks start on Monday (as mondayStartWeek), 00 - 53

In some cases, a field may have one or several values. For example, anevent might occur at a date or a date range, and an article may have oneauthor or a list of authors.

An elegant way to handle these cases is to let the field take either ascalar or a list value, and use stringtemplate's "separator" directiveto format the result appropriately. So, for example, in ourevents.yamlwe have:

- date: 2009-06-28  speaker:  Sam Smith  title: Building a static website- date:  2009-04-15  speaker:    - Sam Smith    - '[Jim Jones](http://foo.bar/baz)'  title: Advantages of static websites- date:    - 2009-04-20    - 2009-04-22  title: Seminar on web security- date: 2009-04-15  speaker: Jim Jones  title:  XSS attacks for dummies

Note that thedate field is sometimes a single date, sometimes alist (with start and end date of a range), and thespeaker field issometimes a single speaker, and sometimes a range.

Here is how we handle the date ineventgroup.st:

**$first(it).date:date(); separator=" - "$**

Herefirst(it).date is the raw data, which may be a single dateor a list.first(it).date:date() is the result of formatting eachdate using thedate.st template (discussed above). Andfirst(it).date:date; separator=" - " is the result of taking thislist of formatted dates and concatenating them, separated by a hyphen.When there is just one date, we just get a date. When there are two,we get a date range.

We can use the same trick in the case ofspeaker. Ifit isan event record, thenit.speaker; separator=" and " will beeither a single speaker (if the value is not a list) or a listof speakers separated by "and" (if it is a list).

In sorting lists withorder by,yst compares two lists bycomparing the first members, then (in case of a tie) the secondmembers, and so on. If one item is a list and the other a scalar,the scalar is compared to the first item of the list. So, inthe example above,Seminar on web security will be sortedan earlier than an event with date2009-04-21, and later thanan event with date range2009-04-20 - 2009-04-21.

If you have a colon in a YAML value, be sure to enclose it in quotes,or you'll get an error. So,

title:  "Cheever: A Life"

not

title:  Cheever: A Life

Or (especially if the string is long), use> or| for a wrappedor unwrapped multiline string literal:

title: |  A very long string that  goes on and on.  You can even have blank lines,  but be sure to maintain indentation.

If you like, you can use a CSV file instead of YAML for your data source.Just give it the extension.csv. Inindex.yaml, you'd have:

data:  events: from events.csv order by date desc

This can be handy if you're using existing data, because spreadsheetsand databases can easily be dumped to CSV. In the case of a SQLdatabase, you can use a query like this to get the CSV:

SELECT * INTO OUTFILE 'result.csv'FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'LINES TERMINATED BY '\n'FROM my_table;

(Thanks tohttp://www.terminally-incoherent.com/blog/2006/07/20/dump-mysql-table-into-csv-file/.)

You can also get the data directly from a database. Just give the filename from the database followed by a query quoted in "'s

In this way you can do joins and other advanced operations on yourdata before handing them over to yst:

data:  meetings : FROM data.sqlite              QUERY "select * from meetings                left outer join persons               on meetings.speaker = persons.name"

Markdown allows raw HTML to be used, so you can embed HTML in templates.Pandoc's extended markdown is different from standard markdown in thatit parses text within HTML block elements as markdown. So, for example,you can include a section in<div> tags, or use raw<ul> and<li>tags instead of markdown formatting.

Layout files are also string templates, but they are not treated asmarkdown by default. They should use a double extension to indicatethe format. So, for example, an HTML layout could bestandard.html.st,and a LaTeX layout could beprinted.tex.st.yst will convert thepage contents appropriately for the format of the layout template.Here are the supported formats and extensions:

• HTML:.html.st,.xhtml.st
• LaTeX:.tex.st,.latex.st
• ConTeXt:.context.st
• Groff man:.1.st
• Rich text format:.rtf.st
• Texinfo:.texi.st
• DocBook:.db.st
• OpenDocument XML:.fodt.st
• Plain text (markdown):.txt.st,.markdown.st

The demo site shows how you can useyst to produce a LaTeX documentfrom the same data sources you use to produce HTML pages.

The following stringtemplate attributes are defined when layoutsare rendered:

• $contents$: the result of rendering the page and converting to the layout's format
• $nav$: an HTML navigation menu created fromindex.yaml
• $gendate$: the date the page was generated
• $sitetitle$: the site title fromconfig.yaml
• $pagetitle$: the page title as defined inindex.yaml
• $root$: the path to the website's root, relative to the page beingrendered. So, for example, if we are renderingrooms/room503.html,$root$ will have the value../. Put$root$ in front of relative URLsin your layout file, so that the links aren't broken on pages insubdirectories.

If you use only relative URLs in your site, you can preview it byopening any of the HTML files in site in your web browser. If you useabsolute links, this won't work, but you can use Jinjing Wang's simplestatic web servermaid:

cabal updatecabal install maid

To use maid to preview your site, just change to the site directory andstartmaid:

cd sitemaid

The site will appear athttp://localhost:3000. If you want to serve itat another port, just pass the port number as an argument tomaid:

maid 5999

yst's source code lives on github athttp://github.com/jgm/yst/tree/master.You can clone the repository with

git://github.com/jgm/yst.git

To install the development code once you've checked it out, just do

cabal install

(But please stick to the released version if you don't like things to breakunexpectedly!)

If you find a bug, please report it usingthe issue tracker on yst's github page.