When it comes to building a website, SEO is pretty important. You want to make sure that when people search for things, your website is relevant enough to be displayed to the user. SEO really is pretty magical, but there's a few things we can add to help out.

I'm going to talk about generating sitemaps for yourLucky app, though, most of this could apply to other frameworks in crystal too.

Sitemaps

Sitemaps are XML documents that live in the public root of your website. These allow search engines to map and understand the entire structure of your website. You could let the search engines just use crawlers to go page by page, but that could take a lot of time, and potentially miss some pages. With a XML sitemap, the search engine knows of all the URLs to the pages, and even some other relevant metadata about those pages. It can go directly to the page, save it, and reference it later.

There's actually a standard format used for these documents. You can check outSitemaps.org for more information on how these are structured, and what other options are available. The one issue with this is that this standard has a lot more information than what their site shows. A lot more information can be found onGoogle Webmaster. This includes how to use videos and images in your sitemaps.

The first thing we need to do is include theSitemapper shard in to our lucky project.

dependencies:lucky:github:luckyframework/luckybranch:mastersitemapper:github:jwoertink/sitemapperversion:~>0.2

Once that's in and you run yourshards install, you'll be ready!

When generating sitemaps, this is a task that happens outside the normal operation of your site. It doesn't require user interaction for your sitemaps to be generated; this is an administrative task. You'll also have new pages added and/or removed all the time, so doing this statically isn't really going to be feasible for most of you. We will want to useLucky Tasks.

Let's create a new task calledgenerate_sitemaps.cr. This file will live inyour_app_root/tasks/generate_sitemaps.cr.

Now you will put this in that file. This is the bare minimum for a task (along with the sitemapper)

require"lucky_cli"require"sitemapper"classGenerateSitemaps<LuckyCli::Taskbanner"Generates the sitemaps"defcallendend

Before getting crazy, make sure you can run this and everything compiles. From our app directory, if we just runlucky generate_sitemaps, it will do nothing! We want nothing at this point. Anything else like compile errors, or whatever, will allow us to catch things before we start getting a ton of actual code in here.

Next step will be to start writing your code. The nice thing about Sitemapper is that if you need to generate a bunch of different sitemaps for different sites, like say a mutli-tenant type site, then you have that ability. I'll start with a single domain, then show a quick example of how you could do multiple.

Sitemap setup

We will be adding our sitemap code to thecall method in our task.

defcallsitemaps=Sitemapper.build(host:"https://mycoolapp.io",max_urls:500,use_index:true)doadd("/",lastmod:Time.now,priority:1.0)endend

Ok, since there's quite a bit going on here, let me break this down.

1. host - this option is to specify your domain. In the sitemaps, all of the URLs are absolute which means your root path will look likehttps://mycoolapp.io/
2. max_urls - According to the official spec, your sitemap "must have no more than 50,000 URLs and must be no larger than 50MB". By default, Sitemapper sets you to 500 per sitemap, but you can make this higher.
3. use_index - This is false by default, but if you know you have more thanmax_urls pages, then you need to set this totrue.

About the index

It's pretty common for a site to have a ton of pages when you consider 1 page for each user you have, or whatever your site is about. Assuming you have user profiles, and you have 20,000 users, then you're going to have over 20,000 pages including your home page, privacy policy and terms pages, an about page maybe? Since this information could easily get huge in size, we have the ability to generate multiple sitemaps for the same domain. They would look likemycoolapp.io/sitemap1.xml,mycoolapp.io/sitemap2.xml, etc... This lets the search engines see that you have a lot of pages, all while keeping the size of each one low. However, the search engines don't know what these sitemaps are called. The only one that they look for is/sitemap.xml. So what we're doing in this case is using our/sitemap.xml file as an index of mini sitemaps that point to where the other files are located. You can read more onsitemap indexes here.

Back to the code

Next we had this lineadd("/", lastmod: Time.now, priority: 1.0). This is adding in the root path of our site (our home page). Then we tell the sitemap when we last modified the page usinglastmod. This is aTime instance. Lastly ispriority. This is a float between 0.0 and 1.0 that shows the priority relative to the other URLs in the sitemap. Thisadd method takes any option that's available in the sitemap protocol, but all of the required options are built in for you.

Let's take a look at our user profiles:

defcall# Select all users with a public profile, and order by their ID in an ASC orderusers=User::BaseQuery.new.profile_public(true).id.asc_ordersitemaps=Sitemapper.build(host:"https://mycoolapp.io",max_urls:500,use_index:true)doadd("/",lastmod:Time.now,priority:1.0)users.eachdo|user|# we could use a messy "/profiles/#{user.username}", but this is nicerpath=Users::Show.path(slug:user.username)add(path,lastmod:user.updated_at,priority:0.6)endend# finalize the sitemaps hereend

At this point you should have enough info to start adding some more pages, but I'd like to take a look at one more set. Let's assume that our users can upload some videos to their profile page, and these videos are paginated. This adds some complexity, but not horrible.

users.eachdo|user|user_path=Users::Show.path(slug:user.username)add(user_path,lastmod:user.updated_at,priority:0.6)# get the videos for this user with most recent upload firstvideos=Video::BaseQuery.new.by_user(user.id).uploaded_at.desc_order# Add pagination for /profiles/jeremy?page=XXtotal_rows=videos.countlimit=12# 12 videos per pagetotal_pages=(total_rows/limit.to_f).ceil# skip page one because we added that already(2..total_pages).to_a.eachdo|page|add("#{user_path}?page=#{page}",lastmod:user.updated_at,priority:0.4)end# Now we add the individual video pagesvideos.eachdo|video|# There's a ton of options heremap=Sitemapper::VideoMap.new(thumbnail_loc:video.thumbnail_url,title:video.title,description:video.description,content_loc:video.source_url,publication_date:video.released_at)video_path=Videos::Show.path(user_slug:user.username,video_slug:video.cached_slug)add(video_path,lastmod:video.updated_at,priority:0.7)endend

NOTE: If you see quotes around the options in the code above, that's an error in Dev.to

You can read up on more video options looking at thesource, or onGoogle.

That's about it. The rest will really depend on your site. The only thing left I wanted to show was, a small example of using a multi-tenant site.

defcallsites=Site::BaseQuery.new.active(true)sites.eachdo|site|sitemaps=Sitemapper.build(host:"https://#{site.host}",max_urls:500,use_index:true)do# add site specific routes hereend# finalize the sitemaps hereendend

Finalizing the sitemaps

Sitemapper currently shoves all of the generated XML in to giant strings in memory. This might not be "optimal", but my app generates over 400 sitemaps, and it's been fine so far...

The reason for this is that not everyone can just write an XML file to their public folder and be done with it. If you're hosted onHeroku, or you're usingDocker, you may want to use something likeS3 or whatever to store the actual sitemaps. If you can just write the XML locally, then you can use the built-in function to do that.

# this is your_app_root_path/public/sitemaps# it will generate the folder for you if it needs toSitemapper.store(sitemaps,"public/sitemaps/")

Eventually there should be some additional options for storing to S3 or whatever, but if you need that, you'll need to take thesitemaps variable, and send that data wherever you need.