Repository files navigation

This is a bunch of scripts, which generate Dartdocs.org contents, for every single package inpub.

It runs on Google Compute Engine instances, and designed to be run there. You still can run it on your local machine,in this case you'll need to create credentials.yaml file with the credentials, which allow access to Dartdocs projectin Google Cloud and also to the dartdocs' account on CloudFlare CDN. Check out credentials.yaml.example for the example.

Unfortunately (or, fortunately :)) you cannot do anything withhttp://www.dartdocs.org without the credentials.

But once you got them, the rest is simple (if you have Dart already installed):

git clone https://github.com/astashov/dartdocorg.gitcd dartdocorgpub get

Now, copy your credentials to credentials.yaml, and you are good to go.

It has several scripts, which you can use to control the generation of docs. All scripts support--help arg, whichwill tell you more about the supported arguments.

• bin/package_generator.dart

That's the main one, it generates documentation for the packages from pub. It downloads the list of available packagesfrom pub, installs them, generates the documentation, and uploads it to Google Cloud Storage. Doing that in batches.

It also supports "sharding". If there are several instances running in the same Compute Engine Instance Group, you canspecify the name of the group inconfig.yaml, and then they will split the work between the instances. So, if youneed to regenerate the docs for the whole pub quickly, you can run 100 instances within the same group, and it probablywill be done in several hours.

It also stores all the meta information about the packages to Google Datastore (like, the status of generation (successor error), when it was generated, etc).

It also can regenerate specific package/version, check--help for details.

• bin/index_generator.dart

Using all the meta information from Google Datastore, stored bypackage_generator.dart, it builds index pages anduploads them to Google Cloud Storage as well. Like, index.html page with all the successfully generated packages,'failures' page with links to logs of failures, 'history' page with the latest generated packages, its status andlinks to logs as well.

• bin/purge_cdn_cache.dart

It just sends a command via CloudFront API to clear all the cache on CloudFront. Useful when you regenerate the wholesite with the new version of dartdoc, for example.

• bin/bump_docs_version.dart

We have a global variable 'docsVersion', which is stored in Datastore, and also every single meta record in Datastorehas a field with that value. It's an integer, which is just a timestamp. When the package's docsVersion value is equalto the global docsVersion value, we consider it generated (with success or error). When they are not equal, the packagedocs will be regenerated again. So,bin/bump_docs_version.dart just bumps global docsVersion to the current timestamp.Which means all the packages will be stale, and will be regenerated again. That's useful when you need to regenerateall the packages with the new version of dartdocs, for example.

There are also some shell scripts, which we use to run the app

• create_google_cloud_project.sh

It configures Google Cloud project to run dartdocorg. It adds indexes to Datastore, creates the bucket, creates instancetemplates and run instances in instance groups.

• {index,package}_generator_boot.sh

This is a startup script for the index and package instances, it provisions the instances:

• Installs latest Dart SDK
• Clones and configures the app
• Creates credentials.yaml
• Configures monit to restart the script if it crashes
• Configures log rotation

• {index,package}_generator_monit.sh

This is the script, which is used by monit to start, stop and restart the Dart scripts.

It uses:

• Google Compute Engine - to runbin/package_generator.dart andbin/index_generator.dart. There are 2 instancegroups where these scripts run -dartdocs-package-generators anddartdocs-index-generators accordingly.dartdocs-index-generators is supposed to haveonly one running instance, which will rebuild the index pages(by runningbin/index_generator.dart)and also will build documentation for the packages (by runningbin/package_generator.dart).dartdocs-package-generators can have any number of running instances (even 0, since we still have one runningbin/package_generator.dart on thedartdocs-index-generators instance), depending on the work load and how fastyou want to regenerate the documentation. You can think about it as an auxiliary group, which helps you to scalegeneration when needed, in case you e.g. need to regenerate documentation for all packages.

  Both have startup scripts, which are run on an instance launch. They install Dart, clone the repo, configure monitand log rotation, and finally start thebin/ scripts. You can find templates for them in the filespackage_generator_boot.shindex_generator_boot.sh.

  To control the number of running instances per group from CLI, you can use the following commands:

$ gcloud compute instance-groups managed resize dartdocs-index-generators --size 1 --zone us-central1-f$ gcloud compute instance-groups managed resize dartdocs-package-generators --size 4 --zone us-central1-f

• Google Cloud Datastore - to store all the meta information about packages (status, updatedAt, docsVersion, etc).It also has custom composite indexes defined inindex.yaml. To load the indexes to the Datastore from CLI, usethe following command:

$ gcloud preview datastore create-indexes index.yaml

To remove old unused instances:

$ gcloud preview datastore cleanup-indexes index.yaml

• Google Cloud Storage - to store all the documentation and index packages. It's configured to serve it as a staticsite.

• CloudFront CDN - to add SSL support and the redirect fromhttp://dartdocs.org tohttp://www.dartdocs.org

• Regenerate the docs with the new version of dartdoc

The instances install the latest dev SDK on boot, (check out the{index,package}_generator_boot.sh script).So, to upgrade to the new version of dartdoc, we need to restart the instances and scaledartdocs-package-generators up.We also need to runbin/bump_docs_version.dart to updatedocsVersion and purge the CDN cache for the package docs.

So, you have to do:

$ gcloud compute instance-groups managed resize dartdocs-index-generators --size 0 --zone us-central1-f$ gcloud compute instance-groups managed resize dartdocs-package-generators --size 0 --zone us-central1-f$ dart bin/bump_docs_version.dart$ gcloud compute instance-groups managed resize dartdocs-index-generators --size 1 --zone us-central1-f$ gcloud compute instance-groups managed resize dartdocs-package-generators --size 40 --zone us-central1-f... after itsdone ...$ dart bin/purge_cdn_cache.dart

So, the commandgcloud compute instance-groups managed resize dartdocs-package-generators --size 40 --zone us-central1-fsets the number of instances in dartdocs-package-generators group according to the--size argument.For now, you only can launch 100 CPU units in the google cloud project, so, limited with 50 instances.

It takes ~6-8 hours to regenerate the whole pub with 40 instances in thedartdocs-package-generators group.

It doesn't scale down automatically, you have to use the commandgcloud compute instance-groups managed resize dartdocs-package-generators --size 0 --zone us-central1-fto scale it back to zero instances once they are done with regenerating everything (remember, you'll still have oneinstance in thedartdocs-index-generators, which also generates the documentation for the packages)

• Regenerate just one specific package

Just simply run:

$ dart bin/package_generator.dart --name blah --version 1.2.3

And it will regenerate the package and override it on thehttps://www.dartdocs.org.

We have lots of versions of docs for a package, and we want search engines togive users the latest copy of the docs. We don't want an old version to appear at the topof the search results.

We achieve that by setting arel="canonical" in the HTML pages of the docs, this way we tell search engineswhat the canonical doc is for a package. The canonical url looks like/documentation/package-name/latest/index.html(note the use oflatest here)

When we generate docs for a package, we setrel="canonical" meta tag, and also we generate the HTMLfor thelatest/index.html, which contains meta refresh tag. So, when user openslatest/index.html, she'llbe redirected to the latest version of the package, like/documentation/package-name/1.2.3/index.html.