If I use the--parserobots argument the crawler simply crawls the main domain but doesn't follow any links dispite the allow directives.

You can try it onhttps://google.com to see the behaviour.