User-agent: * Disallow: /dataset/rate/ Disallow: /revision/ Disallow: /dataset/*/history Disallow: /api/ Crawl-Delay: 20 User-agent: PetalBot Disallow: /