# Allow all crawlers to access everything except private sections User-agent: * Disallow: /admin/ Disallow: /profile/ Disallow: /login Disallow: /signup Disallow: /api/ Allow: / # Facebook crawler (for social sharing previews) User-agent: facebookexternalhit Allow: / # Google News bot User-agent: Googlebot-News Allow: / # Sitemaps Sitemap: https://metbeatnews.com/sitemap.xml Sitemap: https://metbeatnews.com/news-sitemap.xml