Configure search engine indexing

LightNet has settings for asking search engines, such as Google, whether to include content from your site in search results.

Indexing options

LightNet supports search-indexing control at the site and page level. When you request noindex, LightNet adds this robots meta tag to the <head> of the generated HTML:

<meta name="robots" content="noindex">

See also Block Search Indexing with noindex in Google Search Central.

Indexing enabled by default

By default, LightNet allows search engines to crawl and index your site. If search engines discover your site, they may include its pages in search results.

Request site noindex

If you do not want your site to appear in search results, you can ask search engines not to index your site.

Set disallowSearchIndexing to true in your LightNet configuration:

import lightnet from "lightnet";
import { defineConfig } from "astro/config";

export default defineConfig({
  integrations: [
    lightnet({
      disallowSearchIndexing: true,
      // ...
    }),
  ],
});

This adds the noindex directive to all generated HTML output. When a search engine crawls a page, the directive tells it not to include that page in search results. The page can still be visited by people who have the link.

Request page noindex

If you do not want one page to appear in search results, you can ask search engines not to index it.

For a Markdown file, set disallowSearchIndexing in the frontmatter. For example, to request noindex for the About page:

---
layout: "lightnet/layouts/MarkdownPage.astro"
disallowSearchIndexing: true
---

# About

About this site.

For a page built with the Page component, pass disallowSearchIndexing directly:

---
import { Page } from "lightnet/components";
---

<Page disallowSearchIndexing>
  <h1>Public page</h1>
</Page>

Override site noindex for a page

To override a site-wide noindex request, set disallowSearchIndexing to false for the page. The page-level value takes precedence over the global configuration:

For a Markdown file:

---
layout: "lightnet/layouts/MarkdownPage.astro"
disallowSearchIndexing: false
---

# About

About this site.

For a page built with the Page component:

---
import { Page } from "lightnet/components";
---

<Page disallowSearchIndexing={false}>
  <h1>Public page</h1>
</Page>

Media Items

LightNet does not support requesting noindex for individual Media Items.

Do not use `robots.txt`

If your existing LightNet project still uses robots.txt, remove the file and use disallowSearchIndexing instead.

A robots.txt rule like Disallow: / blocks search engines from crawling your site. If a search engine cannot crawl generated HTML, it cannot see the noindex directive. As a result, the page may still appear in search results if the search engine already knows about it from before or finds links to it elsewhere.

See also Understand the limitations of a robots.txt file in Google Search Central.