Robots.txt File

By placing directives in a robots.txt file Google, Bing and other search engines are given detailed instructions about what and what not to index on a site.

In other words, search engines can be directed to index relevant content and ignore ‘less relevant content’.

For more information on robots.txt files on higher education websites read our post: robots.txt - Help HigherEd Website Visitors Find The Good Stuff