By placing directives in a
robots.txt file Google, Bing and other search engines are given detailed instructions about what and what not to index on a site.
In other words, search engines can be directed to index relevant content and ignore ‘less relevant content’.
For more information on
robots.txt files on higher education websites read our post: robots.txt - Help HigherEd Website Visitors Find The Good Stuff
- Next >>