]> gitweb.fluxo.info Git - semanticscuttle.git/commit
Feat: adds a robots.txt
authorSilvio Rhatto <rhatto@riseup.net>
Sat, 5 Dec 2020 20:04:13 +0000 (17:04 -0300)
committerSilvio Rhatto <rhatto@riseup.net>
Sat, 5 Dec 2020 20:04:13 +0000 (17:04 -0300)
commitf9b94e3ba677fd19c311fdde7cd9da8709953000
tree6d95d3bebd6c908368c105af3bc279f340ea3b68
parent37f10bd9d3f955fd648424d3908d23c33bea39fc
Feat: adds a robots.txt

Robots tend to crawl all tag combinations, which make a huge performance
impact in larger datasets, specially regarding database queries.

This robots.txt file will prevent that regular search engines to
index the application, which is already a search index.
www/robots.txt [new file with mode: 0644]