From: Silvio Rhatto <rhatto@riseup.net>
Date: Sat, 5 Dec 2020 20:04:13 +0000 (-0300)
Subject: Feat: adds a robots.txt
X-Git-Url: https://gitweb.fluxo.info/?a=commitdiff_plain;h=f9b94e3ba677fd19c311fdde7cd9da8709953000;p=semanticscuttle.git

Feat: adds a robots.txt

Robots tend to crawl all tag combinations, which make a huge performance
impact in larger datasets, specially regarding database queries.

This robots.txt file will prevent that regular search engines to
index the application, which is already a search index.
---

diff --git a/www/robots.txt b/www/robots.txt
new file mode 100644
index 0000000..1f53798
--- /dev/null
+++ b/www/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /