Feat: adds a robots.txt

Robots tend to crawl all tag combinations, which make a huge performance
impact in larger datasets, specially regarding database queries.

This robots.txt file will prevent that regular search engines to
index the application, which is already a search index.

diff --git a/www/robots.txt b/www/robots.txt
new file mode 100644 (file)
index 0000000..1f53798
--- /dev/null
+++ b/www/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /