about summary refs log tree commit diff stats
path: root/filters/html-converters
diff options
context:
space:
mode:
authorJason A. Donenfeld2013-05-28 14:17:00 +0200
committerJason A. Donenfeld2013-08-12 13:14:10 -0600
commit23debef62104c70600be2b745ec3957538eeac6e (patch)
tree3b68eab7624907836b0e614328a529e686224830 /filters/html-converters
parentuse favicon by default (diff)
downloadcgit-23debef62104c70600be2b745ec3957538eeac6e.tar.gz
cgit-23debef62104c70600be2b745ec3957538eeac6e.zip
robots.txt: disallow access to snapshots
My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.

Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Diffstat (limited to 'filters/html-converters')
0 files changed, 0 insertions, 0 deletions