llvm bugzilla not indexable

Project Infrastructure LLVM Dev List Archives
1

Hi all,

As I was perusing LLVM Bugzilla, I noticed that it’s not indexed by search engines, probably thanks to the robots.txt file* which inhibits all indexing.

I think that it’s extremely useful for these issues to show up in say google search queries, so I wonder if it’s just an oversight or if there’s another reason for this.

Thanks,

/Ties

User-agent: *
Allow: /index.cgi
Disallow: /

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

2

(sorry for the disclaimer at the bottom, please ignore it)

3

Hello

We explicitly disallow indexing of bugzilla due to lots of bogus
requests from broken bots which create major workload on llvm.org.

4

It probably could be setup to whitelist some of the well-behaved search engine bots. If it’s set to disallow by default then the broken bots should not try to index it.

5

Well... the reality is that the broken bots sometimes ignore
robots.txt (and therefore we have to block certain IP ranges) and
broken bots also mimic the benign ones.