okay before everyone trie to explain to me what the purpose of .../robots.txt is, I KNO
so my question is what do you guys think this means if you were to find it on a .../robots.txt
would i have to change my useragent to "*" and what does the dissallow part mean?
Posts: Location: Joined: 01.01.70 Rank: Guest
Posted on 13-03-06 23:07
well apparently u dont kno what robots.txt does. It says which search engines (aka user agents, such as googlebot) can't display search results based on what come after "disallow:". Also, and this is a general note, in robots.txt and in most places "*" is a wildcard meaning anything. Also, "/" specifies the root directory of the web site, so what u posted means that the website with that robots.txt file dosen't allow any search engine to search any of the site. Hope this helps '
Edited by on 13-03-06 23:08
Hellbound Hackers is the collective work of the staff and the community and is therefore licensed under the CC BY-NC-SA license.