Follow us on Twitter!
Your life is ending one minute at a time. If you were to die tomorrow, what would you do today?
Thursday, September 29, 2016
Navigation
Home
 Find:
 Information:
Learn
Communicate
Submit
Shop
Challenges
 Exploit:
 Programming:
 Think:
 Track:
 Patch:
 Other:
 Need Help?
Other
Members Online
Total Online: 39
Guests Online: 37
Members Online: 2

Registered Members: 95756
Newest Member: cantbetraced100252
Latest Articles
View Thread

HellBound Hackers | Computer General | Web hacking

Author

robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 13-03-06 22:27
okay before everyone trie to explain to me what the purpose of .../robots.txt is, I KNO

so my question is what do you guys think this means if you were to find it on a .../robots.txt
"
User-agent: *
Disallow: /
"

would i have to change my useragent to "*" and what does the dissallow part mean?GrinGrin
-ZTB


Author

RE: robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 13-03-06 23:07
well apparently u dont kno what robots.txt does. It says which search engines (aka user agents, such as googlebot) can't display search results based on what come after "disallow:". Also, and this is a general note, in robots.txt and in most places "*" is a wildcard meaning anything. Also, "/" specifies the root directory of the web site, so what u posted means that the website with that robots.txt file dosen't allow any search engine to search any of the site. Hope this helps Smile'




Edited by on 13-03-06 23:08