Follow us on Twitter!
One mans freedom fighter, another's terrorist.
Thursday, April 17, 2014
Navigation
Home
HellBoundHackers Main:
HellBoundHackers Find:
HellBoundHackers Information:
Learn
Communicate
Submit
Shop
Challenges
HellBoundHackers Exploit:
HellBoundHackers Programming:
HellBoundHackers Think:
HellBoundHackers Track:
HellBoundHackers Patch:
HellBoundHackers Other:
HellBoundHackers Need Help?
Other
Members Online
Total Online: 14
Guests Online: 14
Members Online: 0

Registered Members: 82815
Newest Member: medjiking
Latest Articles
View Thread

HellBound Hackers | Computer General | Web hacking

Author

robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 13-03-06 22:27
okay before everyone trie to explain to me what the purpose of .../robots.txt is, I KNO

so my question is what do you guys think this means if you were to find it on a .../robots.txt
"
User-agent: *
Disallow: /
"

would i have to change my useragent to "*" and what does the dissallow part mean?GrinGrin
-ZTB


Author

RE: robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 13-03-06 23:07
well apparently u dont kno what robots.txt does. It says which search engines (aka user agents, such as googlebot) can't display search results based on what come after "disallow:". Also, and this is a general note, in robots.txt and in most places "*" is a wildcard meaning anything. Also, "/" specifies the root directory of the web site, so what u posted means that the website with that robots.txt file dosen't allow any search engine to search any of the site. Hope this helps Smile'




Edited by on 13-03-06 23:08