Follow us on Twitter!
Society leans ever heavily on computers, if you have the power to take out computers you can take out society. - cubeman372
Wednesday, April 23, 2014
Navigation
Home
HellBoundHackers Main:
HellBoundHackers Find:
HellBoundHackers Information:
Learn
Communicate
Submit
Shop
Challenges
HellBoundHackers Exploit:
HellBoundHackers Programming:
HellBoundHackers Think:
HellBoundHackers Track:
HellBoundHackers Patch:
HellBoundHackers Other:
HellBoundHackers Need Help?
Other
Members Online
Total Online: 25
Guests Online: 23
Members Online: 2

Registered Members: 82885
Newest Member: ConiBE
Latest Articles
View Thread

HellBound Hackers | Computer General | Web hacking

Author

robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 13-03-06 22:27
okay before everyone trie to explain to me what the purpose of .../robots.txt is, I KNO

so my question is what do you guys think this means if you were to find it on a .../robots.txt
"
User-agent: *
Disallow: /
"

would i have to change my useragent to "*" and what does the dissallow part mean?GrinGrin
-ZTB


Author

RE: robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 13-03-06 23:07
well apparently u dont kno what robots.txt does. It says which search engines (aka user agents, such as googlebot) can't display search results based on what come after "disallow:". Also, and this is a general note, in robots.txt and in most places "*" is a wildcard meaning anything. Also, "/" specifies the root directory of the web site, so what u posted means that the website with that robots.txt file dosen't allow any search engine to search any of the site. Hope this helps Smile'




Edited by on 13-03-06 23:08