Donate to us via Paypal!
One mans freedom fighter, another's terrorist.
Tuesday, October 20, 2020
Navigation
Home
 Find:
 Information:
Learn
Communicate
Submit
Shop
Challenges
 Exploit:
 Programming:
 Think:
 Track:
 Patch:
 Other:
 Need Help?
Other
Members Online
Total Online: 80
Guests Online: 80
Members Online: 0

Registered Members: 129287
Newest Member: carolynoc60
Latest Articles

View Thread

HellBound Hackers | Computer General | Web hacking

Author

Mirroring and robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 14-01-07 21:47
Is there a way to mirror a site to my HD without robots.txt making shit? I learned this the hard way.


Author

RE: Mirroring and robots.txt

Uber0n
Member



Posts: 1963
Location: Sweden‭‮
Joined: 13.06.06
Rank:
Hacker Level 3
Posted on 15-01-07 06:24
Um.. Exactly what do you mean? Robots.txt is for making sure that web crawlers don't add any pages you want to keep secret and some other settings.


img230.imageshack.us/img230/724/uber0nsig3hj6.gif
http://uber0n.web. . .
Nope http://uber0n.webs.com/
Author

RE: Mirroring and robots.txt


Member

Your avatar

Posts:
Location:
Joined: 01.01.70
Rank:
Guest
Posted on 15-01-07 08:19
But isn't that the file that stops mirroring bots as well? I think I made the wrong assumption here. :whoa: