Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Are you verifying the IP, or just blindly trusting the user agent?


Good question - I am just putting up robots.txt, and seeing little to no decrease in traffic. I have not tried verifying that server logs user agent corresponds to specific IP addresses. Do you have resources where all the AI bots post their list of IP addresses? Would be easier to just ban by IP completely. From what I've read these bots rotate and use residential blocks so I am not sure I can even see all of them.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: