Comments on: Blocking Baidu and Yandex Search Spiders

By: CathyL

CathyL — Mon, 18 Feb 2013 10:45:41 +0000

It’s working for us and has reduced bandwidth usage. We do not mind blocking spiders from China or Russia as were only selling into the UK.

Thanks x

By: Paul

Paul — Mon, 18 Feb 2013 04:36:36 +0000

Hi Daniel, there’s more than one way to send the bots away and either method should work just fine.
You can even catch the request in your CMS and issue 403 messages to the bot, you’ll still see the request to the site in that case but you won’t be serving up any data, therefore if it’s done right there’s minimal load on the server.

By: Daniel

Daniel — Tue, 12 Feb 2013 14:42:22 +0000

hy vicent. Sry for my english
i have this rules in my htaccess. is ok like that or.. is better in your way ? I don’t know the difference. Some use this, some use like u with SetEnvIfNoCase how is better?
RewriteCond %{HTTP_USER_AGENT} ^Baidu [OR]
RewriteCond %{HTTP_USER_AGENT} ^Yandex [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sosospider [OR]
RewriteRule ^.* – [F,L]

By: Jim

Jim — Sun, 10 Feb 2013 18:06:46 +0000

I was hoping this was going to resolve my issue with these bots and their CPU drain but alas your fix at least for me has not been successful in blocking them. 🙁 I have added this code to five off my domains and they are still getting bombarded. Total Bummer…

By: Vincent

Vincent — Tue, 29 Jan 2013 11:29:21 +0000

In reply to Phil.

Hi Phil,
Thanks for the response. It has been almost six months since we implemented it onto a couple of e-commerce sites we manage. So far so good. Your response prompted me to have a look through the stats and I found no incidents of any of the above bots. However, I have updated the post to include the ‘Sosospider’ another resource draining spider / bot.
Let us know how you get along.

By: Phil

Phil — Mon, 28 Jan 2013 21:31:37 +0000

Hey Vincent,

I came here because Baidu and Yandex bots are the main drain on my bandwidth for all the sites I administer. I actually wrote to baidu’s support department to beg them to stop hitting my website and for a while they did, but of course they came back.

Have you found the above fix to work without side effects since you’ve implemented it, because if so this will be awesome news!

Phil