There are thousends of bots and web crawlers working the internet but below is my list of the 10 popular search engines user-agents.
If you browse the logfiles of your website, you will always see the access to a file called “robots.txt”. These are usually calls from search engines. Their web crawlers with there user-agents that read the robots.txt file (hopefully you have one). They check if a visit is allowed, which folders are not allowed and which delay is desired after each page call.
Here is a list of all user agents for the major, leading search engines. I often use this information to analyze my log files, so I thought it would be useful to publish the information online for the benefit of others. It can be useful to have user agents for these popular bots in one place. Each search engine contains references and a list of the most common user agents.
Search Engine Bot Names
- Google = Googlebot
- Bing/MSN = Bingbot
- Yahoo = Slurp
- DuckDuckGo = DuckDuckBot
- Baidu = Baiduspider
- Yandex = YandexBot
- Sogou = Sogou
- Exalead = Exabot
- Facebook = facebot
- Alexa = ia_archiver
Full User-Agent Strings
1 2 3 4 5 6 7 8 9 10 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm) Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) Sogou ... spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com) |
Table of Contents
