User-agent: * Disallow: / User-Agent: http://www.almaden.ibm.com/cs/crawler Disallow: / User-agent: http://www.almaden.ibm.com/cs/crawler [c01] Disallow: / User-Agent: Baiduspider+(+http://www.baidu.com/search/spider.htm) Disallow: / User-Agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) Disallow: / User-Agent: MCUPDATE Disallow: / User-Agent: fast Disallow: / User-Agent: Gator/4.1 Disallow: / User-Agent: WEP Search 00 Disallow: / User-Agent: IEFav172Free Disallow: / User-agent: DiaGem Disallow: / User-agent: Blocked by Norton Disallow: / User-agent: nabot Disallow: / User-Agent: sitecheck.internetseer.com Disallow: / User-Agent: Jabot/7.2.3 (http://odin.ingrid.org/) Disallow: / User-Agent: Openbot Disallow: / User-Agent: PureSight Disallow: / User-agent: mogimogi Disallow: / User-Agent: Googlebot Disallow: / User-agent: girafa Disallow: / User-agent: Steeler Disallow: / User-agent: TurnitinBot Disallow: / User-agent: InfoSeek Sidewinder/0.9 Disallow: / User-agent: Infoseek SideWinder/1.1A Disallow: / User-agent: Infoseek SideWinder/1.4 (Compatible; MSIE 10.0; UNIX) Disallow: / User-agent: Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX) Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: Netscape-Catalog-Robot Disallow: / User-agent: W3CRobot/4.0D Disallow: / User-agent: MOMspider/1.00 libwww-perl/0.40 Disallow: / User-agent: libwww-perl/5.02 Disallow: / User-agent: iSpi/2.0 Disallow: / User-agent: NetRecorder/1.0 Disallow: / User-agent: WWWC/1.04 Disallow: / User-agent: WWWC/0.24 Disallow: / User-agent: InternetLinkAgent/3.1 Disallow: / User-agent: indexpert/2.0 Disallow: / User-agent: Microsoft URL Control - 6.00.8862 Disallow: / User-agent: gazz/2.1 Disallow: / User-agent: Robot/www.pj-search.com Disallow: / User-agent: WebAuto/3.40 Disallow: / User-agent: Pockey-GetHTML/4.11.4 Disallow: / User-agent: MailSpider Disallow: / User-agent: ia_archiver Disallow: / User-agent: Csice-Agent Disallow: / User-agent: HyperRobot Disallow: / User-agent: Mercator-2.0 Disallow: / User-agent: Scooter/1.0 Disallow: / User-agent: Scooter/3.2 Disallow: / User-agent: Scooter/3.3 Disallow: / User-agent: minibot Disallow: / User-agent: Hatena Antenna/0.3 Disallow: / User-agent: Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) Disallow: / User-agent: mogimogi/1.0 Disallow: / User-agent: moget/2.1 (moget@goo.ne.jp) Disallow: / User-agent: FreshGet/4.0 Disallow: / User-agent: indexpert/2.0 Disallow: / User-agent: Mozilla/3.0 (aruyo/0.01;http://www.aaacafe.ne.jp/ ;support@aaacafe.ne.jp) Disallow: / User-agent: Down Site Checker V2.0 Disallow: / User-agent: Mozilla/3.0 (compatible; Indy Library) Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: Lycos_Spider_(T-Rex) Disallow: / User-agent: nttdirectory_robot/0.9 Disallow: / User-agent: InfoNaviRobot(F104) Disallow: / User-agent: Sleipnir Version 1.40 beta7 Disallow: / User-agent: Sleipnir Version 1.41 Disallow: / User-agent: Sleipnir Version 1.40 beta3 Disallow: / User-agent: Cuam Ver0.050bx Disallow: / User-agent: WebPatrol/1.0 OS X Disallow: / User-agent: yukikaze/3.0 Disallow: / User-agent: dloader(NaverRobot)/1.0 Disallow: / User-agent: UP.Browser/3.04-MA13 UP.Link/3.4.5.6 Disallow: / User-agent: UP.Browser/3.04-MA12 UP.Link/3.4.5.6 Disallow: / User-agent: WebFetch Disallow: / User-agent: Jabot/7.0.7 (http://odin.ingrid.org/) Disallow: / User-agent: Jabot/7.0.6 Disallow: / User-agent: Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Disallow: / User-agent: TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html Disallow: / User-agent: http_get Disallow: / User-agent: RMA/1.0 (compatible; RealMedia) Disallow: / User-agent: Mozilla/2.0 (compatible; Ask Jeeves) Disallow: / User-agent: Internet Ninja 5.0 Disallow: / User-agent: Internet Ninja 6.0 Disallow: / User-agent: Download Ninja 7.0 Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Disallow: / User-agent: Mozilla/4.0 efp@gmx.net Disallow: / User-agent: Mozilla/3.01 (compatible;) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent Disallow: / User-agent: Scooter-ARS-1.1 Disallow: / User-agent: libwww-perl/5.53 Disallow: / User-agent: Mozilla/5.0 Galeon/1.2.6 (X11; Linux i686; U;) Gecko/20020922 Disallow: / User-agent: Java(TM) 2 Runtime Environment, Standard Edition Disallow: / User-agent: Windows Update Disallow: / User-agent: CryptRetrieveObjectByUrl::InetSchemeProvider Disallow: / User-agent: WebTrans Disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org) Disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org) Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget/1.8.2 Disallow: / User-agent: DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) Disallow: / User-agent: Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Disallow: / User-agent: CCU;MSIE Disallow: / User-agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html) Disallow: / User-agent: Gaisbot/3.0+(robot@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) Disallow: / User-agent: Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) Disallow: / User-agent: DoCoMo/2.0 P2101V(c100) Disallow: / User-agent: TMhtload/1.31.00.1717 Disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org) Disallow: / User-agent: LinkSonar/1.35 Disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org) Disallow: / User-agent: Openfind data gatherer, Openbot/3.x+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) Disallow: / User-agent: Sleipnir Disallow: / User-agent: FAST-WebCrawler/3.x (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: FAST-RealWebCrawler/3.8 (crawler@fast.no; http://fast.no/support/crawler.asp) Disallow: / User-agent: Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) Disallow: / User-agent: Scooter/3.x Disallow: / User-agent: moget/2.x (moget@goo.ne.jp) Disallow: / User-agent: Mozilla/5.0 (compatible; Konqueror/2.1.2; X11) Disallow: / User-agent: FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) Disallow: / User-agent: Microsoft Data Access Internet Publishing Provider Protocol Discovery Disallow: / User-agent: Microsoft URL Control - 6.00.8862 Disallow: / User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; \xbe\xc6\xc6\xae\xb9\xcc\xb5\xf0\xbe\xee (artmedia.org); .NET CLR 1.0.3705) Disallow: / User-agent: DLC/0.55 Disallow: / User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSN 6.1; MSNbMSFT; MSNmko-kr; MSNc00; v5m) Disallow: / User-Agent: PlantyNet_WebRobot_V1.9 dhkang@plantynet.com Disallow: / User-agent: WireAction URLCheckSpider Disallow: / User-agent: Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org) Disallow: / User-agent: Webshuttle Disallow: / User-agent: FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) Disallow: / User-agent: Charlotte/2.00 (charlotte2@tinami.com; http://charlotte.tinami.com/robot.html) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; SEARCHALOT IE6) Disallow: / User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; .NET CLR 1.0.3705; Alexa Toolbar) Disallow: / User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Alexa Toolbar) Disallow: / User-Agent: Mozilla/3.0 (compatible; Indy Library) Disallow: / User-agent: Mozilla/3.0 KOMATSU (AveFront/2.6 WebLight) WL/3.0 V2.45ohi M=11E2A2499B6D Disallow: / User-agent: searchpreview Disallow: / User-Agent: Googlebot-Image Disallow: / User-Agent: Baiduspider+ Disallow: / User-Agent: Bloglines/1.0 Disallow: / User-Agent: Iria/1.07a Disallow: / User-Agent: Irvine/1.0.3b Disallow: / User-Agent: Nutscrape/1.0 (CP/M; 8-bit) Disallow: / User-Agent: Openfind data gatherer, Openbot/3.0+ Disallow: / User-Agent: Mozilla/4.0 (compatible; PerMan Surfer 3.0; ) Disallow: / User-Agent: yahoo-fetch Disallow: / User-Agent: WebCrawler-AddURL/1.0 Disallow: / User-Agent: W3CLineMode/5.0a libwww/5.0a Disallow: / User-Agent: Rcaau/1.1 Disallow: / User-Agent: Wapibot Disallow: / User-Agent: msnbot Disallow: / User-Agent: Cowbot-0.1.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) Disallow: / User-Agent: MCUPDATE Disallow: / User-Agent: Industry Update Control Disallow: / User-Agent: WebGo IS - 1784 Disallow: / User-Agent: w3m/0.4.1-m17n-20030308 Disallow: / User-Agent: Sleipnir Version 1.42 Disallow: / User-Agent: CMC/0.01 Disallow: /