How To Block Fasterfox Requests 50
Over the last few months the amount of visitors using Mozilla Firefox has grown to about 25%. The amount of bandwidth being used has also increased a large amount. Part of the reason behind this is that many Firefox users use an extension called “Fasterfox”. This extension “pre-fetches” links on a page so that if the user were to click on a link it would load much faster because its already been downloaded. This may be more convenient for viewer, but is a major problem for many webmasters who are low on bandwidth. Since Fasterfox constantly requests new files, it can cause many servers to overload much faster than if a person viewing the same content without Fasterfox were to view it.
Fasterfox is in fact one of the most popular extensions for Firefox. It is currently ranked as the 3rd most downloaded extension for Firefox on Mozilla Update page. (aka. Firefox Add-ons) The latest version of Fasterfox, v1.0.3, checks the for the robots.txt file on the site the viewer is visiting to check whether it should pre-fetch or not. This new feature allows webmasters to add the following text (in bold) to their robots.txt file to prevent Fasterfox from pre-fetching links. Text To Add To “robots.txt”:
- User-agent: Fasterfox
- Disallow: /
Adding these two lines somewhere in your robots.txt file and placing it in the root folder will prevent Fasterfox from pre-fetching links anywhere on your site. (ex: yourwebsite.com/robots.txt) Webmasters can modify the text so that Fasterfox will only be prevented from pre-fetching on specified directories. Still have questions? Ask us! (reply in the comments)
Links: Fasterfox | Firefox Add-ons
Related: “robots.txt” Tutorial
Recent: Firefox Extend Contest Finalists
50 Comments
Write A Comment
Web Reactions:
Leave A Comment
Random
Recent
- HD Radio (Review) + Insignia Portable Player
- Skullcandy SK Pro Headphones (Review)
- Microsoft Arc Mouse (Review)
- Super Street Fighter IV (Preview)
- Lenovo IdeaPad S10-2 Netbook (Review)
- Sonic the Hedgehog 4
- Palm Pre Plus – Verizon Wireless (Review)
- Apple Announces The iPad
- RIM BlackBerry Curve 8530 (Review)
- Sennheiser-Adidas CX 680 (Review)
Categories
- Accessories
- Audio
- Autos
- Business
- Deals
- Entertainment
- Gaming
- Hacking
- Hardware
- Internet
- Mobile
- News
- Opinion
- Podcasts
- Reviews
- Site News
- Software
- Tips & Tweaks
Sponsors
- cheap website hosting from 2MHost.com
- Toners and inks with pleasant additions
- web design company
- Cell Phones from Wirefly
- Advertise Here $40/month
i just added this to the robots file. ill see if this actually works since fasterfox is sorta stupid at times.
if i knew about this i would have used it a long time ago. i just checked the fasterfox change log and indeed they have added “robots.txt” check for instructions.
this is actually neat that they created this so ppl w/ bandwith issues can block it and others who have excess of bandwidth can allow it. i guess noone needs to complain about fasterfox anymore :)
This is a good move on the part of the developers. While I’m not sure that the entire concept of these things particularly sound – most people will never stumble across this fix, particularly casual bloggers and such who might attract a lot of traffic but are bandwidth limited – having a way to block it is probably more than many developers would bother to implement I’m sure.
fasterfox isnt all that anyway… use google web accelerator
::most people will never stumble across this fix, particularly casual bloggers and such who might attract a lot of traffic but are bandwidth limited ::: Digg Frontpage enough advertising for it?
Great tip! Adding this to robots.txt now.
No, DD32. Unless the fix is integrated as an option with popular software such as WordPress, a lot of *casual* users will not be aware of the fix. Still, the fact it exists at all is a good step. :)
Cool! Thanks for the warning, just downgraded to 1.0.0, the last version before the robots check!
get it here:
http://downloads.mozdev.org/fasterfox/fasterfox-1.0.0-fx.xpi
Daniel is a fine example of human decency.
If I have the robots.txt file setup like this:
User-agent: *
Disallow: /
Will that block fasterfox as well, or do I have to make a separate entry?
if most webhosts use this robots trick, then the extension becomes useless. i download this extension to decrease load times on pages on the web, not just the ‘high bandwidth’ pages.
shawgo, your setup should technically work, because all standard search engines would read that and know not to index anything. But the creators of Fasterfox have specified the method posted in this article, therefore if you are really worried I’d just place another two lines along with what you have to once again specify a command for Fasterfox.
Daniel, I don’t think you understood the purpose of this article. It is not to help people bring servers down, but is to help everyone. Please upgrade to 1.0.3 once again. This will help the sites you like stay up. If you do use this on severs that are already over loaded, this will just help bring them down.
I don’t think Daniel understands that webmasters generally pay for limited bandwidth resources, and that in a lot of cases this limit isn’t awfully high – especially on blog websites and such. If one of these websites were to post a good article and have lots of visitors come to read it, there is a significant risk that the site would be knocked offline for the remainder of the month unless the webmaster foots the bill for additional resources. If it’s a hobby site, this is unlikely.
This risk is basically quadrupled (or more) on a per-user basis for everybody using the unfixed extension. So, Daniel, if you’re happy with sites you visit being knocked offline, then sure, go ahead and spread the good word about 1.0.0 ignoring robots.txt. You will then see your entire browser become useless. ;)
I added the extra line, just to be safe. Thanks for the advice, Sahas.
Daniel:
fasterfox won’t become useless, just restricted. If it becomes annoying enough to content providers, the backlash would make firefox less popular, and firefox themselves would move to block it. controlled is better for everyone. Besides, like the poster above mentions, most of the world will never utilize this.
Personally, I think the prefetch aspect is dumb anyways. I am never interested in 100% of the links on any given page. I use fasterfox but turned off prefetch.
Does Fasterfox send an X-Moz header with its prefetch requests? I blocked that back when Google Web Accelerator came out, would it also stop Fasterfox prefetches?
The sad part is that Fasterfox is open source, much like every other extension and the mozilla browser itself, which means that any user can just remove the useragent header from the fasterfox extension and totally bypass the robots.txt file.
It would have been nice to say that the developers! of the extension have this information publicly posted:
Admin: Link has been removed because of some part of the URL that corrputed the page.
Here is the new URL: http://tinyurl.com/zctne
User-agent: *
Disallow: /
Should NEVER be done unless you want to take yourself out of all the search engines…. it’s search engine ranking suicide.
Not everyone depends on search engines for traffic.
Thanks for the heads up. Its getting ridiculous the number of ways Firefox is being extended (even in the trunk build) to sap bandwidth from websites to give the user a false impression of speed. If only Firefox itself supported robots.txt for its own href prefetching.
Fred: If there is one thing I have learned in all my years of Open Source development, is the overwhelming majority of folks are lazy. I don’t see that to be a problem;
Thanks for the tip!
If the problem becomes seriousness enough than scripts which block what are clearly prefetching from a site for a temporary duration are likely to be be developed – it could even perhaps be set to ignore search engines, using the IP database of something else out there as a headstart.
If people like Daniel insist on sapping people’s bandwidth, the chances are that it will get to the extent were webmasters retaliate with more than casual blocking.
See more extensions for power surfing.
http://pchere.blogspot.com/2005/12/50-best-firefox-extensions-for-power.html
> Stwo says
> fasterfox isnt all that anyway… use google web accelerator
While I agree Fasterfox aint all that perhaps you should open your eyes to the fact that not everyone can meet this requirement :
To use Google Web Accelerator, your computer must have a Windows XP or Windows 2000 SP 3+ operating system. Google Web Accelerator works for the Internet Explorer 5.5+ or Firefox 1.0+ browsers.
plus I have 100+ links on quite a few of my pages
Fasterfox is virtually no help to anyone coming to my website
Yeah, but, at least in the version I have, prefetching is turned off by default…
Hey Daniel
Thanks for that like to fasterfox 1.0.0
get it here:
http://downloads.mozdev.org/fasterfox/fasterfox-1.0.0-fx.xpi
Update: BUGGER it doesn’t work with firefox 1.5.0.1
The Fasterfox just forked, so now you can either stick with the original Fasterfox branch, or get FastestFox which does not include the robot.txt check.
Daniel, Matt: Why is it so critical to leave prefetch on? Are you on network connections with enough latency that you really need that enabled for a fast experience? Besides, if you disable the check, webmasters may just decide that they’ll add absolute throttling or write behavioral plugins (easy enough to do — I’m no programmer and even I could do it) to block pre-fetching or ban hosts using it. In fact, if enough folks like you are insistant enough to ignore it when people have politely asked you to stop, I fully expect that to happen. If they’ve asked you to stop, why not play nice and just stop? It doesn’t seem like it’s too much to ask for.
Fortunately my website isn’t really large enough for prefetching to cause too much of an issue (I think) – but I know if people rammed my bandwidth, and I went offline, I don’t think I could easily sort out the cash to pay for more. Ideally I would prefer my bandwidth to be used by people downloading my music than prefetching!
The idea that bandwidth could be used by someone who might not visit those prefetched pages seems crazy.
Very handy, fasterfox is bad news for us web developers.
http://www.thewebdesignblog.com
At last, it reads robots.txt. Well, that was much needed.
Now that is really good news.
Have added it to my robots.txt Now wait and monitor :)
Any clue on how to check how much bandwidth was it actually eating up?
My websites are update, but most won’t.
This is another invention to save our beloved cyberspace… much like segregating nodes via ip address (for the elite), p2p “traffic shaping” (censorship) and use of DRM to (jacking the consumers some more)… Why don’t you twats write a pluggin that will make http requests more effiecent rather then making some crap that limits what people can do… You make the users to be the enemies when in fact the bandwidth providers are the bastards…
Thats great that it is finally possible to block prefetching. Before that people could download gigabytes of data that would never be read. As for Daniel and Matt, I ban that type of person from ever entering my websites
Prefetch makes web browsing 10x faster easily! I use 1.0.3 & respect those sites that do not want prefetch, but I have never noticed any problems with my own site. (Than again, i’m on University webspace)
Unfortunately, in some of my tests the robots.txt entry doesn’t stop it…
Good, at least they can bork the robots.txt thing :D
Thanks for the info. I was just looking up fasterfox to see if it was worth using. But after reading this i think i will wait the extra half a second to download pages rather than cause problems for some people who spend a lot of their own personal time for the benifit of others. I only checked this out because firefox put a block on the download, maybe they are be responsible.
Do you know how to go about discovering installed instances of this plug-in? As in, what if I have some users who have installed this and I don’t want them using it… I didn’t see any additional processes or executables caused by its installation.
Well, if people are going to ignore the request anyway then webmasters will have to take more drastic measures. We implemented a system where if you grab more than x pages in a 5 second period you are banned. Stopped all that fasterfox traffic. Shouldn’t have to do that. Fasterfox is broken. We had some Firefox users downloading the same pages HUNDREDS of times. We had some users machines that would download 1500 pages in a 5 minute period because of fasterfox… That is just dumb. There is NO WAY a human could view all of them.
The old fasterfox xpi (1.0.0) does work with Firefox 2.0.0.4 if you alter the archive with WinRAR to change the install.rdf version checking.
SiteSucker is still around if you really want everything all at once and don’t care how much damage you do. It’s all about personal responsibility.
I just bumped Fasterfox 1.0.0 to work with Firefox 2.
Now ignoring robot.txt. BWAHAHHAHA!
The user can easily switch off prefetching. Go to Add-ons in the Tools menu, click Options in the Fasterfox entry in Extentions, switch to Courteous and then back to Custom in the Presets menu, and that is the right place to do this: In the Pipelining menu, Enable Pipelining and Enable Proxy Pipelining, and set Max Pipelining Requests to 10. In Rendering, set Pages In Memory to 10. This procedure should get rid of any prefetching problems and still allow great speed.
It is too bad that the XPI’s code has released to the general population, now it is a simple matter for anyone with a smidgen of programming knowledge to remove the polite-man check and one rouge group called something like Frozey Crew 200* has released a version that is simply nastier than anything that has come before it that utilizes the entire available bandwidth and when it runs out it seeks other sites that have been somehow selected at ramdom to continue the data pull. Just friggen nasty…Why people have to do crud like that is beyond me, there out to be international laws against those kind of nasty-hacker groups.
I just implemented this onto my website. That’ll teach those bandwidth thieves a lesson or two. Also might lower the “Digg Effect” considering that a lot of people who visit from Digg are probably using this extension. When your server is taking a beating from 100,000 people and 20,000 of them are using Fasterfox without this your server is already gone.
- Dwayne Charrington.
http://www.dwaynecharrington.com
what a hell are u talking about ???
I didn’t figure it out!!!!!
is it dangeros ??
Since there is a FasterFox fork which ignores the robot.txt I will implement a blocking mechanisim. Care needs to be taken not to block search engines but I think this can be done easily.
- Tell all search engines not to spider a specific directory via robots.txt
- Setup a hidden link on every page, let’s call it VandalTrap in this example
- Run a server side script such as PHP on the index page of the VandalTrap directory. If the page is loaded, the .htaccess file is modified to block the user
This could also be extended to display a informing message but if they choose to install software which circumvents the rules then blocking the IP for 20 minutes should be the best choice.
Faster 3.1.2 for Firefox 2.0 at 3 !! https://addons.mozilla.org/fr/firefox/addon/14833?collection_id=4985b32f-408b-7861-5ee1-9acc446a9cfb