I revisited my old add-on performance article and noticed that the contents of the Measuring Startup wiki page have changed substantially since I originally linked to it. It now recommends installing an add-on  to measure startup performance. I haven’t tried it, but there are a few reasons I can think this is not the best approach. (Update: I’ve been informed that the add-on is only a display for data that is gathered and stored locally. You can make test runs and then install the add-on to look at the data. That dispels my previous doubts about this approach.) Regardless, I’m documenting the old testing method here, because it is the one I have been using for a while and is also very similar to the one implemented on our automated Talos testing framework.

I have been doing lots of add-on startup testing recently, mainly to double check if the results of the Talos tests are sound. We also correlate them to real world usage data that we have been collecting since early versions of Firefox 4. This data, manual testing and source code review are the main backup sources that give us a good confidence level in the results we display on our infamous performance page (it has been linked enough).

Here’s what I do.

Setup

1. Create a new profile dedicated for testing add-on performance (I called it startuptest).
2. Download this HTML page and save it somewhere convenient. The page is blank if you open it directly. All it does is run some JS that extracts a timestamp after the # character in the URL, compares it against the timestamp when the script is run, and shows the difference on the page.
3. Set up a console command that opens Firefox in your testing profile and opens the downloaded file, with the current timestamp embedded after the # character. On my system (Mac OS), this command is the following:

/Applications/Firefox.app/Contents/MacOS/firefox-bin -P startuptest -no-remote file:///Users/jorge/startup.html#`python -c 'import time; print int(time.time() * 1000);'`

The old version of the Measuring Startup page explains how to set this up on Windows.

Testing

1. Locate the testing profile folder and delete all files in it, if there are any.
2. Open Firefox on this profile. You can use the console command or any other shortcut if you prefer.
3. Copy and paste the add-on listing page URL on the new profile and open the page.
4. Install the add-on using the install button and restart if necessary.
5. Optionally, set up the add-on in a realistic way. For example, if this is a Facebook add-on, it may make sense to log in to a Facebook account since otherwise most of the add-on’s functionality would be inactive.
6. Quit Firefox.
7. Run Firefox using the console command.
8. Note the result in the startup page.
9. Quit Firefox. I prefer using the Quit key shortcut, to interact with Firefox as little as possible.
10. Repeat steps 7-9. I discard the 2 first runs, which are normally much slower than the rest, and measure the 10 runs after that.

Interpreting results

For your results to make any sense, it’s also necessary to make a test run without any add-ons installed, and use that as your baseline. It’s also a good idea to run all the tests consecutively to have some certainty that they are all running under similar conditions. I record and compare my results on a spreadsheet, like this one where I tested both my add-ons.

Looking at the results, Fire.fm has a somewhat noticeable impact on startup. This is not surprising because it is a complex add-on with a very complex overlay and startup processes. I documented on improving startup code in my old blog post, and we’re planning on greatly simplifying its overlay soon(ish). I doubt we’ll make the coveted 5%, but we’ll see. Remote XUL Manager is clearly simpler, and it shows how the results should not be taken at face value. Since all it does in the overlay is add a menu item that opens a separate window, it’s understandable that its impact is negligible. But does it really improve startup? No, of course not. This just means that the error margin is larger than its real performance impact.

The key takeaway here is that the results of manual tests shouldn’t be taken literally, but they’re still a good indicator of the performance impact of an add-on. Even if the error margin is not ideal (or even measurable under these conditions), you can still get a good idea of who’s fast and who’s slow. They have been very valuable to us when comparing them against Talos results.

How does this compare to Talos?

On one hand, these tests are influenced by how the testing system is set up. I have several applications open at all times, and I don’t close them all for testing. I do take care in not running anything heavy simultaneously, like Time Machine or MobileMe Sync. And then there’s clearly the fact that I have to spend some time setting things up and running the tests. The longer the tests take, the more likely it is that some other process affects the results.

On the other hand, it’s easier for me to recognize errors during testing. Many of the complaints we’ve received about the testing system is that it makes silly mistakes like trying to install an add-on from an incorrect URL, or trying to install an add-on that is not compatible with the Firefox version being tested. These are things that one can clearly see when testing manually, but they weren’t obvious when running the tests automatically. Those add-ons have been getting very good performance rankings because they’re not really being loaded, so those results are not reliable.

Luckily, the people complaining about our testing are also filing bugs and talking to us directly, so we’re looking into the issues and trying to get them resolved as soon as possible. Special thanks to Wladimir and Nils, who have been very helpful filing and categorizing bugs. More details coming up in the Add-ons Blog.

As always, the developer community proves itself as an invaluable asset for Mozilla (well, you are Mozilla). Even if our discussions can become harsh and are generally very public, the outcome is almost always a set of improvements both in our technical and communication fronts. Getting things right take a lot of work and a lot of patience, and I hope we can quickly get to a place where we’re all satisfied.

• Remote XUL Manager has taken most of my spare time. It’s been exciting just because it’s a fairly new project and it had a pretty well-defined goal, which was reached only a few days ago. Version 1.0 was released on March 10th, and it does every I planned for this add-on to do. There are a couple of feature requests that I will consider in due time, but they’re pretty minor and I’m satisfied with the current state of this project.
• Fire.fm has taken the back seat. In the past few months all we’ve done is update locales, clean up some code, and push a few bug fixes (and then fixes on the fixes, and so on…). We have some plans for version 1.5, but it’s getting harder and harder to find the time for this. Specially because Last.fm keeps closing down their service and that has a very negative effect on my motivation. If it weren’t for the people that still use it – close to 200K – I would have abandoned this add-on a long time ago.
• My Add-on Packaging Proposal for the Google Summer of Code never gained any traction. Unfortunately the project was abandoned by the student assigned to it, so the whole idea stalled and was discarded. Now I’ve been approached by somebody who is interested in working on it. That’s very encouraging, but I need to give this some thought. I’ve been planning on reviewing the whole spec and giving it an overhaul based on all the stuff that has changed with Firefox 4 and the feedback I received. I’ll post here when (and if) I have an updated draft.
• The band I help as Executive Producer, Pneuma, has had a great deal of success locally, and they’re getting ready to tour Mexico and Europe. It’s very exciting times for them and I’m glad to be a part of this.

It’s been crazy times for me and the AMO Editor team. We had a very successful competition that yielded a incredible throughput and over 2500 reviews by the top 4 winners alone. And yet we’re still swamped with review submissions due to the upcoming Firefox 4 launch and the mass disabling of sandboxed add-ons, which resulted in a great deal of new submissions of old add-ons.

I have a bunch of ideas for new add-ons that I wish I had the time to work on. For now, I just write them down on my Someday list and wait

1. Most of all, I’ve been contributing to the AMO editor team to keep the add-on review queues down to a manageable size. There’s a great deal of new submissions and updates due to the release of Firefox version 3.5. There are hundreds of updates that are minimal to non existent compatibility changes, so I’ve been able to review close to 200 updates on this month alone. I enjoy my work with the editor team, specially because I’ve very fond of doing code review and being ruthlessly critical. It’s in my nature, what can I say… There are plans for removing the AMO sandbox, which I find intriguing, to say the least. I wonder how that will change the editor group and their work.
2. The Extend Firefox 3.5 competition has been officially announced. With an October deadline, it includes several new interesting categories to compete in. Jose and I are already working on Fire.fm 1.3, which will be our entry for best extension update. There’s a great deal of new features we’ll be putting into it, which I think will greatly improve our user experience. We’re also working on a few ideas for new add-ons. I hope to be able to submit a couple of entries to increase our chances of winning.
3. Finally, the Add-ons Contributions pilot was introduced to AMO. This enabled add-on authors to request donations for their project right on AMO, giving our donation links more exposure and possibly allowing a few authors to remove donation requests from their add-on UI, which is always a loss for user experience. We have activated contributions for Fire.fm, and so far the response has been pretty good. We’re very happy about it and hope people will continue helping us out.

I still feel in ‘break mode’, and hope to start working in full gear soon. There’s so much I want to do, and I still need to get everything sorted out. More updates soon!

Anyway, at the time we were pretty excited about it, and we spent a great deal of time and effort making Fire.fm a sure winner. And we won! We won the Best Music Add-on category, and I think we weren’t allowed to win in others, because I think we should have won the general competition as well . Needless to say, we were super happy with this, and we have been actively maintaining the add-on in our spare time.

Fire.fm is, in a nutshell, a Last.fm radio player. It used to be a very easy way to tune into Last.fm radio without having to be on top of the website all day long. Our users loved it, and shortly we were in the AMO Recommended List.

Sadly, Last.fm cut off their free radio service to most of the world shortly afterward, so at least I lost a lot of the motivation I initially had to continue making it grow. Still, I don’t want to abandon a project that still has much to be fixed, so I’ll certainly won’t quit until all the annoying bugs are solved. Most users have a very smooth experience with Fire.fm, but there are a few unlucky ones that have some serious problems, like not being able to listen to stations at all… New features in Fire.fm are probably going to be very rare from version 1.2.3 and later.

Xulforge was already in my mind when we began working on Fire.fm, so I always kinda saw it as a Xulforge project. Xulforge was put on hold until very recently, so it feels weird trying to associate the two. I’ll keep Fire.fm in my project list, though, because I it was one of my first independent projects, and also because it has been very successful and I’m very proud of it.

Fire.fm is open source, BSD-licensed (more on that in another post). So, if you want to take a look into our code, you’re more than welcome. See the Fire.fm Sourceforge Project Page for more information.

Fire.fm Logo