Xulforge Blog

I began this discussion in the newsgroups today. Keeping add-ons compatible in the rapid release process. It is mostly aimed at Mozilla developers, but this should interest add-on developers just the same. We’re establishing a better system to communicate breaking changes, which should make it easier and quicker to identify what needs to be added to the compatibility validator for the automatic version bumps.

Discuss!

Gerv started a fairly intense discussion about the new rapid release cycle, from the perspective of browser versions and their meaning. As expected, many have replied that the discussion is silly and version numbers are meaningless. This is true for most software developers, and it should be true for most web developers. In software, we have been learning that version numbers have been hijacked by marketing and we shouldn’t pay that much attention to them from the technical perspective. We have branches and revision ids for that anyway. On the web, what should matter are the features the browser offers, not the browser itself or its version. Browser and version detection are frowned upon, and most web devs worth their salt avoid it when possible.

People saying that this discussion is pointless neglect that Firefox is one more thing. It’s a platform, and most add-on developers rely on it. Version numbers matter to add-on developers because the add-on compatibility system relies on application version, not features. And since the features are pretty much every access point in the platform, doing feature detection is not realistic. So, if you’re an add-on developer and you want to keep your add-on up to date, you’ll have to test and increase its compatibility every six weeks, potentially having to make code changes.

On AMO, we have implemented a system where we automatically detect which add-ons are compatible with the latest Aurora version (the version that is between 12 and 6 weeks from release) and then upgrade those that do. All developers for the tested add-ons should get an email explaining their add-on was upgraded, and if not, why. This has been fairly successful, and we have a high compatibility percentage (relative to usage) at the moment (it’s even higher if you ignore the .NET Assistant, which could be compatible now). However, there are still many popular add-ons that aren’t compatible, specially those hosted elsewhere. And the burden on those who develop add-ons with binary components is pretty much constant, because they’ll have to update their add-ons every 6 weeks, with no exception.

So, while only a small percentage of add-on usage consists of add-ons that are incompatible (hovering something between 10% and 20%, I think), the probability that one of our add-on users has at least one incompatible add-on is even higher. And having to roll the dice every 6 weeks and see if your add-ons are compatible or not puts a great deal of stress and dissatisfaction on users. And like Colin Coghill comments in Gerv’s post:

If add-ons break randomly every few weeks, effectively FF no longer has add-ons.

Version numbers have a real impact on our users then, and they should be taken very seriously. Now, would the previous releases have worked as minor version increases? I think so, yes. The breaking compatibility changes have been fairly minor so far, with the exception of binary add-ons. Even if we hadn’t done any communication about it, I’m fairly sure that the problems would have gone mostly unnoticed, by both developers and users. Of course, that’s only because so far the changes have been minor. This shouldn’t be the case for every release, and that’s a big concern. For example, Firefox 7 will remove a couple of JSON parser functions that are heavily used by add-on developers. We’ll see how that goes.

So, what are the alternatives? There are a number of proposals, with their own advantages and limitations. I personally think that going back to the initial idea or 3 month-long tracks (instead of 6 weeks) would be a major improvement. Nils Maier (author of DownThemAll!) has a very detailed proposal in the comments on Gerv’s blog (unfortunately, I can’t link to the comment directly), which consists of having planned minor and major releases, where major releases happen only twice a year. I like that one as well.

As usual, I expected this be a short post and then it grew into a monster. Oh well. I’ll close by saying that we’re working on a number of ideas to make this release cycle as smooth and stress-free as possible to add-on developers. Most releases will change nothing you care about, so you shouldn’t need to worry about it. I strongly recommend all of you to track the Add-ons Blog, which is becoming an increasingly important source for updates. I also recommend that you give the Add-on SDK a shot. If your add-on can be implemented using the SDK, it’s better for you that it is.

Our add-on performance initiative is getting lots of attention for, lets say, various reasons. There have been objections about transparency and our testing methods, so I decided to add something valuable to the discussion and document my own testing process.

I revisited my old add-on performance article and noticed that the contents of the Measuring Startup wiki page have changed substantially since I originally linked to it. It now recommends installing an add-on  to measure startup performance. I haven’t tried it, but there are a few reasons I can think this is not the best approach. (Update: I’ve been informed that the add-on is only a display for data that is gathered and stored locally. You can make test runs and then install the add-on to look at the data. That dispels my previous doubts about this approach.) Regardless, I’m documenting the old testing method here, because it is the one I have been using for a while and is also very similar to the one implemented on our automated Talos testing framework.

I have been doing lots of add-on startup testing recently, mainly to double check if the results of the Talos tests are sound. We also correlate them to real world usage data that we have been collecting since early versions of Firefox 4. This data, manual testing and source code review are the main backup sources that give us a good confidence level in the results we display on our infamous performance page (it has been linked enough).

Here’s what I do.

Setup

1. Create a new profile dedicated for testing add-on performance (I called it startuptest).
2. Download this HTML page and save it somewhere convenient. The page is blank if you open it directly. All it does is run some JS that extracts a timestamp after the # character in the URL, compares it against the timestamp when the script is run, and shows the difference on the page.
3. Set up a console command that opens Firefox in your testing profile and opens the downloaded file, with the current timestamp embedded after the # character. On my system (Mac OS), this command is the following:

/Applications/Firefox.app/Contents/MacOS/firefox-bin -P startuptest -no-remote file:///Users/jorge/startup.html#`python -c 'import time; print int(time.time() * 1000);'`

The old version of the Measuring Startup page explains how to set this up on Windows.

Testing

1. Locate the testing profile folder and delete all files in it, if there are any.
2. Open Firefox on this profile. You can use the console command or any other shortcut if you prefer.
3. Copy and paste the add-on listing page URL on the new profile and open the page.
4. Install the add-on using the install button and restart if necessary.
5. Optionally, set up the add-on in a realistic way. For example, if this is a Facebook add-on, it may make sense to log in to a Facebook account since otherwise most of the add-on’s functionality would be inactive.
6. Quit Firefox.
7. Run Firefox using the console command.
8. Note the result in the startup page.
9. Quit Firefox. I prefer using the Quit key shortcut, to interact with Firefox as little as possible.
10. Repeat steps 7-9. I discard the 2 first runs, which are normally much slower than the rest, and measure the 10 runs after that.

Interpreting results

For your results to make any sense, it’s also necessary to make a test run without any add-ons installed, and use that as your baseline. It’s also a good idea to run all the tests consecutively to have some certainty that they are all running under similar conditions. I record and compare my results on a spreadsheet, like this one where I tested both my add-ons.

Looking at the results, Fire.fm has a somewhat noticeable impact on startup. This is not surprising because it is a complex add-on with a very complex overlay and startup processes. I documented on improving startup code in my old blog post, and we’re planning on greatly simplifying its overlay soon(ish). I doubt we’ll make the coveted 5%, but we’ll see. Remote XUL Manager is clearly simpler, and it shows how the results should not be taken at face value. Since all it does in the overlay is add a menu item that opens a separate window, it’s understandable that its impact is negligible. But does it really improve startup? No, of course not. This just means that the error margin is larger than its real performance impact.

The key takeaway here is that the results of manual tests shouldn’t be taken literally, but they’re still a good indicator of the performance impact of an add-on. Even if the error margin is not ideal (or even measurable under these conditions), you can still get a good idea of who’s fast and who’s slow. They have been very valuable to us when comparing them against Talos results.

How does this compare to Talos?

On one hand, these tests are influenced by how the testing system is set up. I have several applications open at all times, and I don’t close them all for testing. I do take care in not running anything heavy simultaneously, like Time Machine or MobileMe Sync. And then there’s clearly the fact that I have to spend some time setting things up and running the tests. The longer the tests take, the more likely it is that some other process affects the results.

On the other hand, it’s easier for me to recognize errors during testing. Many of the complaints we’ve received about the testing system is that it makes silly mistakes like trying to install an add-on from an incorrect URL, or trying to install an add-on that is not compatible with the Firefox version being tested. These are things that one can clearly see when testing manually, but they weren’t obvious when running the tests automatically. Those add-ons have been getting very good performance rankings because they’re not really being loaded, so those results are not reliable.

Luckily, the people complaining about our testing are also filing bugs and talking to us directly, so we’re looking into the issues and trying to get them resolved as soon as possible. Special thanks to Wladimir and Nils, who have been very helpful filing and categorizing bugs. More details coming up in the Add-ons Blog.

As always, the developer community proves itself as an invaluable asset for Mozilla (well, you are Mozilla). Even if our discussions can become harsh and are generally very public, the outcome is almost always a set of improvements both in our technical and communication fronts. Getting things right take a lot of work and a lot of patience, and I hope we can quickly get to a place where we’re all satisfied.

I’ve been meaning to write about this for a while now, but I just haven’t had the time for it given all the work I’ve been doing surrounding the Firefox 4 release. The editor review queues are predictably getting much more submissions, and I’ve been trying to keep add-on developers up to date with Firefox 4 breaking changes.

A while ago I suggested a different packaging system for add-ons that would (arguably) make it easier to work with. I proposed it as a project for the Google Summer of Code, and it was accepted. That was kind of unexpected, and I was very excited to be part of that experiment. There was a good student submission for the project and everything looked well underway.

Fast forward a few months, and I was contacted by the student working on the project. He wanted to drop it due to lack of time and progress . After chatting for a while, I agreed with him that dropping it was for the best. In the end the problem was a mixture of poor communication and a scheduling mismatch between the summer break in the US and other countries (Brazil in this case). He just couldn’t focus on the project due to school work.

So, the experimental implementation never happened, but the idea lives on. The project’s approval motivated me to create a pretty decent spec that I’ll update once Firefox 4 is released. Many changes have been introduced in Firefox 4 that affect manifest files, and others that overlap with some of the proposed features. Once Firefox 4 is stable for add-on developers, I’ll update the wiki and probably post an update here. I don’t have any plans to implement this or push the idea too hard, but I still think there’s some value there and it’s worthwhile to keep it alive.

I’ve been spending some time working on the new wiki for the Add-on Packaging Spec. Here it is.

At the moment I’m writing the complete manifest file specification, basically copying and adapting the current specifications for install manifests and chrome manifests. I’m pretty much done with the former, and will be adding the latter probably tomorrow.

After receiving lots of feedback regarding the manifest format, I agreed to take the manifest file the XML way. I followed JJ Barton’s suggestion of using HTML instead of a custom XML format, which leads to a very simple and familiar manifest format, with some very nice visualization options. This is the first time I’ve used the :before pseudo-class and the CSS content property, and I must say that they blew me away, limited as the latter is. Cool stuff.

If you want to discuss the pros and cons of the HTML manifest format, please fire away in the comments. I will be adding more content to the wiki as the project progresses.