Firefox 4 for Add-on Developers

For now, this is a pretty good reference if you want to start supporting the Firefox 4 betas in your add-ons. I’ll be elaborating on this subject through the following weeks, in the Add-ons Blog.

I’ve been an extension developer for a long time, and I like to think that developing extensions is actually quite easy. Maybe it has to do with my C++/Java background, where setting up a development environment is much more involved than using a text editor and zipping some files together.
That doesn’t negate the fact that the Mozilla add-ons platform is old, and showing its age. There are a number of problems with it that make it hard to take the first steps into add-on development, and it’s amazing how little has been done to solve them over the past years. These are the top 3 in my mind:

• Extensions can’t be installed or uninstalled without a browser restart.
  • That’s bug 256509. Can it be fixed? Probably, but it would take a major development effort.
• The documentation about extension development is incomplete or outdated.
  • I’m working on this in the XUL School project, to be finished soon.
• Getting started with a basic “hello world” extension is too much effort: install.rdf, chrome.manifest, chrome JAR, content, skin, locale, defaults, OMG!

The last point is the one I want to tackle in this post. There’s a ridiculous amount of boilerplate, redundant coding to do even for the most basic of add-ons, specially if you’re making an extension that should be skinable and localizable. These are the specific problems I’ve identified:

• There are two manifest files: install.rdf and chrome.manifest, in completely different formats.
• They are both defined in relatively obscure formats. The install.rdf file is one of the very few places where RDF is still used in Firefox. Sticking to a dying format (at least in this context) is a very bad idea.
• The default chrome structure (with content, locale, skin, etc.) is too bureaucratic and inflexible, and almost completely redundant. Most add-ons have exactly the same structure, and having to define it every single time is unnecessary.

I think we can reimagine add-on packaging in way that simple tasks can be performed in the simplest of ways, and so that it can scale to be as fine-tuned as it is today. So here are my ideas.#1 Merge install.rdf and chrome.manifest into a single file. What format should be used? I think JSON is as good as any other, and Mozilla already includes a native and very fast JSON parser. This manifest file could also match the package.json manifests being used for Jetpacks. package.json is actually a pretty good name for the manifest file. Perhaps the same standard can be use for Jetpacks and other add-ons?

#2 Default to the root extension directory for chrome URLs in order to minimize chrome.manifest declarations. This means that a hello world extension could have this structure:

• helloworld.xpi2
  • package.json
  • overlay.xul
  • overlay.js
  • overlay.dtd

And the manifest file would be something like:

{
  id : “helloworld@xulforge.com”,
  name : “Hello World!”,  
  type: “2”
  compatibility :    
    { id : “{ec8030f7-c20a-464f-9b0e-13a3a9e97384}”,
      minVersion : “3.5”,
      maxVersion : “3.6.*”},     
  domain: “helloworldchrome”,
  overlays:      
    { source : “chrome://helloworldchrome/content/overlay.xul”,
      target : “chrome://browser/content/browser.xul”}
}

(Note: ‘compatibility’ and ‘overlays’ can be arrays when there’s more than one item. And the ‘domain’ value is a general declaration of the chrome domain.)

In this new system, you would be able to have all of your chrome files in the root directory and have no need for chrome directives other than declaring your main overlay. Also, in your root directory you can have a locale or skin folder that the system would know how to handle without any changes to the manifest. If a file isn’t found in the locale or skin folder, then the system falls back to the root directory as a last resource. Of course it would still be possible to declare specific locations for content, locale and skin in the manifest, in order to allow the “old style” to be used.

What about other special locations?

• platform and components will continue to have their special meanings.
• JSM files can be handled just the same as chrome files, except that they use resource:// instead of chrome://.
• The default preferences file should also be moved to the root and have a predetermined name, like defaultPrefs.js.

#3 Installing the package. When you install an XPI file, the file is unpacked in your profile directory. It is common (but not mandatory) practice for chrome files to be packed in a JAR file, which remains packed after installation. According to recent discussions about add-on performance, it’s more efficient to keep files packed together. Instead of requiring authors to use the internal JAR approach, I think it makes more sense to require authors *not* to use JARs, and then keep the packed XPI in the profile on installation. Files that need to be extracted, like the manifest (maybe) and binary components (according to this) can be extracted upon installation. This way it’s up to the platform and not the developer to look after performance.That’s it. Given that this new packaging system would be backwards-incompatible with the current one, it might make sense to change the file extension to something like XPI2, in order to make a clearer distinction between the 2. However it should suffice to look for the manifest file in order to identify the system in use.
How hard is this to implement? I think #1 and #3 are fairly simple to implement. #2 is the one that may present a bigger challenge, since the chrome URL system is something that runs deep in the Mozilla code, and changing its file location rules could cause breakage or vulnerabilities in non-add-on code. It might also be possible to limit the scope of these changes to add-ons only, but again, that may require lots of work. I’d love to hear the opinions of those who work on this part of the platform.

Some may wonder how does Jetpack fit into this whole idea. Well, Jetpack is a different platform, and it may very well replace traditional add-ons in the long term, but we’re still a long ways to go. We shouldn’t see Jetpack as some kind of competition, but as a lesson in what we can do better. If we improve add-on packaging to be closer to Jetpack packaging (I think this would be), then it’s a win for both, because it’ll make it less painful for developers to choose and switch between either platform.

I’d love to hear what experienced developers (both add-on and platform) think about this. I think there’s a real gain for novice developers in this if we were to implement it. I intentionally left out some details for brevity, but I’ll be happy to discuss them in the comments. Thanks for reading.

On Thursday, the update queue had about 80 add-on updates in line to be reviewed. This is a short as it’s been for a very long time. Today it stands close to 200 updates, and will continue to grow in the following couple of weeks. All authors are now rushing out updates because they just realized Firefox 3.6 is out.

Well, now it’s up to the editor team to catch up. It’ll take a while, I think. Authors that decided to update after the 3.6 release will now have to wait. It’s bad for everyone, and the result of bad communication. That’s something we’ll need to work on.

First of all, let me be very clear about this: there is no short term plan to eliminate the extension platform as we know it. XUL and XPCOM run deep in Firefox. They are Firefox. Eliminating the technologies that make extensions possible would require a rewrite of pretty much everything in the platform, which is massive. I’m not saying this couldn’t happen some time in the future (I don’t know, really), but it isn’t something that can be accomplished within a few weeks, or even a few months. It’s something that requires a great deal of planning and the collaboration of the whole community. Extension developers shouldn’t worry about their add-ons being obsolete overnight.

We should all look into the future, though. Not as something that we’ll have to accept, but as something we can shape. Jetpack and Personas are still experiments in many ways, and there’s much we can do to make them what we want them to be. I personally doubt they will ever reach the point where they will replace the current add-on options, but I am confident that they can come very close, and that’s a big win for everyone.

Jetpack

The goals of the Jetpack project are ambitious: no restarts for install / uninstall, a clean and more stable API, complete security, and a much easier development experience. They’re so ambitious that to think all of these can be accomplished while preserving the flexibility of the current platform would be naive at best.

There are no stable APIs. You can make higher abstractions that are less likely to change. But they will change. Jetpack only makes its add-ons dependent on its API, instead of the XUL/XPCOM platform. So, instead of updating your add-on to the next Firefox version, you’ll update it to the next version of Jetpack, which should happen much less often. That is of course assuming there will be some sort of versioning of the Jetpack API. If that’s not the case, well, then we have bigger problems to be concerned about.

Jetpack, unlike XUL and XPCOM, is not a fully open system. It can’t be. Not without sacrificing the security it’s meant to bring. As a secure system, it should be closed by default, enabling through its API only the features that are considered to be safe and necessary. This limits add-on creativity to the API designers’ imaginations, as opposed to the developers’. On the other hand, in the current system extensions can do pretty much anything. They can read, write and execute files. They can change your preferences and access your saved passwords. They can monitor your online activities and send information to third parties. The only real protection between you and the extensions you install is the review system that all of them have to go through in order to be publicly listed on AMO. A group of reviewers (also known as editors) make sure these extensions are safe to use and respect user choice. Which one is better? To me, the answer is simple: if it’s possible on Jetpack, use Jetpack. It’s simpler and safer. If it isn’t, then fall back to extensions, where you have almost limitless control.

Jetpack add-ons will be easier to develop, and it will be possible to install and remove them without restarts. This is a huge win for users and developers. Many, if not most add-ons will be easily portable to this new platform, and they will benefit from it. It remains to be seen, however, if highly complex (and extremely popular) add-ons like AdBlock Plus, NoScript and Firebug will be able to live in the Jetpack world. These extensions are strongly tied to the platform, and their interactions would be very hard to translate into a general use API. Maybe we can implement jetpack.magic.doWhatNoScriptDoes .

Personas

Personas are not even close to being a replacement for themes. Personas allow some very basic skinning using header and footer images and setting font colors for the main toolbox. Surely they could be extended to include images for the toolbar icons and some more advanced customizations, but that’s not the case now, and even then they wouldn’t be a complete replacement for themes. Themes can change the appearance of the application in very significant ways, and this can’t be accomplished without the complexity inherent in theme development. So, theme developers, you’re not done yet either.

The future

I think the future for extensions and themes is still bright. Jetpack and Personas have shown us how things can be different, and opening the field for new development and competition is a win for all.

There’s much we can do to improve the “classic” add-on world, in the area of documentation and tutorials, and even in the platform itself. We’re being shown how to improve. We should take this as a call to action, and improve. Let’s work on the platforms that will support the development of the future, but let’s not forget the ones that are still active and thriving.

Let’s not forget how we got here.

This discussion came up due to a denied update for Text Formating Toolbar 0.1.4.11. I was requested to come up with a POC that demonstrates how name conflicts can cause problems, so I created this add-on, which conflicts with Text Formatting Toolbar using seemingly harmless code. Here’s the link:

Name Conflict

If you install this with the Text Formatting Toolbar, you’ll notice that the bold, italics, code and color features are broken.

Here’s the code in my overlay:

window.addEventListener(
  "load",
  function() {
    window.setTimeout(
      function () {
        formatItalics = "i";
        formatBold = "b";
        formatColor = "white";

        formatCode =
          function(aString) {
            if (typeof(aString) != "string") alert('Invalid Code!'); }
      },
      100);
  },
  true);

As you can see, it’s fairly simple. The load event and setTimeout call are only used to make sure that it is the other extension and not this one the one that experiences the compatibility problems. If two add-ons use the same names in the global scope, then one is going to experience problems in one way or another. There’s nothing more to it.

Obviously I can intentionally create an add-on that causes conflicts with any other add-ons, but I hope you see past this and realize that the code I posted could very well be written by another author who is unaware of the rules we enforce. This is what we look after, and this is why we should reject global namespace pollution in almost all instances.

1. Most of all, I’ve been contributing to the AMO editor team to keep the add-on review queues down to a manageable size. There’s a great deal of new submissions and updates due to the release of Firefox version 3.5. There are hundreds of updates that are minimal to non existent compatibility changes, so I’ve been able to review close to 200 updates on this month alone. I enjoy my work with the editor team, specially because I’ve very fond of doing code review and being ruthlessly critical. It’s in my nature, what can I say… There are plans for removing the AMO sandbox, which I find intriguing, to say the least. I wonder how that will change the editor group and their work.
2. The Extend Firefox 3.5 competition has been officially announced. With an October deadline, it includes several new interesting categories to compete in. Jose and I are already working on Fire.fm 1.3, which will be our entry for best extension update. There’s a great deal of new features we’ll be putting into it, which I think will greatly improve our user experience. We’re also working on a few ideas for new add-ons. I hope to be able to submit a couple of entries to increase our chances of winning.
3. Finally, the Add-ons Contributions pilot was introduced to AMO. This enabled add-on authors to request donations for their project right on AMO, giving our donation links more exposure and possibly allowing a few authors to remove donation requests from their add-on UI, which is always a loss for user experience. We have activated contributions for Fire.fm, and so far the response has been pretty good. We’re very happy about it and hope people will continue helping us out.

I still feel in ‘break mode’, and hope to start working in full gear soon. There’s so much I want to do, and I still need to get everything sorted out. More updates soon!