Skip to content

XPI v2 – Making Extension Development Easier

Note: this is just me throwing some ideas around. It is not an official proposal or spec. Having said that, I would like everyone that has interest in extension development to read this post and tell me what they think.

I’ve been an extension developer for a long time, and I like to think that developing extensions is actually quite easy. Maybe it has to do with my C++/Java background, where setting up a development environment is much more involved than using a text editor and zipping some files together.
That doesn’t negate the fact that the Mozilla add-ons platform is old, and showing its age. There are a number of problems with it that make it hard to take the first steps into add-on development, and it’s amazing how little has been done to solve them over the past years. These are the top 3 in my mind:

  • Extensions can’t be installed or uninstalled without a browser restart.
    • That’s bug 256509. Can it be fixed? Probably, but it would take a major development effort.
  • The documentation about extension development is incomplete or outdated.
    • I’m working on this in the XUL School project, to be finished soon.
  • Getting started with a basic “hello world” extension is too much effort: install.rdf, chrome.manifest, chrome JAR, content, skin, locale, defaults, OMG!

The last point is the one I want to tackle in this post. There’s a ridiculous amount of boilerplate, redundant coding to do even for the most basic of add-ons, specially if you’re making an extension that should be skinable and localizable. These are the specific problems I’ve identified:

  • There are two manifest files: install.rdf and chrome.manifest, in completely different formats.
  • They are both defined in relatively obscure formats. The install.rdf file is one of the very few places where RDF is still used in Firefox. Sticking to a dying format (at least in this context) is a very bad idea.
  • The default chrome structure (with content, locale, skin, etc.) is too bureaucratic and inflexible, and almost completely redundant. Most add-ons have exactly the same structure, and having to define it every single time is unnecessary.

I think we can reimagine add-on packaging in way that simple tasks can be performed in the simplest of ways, and so that it can scale to be as fine-tuned as it is today. So here are my ideas.#1 Merge install.rdf and chrome.manifest into a single file. What format should be used? I think JSON is as good as any other, and Mozilla already includes a native and very fast JSON parser. This manifest file could also match the package.json manifests being used for Jetpacks. package.json is actually a pretty good name for the manifest file. Perhaps the same standard can be use for Jetpacks and other add-ons?

#2 Default to the root extension directory for chrome URLs in order to minimize chrome.manifest declarations. This means that a hello world extension could have this structure:

  • helloworld.xpi2
    • package.json
    • overlay.xul
    • overlay.js
    • overlay.dtd

And the manifest file would be something like:

{
  id : “helloworld@xulforge.com”,
  name : “Hello World!”,  
  type: “2”
  compatibility :    
    { id : “{ec8030f7-c20a-464f-9b0e-13a3a9e97384}”,
      minVersion : “3.5”,
      maxVersion : “3.6.*”},     
  domain: “helloworldchrome”,
  overlays:      
    { source : “chrome://helloworldchrome/content/overlay.xul”,
      target : “chrome://browser/content/browser.xul”}
}

(Note: ‘compatibility’ and ‘overlays’ can be arrays when there’s more than one item. And the ‘domain’ value is a general declaration of the chrome domain.)

In this new system, you would be able to have all of your chrome files in the root directory and have no need for chrome directives other than declaring your main overlay. Also, in your root directory you can have a locale or skin folder that the system would know how to handle without any changes to the manifest. If a file isn’t found in the locale or skin folder, then the system falls back to the root directory as a last resource. Of course it would still be possible to declare specific locations for content, locale and skin in the manifest, in order to allow the “old style” to be used.

What about other special locations?

  • platform and components will continue to have their special meanings.
  • JSM files can be handled just the same as chrome files, except that they use resource:// instead of chrome://.
  • The default preferences file should also be moved to the root and have a predetermined name, like defaultPrefs.js.

#3 Installing the package. When you install an XPI file, the file is unpacked in your profile directory. It is common (but not mandatory) practice for chrome files to be packed in a JAR file, which remains packed after installation. According to recent discussions about add-on performance, it’s more efficient to keep files packed together. Instead of requiring authors to use the internal JAR approach, I think it makes more sense to require authors *not* to use JARs, and then keep the packed XPI in the profile on installation. Files that need to be extracted, like the manifest (maybe) and binary components (according to this) can be extracted upon installation. This way it’s up to the platform and not the developer to look after performance.That’s it. Given that this new packaging system would be backwards-incompatible with the current one, it might make sense to change the file extension to something like XPI2, in order to make a clearer distinction between the 2. However it should suffice to look for the manifest file in order to identify the system in use.
How hard is this to implement? I think #1 and #3 are fairly simple to implement. #2 is the one that may present a bigger challenge, since the chrome URL system is something that runs deep in the Mozilla code, and changing its file location rules could cause breakage or vulnerabilities in non-add-on code. It might also be possible to limit the scope of these changes to add-ons only, but again, that may require lots of work. I’d love to hear the opinions of those who work on this part of the platform.

Some may wonder how does Jetpack fit into this whole idea. Well, Jetpack is a different platform, and it may very well replace traditional add-ons in the long term, but we’re still a long ways to go. We shouldn’t see Jetpack as some kind of competition, but as a lesson in what we can do better. If we improve add-on packaging to be closer to Jetpack packaging (I think this would be), then it’s a win for both, because it’ll make it less painful for developers to choose and switch between either platform.

I’d love to hear what experienced developers (both add-on and platform) think about this. I think there’s a real gain for novice developers in this if we were to implement it. I intentionally left out some details for brevity, but I’ll be happy to discuss them in the comments. Thanks for reading.

{ 17 } Comments

  1. Dave | 2010/03/23 at 1:42 PM | Permalink

    Leaving the XPI fully intact would mean decompressing things on each use, which probably wouldn’t be desired. Some sort of hybrid where maybe the installer is decompressed and a single package file is generated might be good. (i.e. something automatic similar to JAR packing, assuming associated bugs are fixed)

    While the subject is up it would be great if any new system gave us better compression options for the XPI. LZMA gives fantastic file size improvements over crappy old DEFLATE.

  2. jorge | 2010/03/23 at 2:08 PM | Permalink

    Well, most developers compress the JAR file anyway, so I wouldn’t consider that to be a loss. The typical way of generating the JAR and XPI is just to ZIP them together and rename the file, which is less than ideal as you point out.

    It’d be great to create a good add-on packager that knows about these performance optimizations, and is practical enough for developers in general to actually use it.

    Edit: thinking about it some more, it does make sense to run some repackaging optimization during installation, so that there’s no need to rely on the developer’s choices.

  3. Mardeg | 2010/03/23 at 2:28 PM | Permalink

    About the extension installing without restarts… I would have thought you subscribed to http://www.oxymoronical.com/blog/2010 where that work has been ongoing (See “Look Ma, no restarts!”)

  4. jorge | 2010/03/23 at 2:32 PM | Permalink

    @Mardeg: Yes, I’m aware of that, but it only applies to specialized extensions generated from Jetpacks. A *LOT* needs to be done in order to really support extensions with no restarts (like cleaning up event listeners, observers, remove dynamically added nodes, etc).

  5. James | 2010/03/23 at 2:44 PM | Permalink

    Actually, the way the restarts are being implemented regular XPIs can opt-in to being restartless. Since jetpacks are now just XPIs with a runtime, it seems like a lot of this can be done pretty easily.

  6. Asztal | 2010/03/23 at 2:45 PM | Permalink

    As I understand it, it’s not limited to Jetpacks:

    “This is of course to allow add-ons developed on the Jetpack platform to install without restarts but the feature is going to be available to any extension author, there are just some restrictions to how these extensions work.”

  7. jorge | 2010/03/23 at 2:52 PM | Permalink

    @James, Asztal: thank you for the clarification. I’m still skeptical though, considering that the linked bug (which comes down to dynamic overlays) hasn’t been fixed or touched for a long time. It may be possible to have restart-less extensions under some circumstances, but I doubt this will help in the general case. Remember that Jetpacks are a completely different breed, and they are easy to install and uninstall without restarting Firefox, by design.

  8. Nils Maier | 2010/03/23 at 6:50 PM | Permalink

    I agree that merging install.rdf and chrome.manifest into one single manifest (based on json or xml or whatever) is a nice idea.
    But it has to offer all the important capabilities of the two “legacy” formats.
    Some things that aren’t mentioned yet are:
    * em:localized
    * override
    * style
    * flags (application, appversion, os, contentaccessible, platform etc.)
    * window icons (chrome/icons/)
    We currently use (or at least will be using) all of these.

    Examples for somewhat more “complex” chrome.manifests I wrote:
    DownThemAll!: http://bugs.code.downthemall.net/trac/browser/trunk/chrome.manifest
    (two chrome “packages”, style, application, appversion, contentaccessible, os, osversion)
    MinTrayR: http://svn.tn123.ath.cx/mintrayr/trunk/jar.mn
    (override, application)

    I don’t see a problem with user defined locations, actually, nor do I really think some structure hurts in any way (so I’m not to fond of the idea to dump everything in the root).
    But document it better and make hard-coded stuff disappear:
    * Make icons location not hard-coded anymore (and one should be able to jar icons).
    * Make preferences not hard-coded anymore (along with some kind of fastload cache for preferences)
    * Maybe even make platform and components not hard-coded anymore.

    Thing is, that writing some location definitions shouldn’t be that hard to deal with. After all the authors manage to write code, so they should be able to write a few lines of manifest and create a few subfolders. ;)

    But it needs documented better, and old documentation should be revised/refined.
    One of the most irritating things is over-structured extensions like
    myExt/chrome/content/myExt/overlay.xul while myExt/content/overlay.xul would be perfectly fine
    (or jar:chrome.jar!/content/overlay.xul for that matter).
    I didn’t check, but it should
    But that’s mostly a matter of old documentation actually telling developers to use the former (expect the few instances of hard-coded paths mentioned above).

    Actually, it is already possible to dump everything in the root (again, except hard-coded stuff).
    I prepared the following as demo; my FastPrevNext repackaged to be “flat” having just an en-US directory:
    http://tn123.ath.cx/share/fastprevnext.xpi
    Of course, there was no need to use a jar, “content .” would have worked as well.

  9. johnjbarton | 2010/03/23 at 8:59 PM | Permalink

    jorge, while I am sympathetic to your goals, better support for non-jetpack extensions, we should be realistic. With so many resources going into jetpack, we should expect that any non-jetpack extension work is ‘on our own’. Furthermore, jetpack is focused on ‘simple’, newly created extensions. With that in mind, we should next ask: what kinds of features/changes would experienced existing extension developers collaborate on?

    My list includes:
    collaborative testing of extensions: we need to work together to work together,
    re-modularization: deconstruct jetpack to re-use the modularity infrastructure,
    lazy loading/unloading: lightweight infrastructure to support upgrades without restarts,

    Is there any hope for a “Union of Concerned Extension Developers”? Or are we really on our own?

  10. skierpage | 2010/03/23 at 10:53 PM | Permalink

    “Leaving the XPI fully intact would mean decompressing things on each use, which probably wouldn’t be desired”
    It can be if performance improves thanks to fewer open()s, seeks, and disk reads. Firefox can read things from zip files (thus XPI and jar) without decompressing the entire zip file, thanks to nsiZipReader and the jar: protocol. That’s the motivation for moving files into jars and the very cool omnijar work for Android.

    jar:file:///path/to/thefile.zip!/ , cool and well-hidden.

  11. jorge | 2010/03/24 at 12:16 AM | Permalink

    @Nils Maier: I think that in order to write your first extension, it’s easier to start as simple as possible. I agree that it can be messy to have everything in the root, and I know that it’s currently possible. What I want is for it to be the default. If you want folders, knock yourself out. My approach is to minimize the amount of work needed to make the simplest work. My experience is that the majority of add-ons are extremely simple, and would benefit from a simple structure.
    Of the manifest features you mentioned, the only I didn’t think about was em:localized. All of them can be easily included in the JSON format, although I would love to see some separation of manifest descriptions into locale files, since we’re doing a wish list anyway ;)

    @johnjbarton: I try not to think about this as an “us vs them” issue. But you are correct that something like this won’t materialize unless those interested really step up to the challenge. I’m trying to measure that interest with this post, partly. So far I’m pleased with what I see :). There are thousands of extension developers out there, and only tapping a fragment of that force should be enough to move things forward and improve the platform we rely on.

  12. johnjbarton | 2010/03/24 at 10:48 AM | Permalink

    @jorge You may not wish to think about these issues as “us vs them” but if you change the current extension system to be more accessible to first time users, then you are creating “us vs them” by competing with jetpack for their target audience. Whether you *intend* to compete or not does not change the fact that you are. This is the same situation as the new Mozilla Inspector/Console/CommandLine: they don’t *intend* to compete with Firebug, but they are addressing the same user needs.

    Assuming that jetpack succeeds, then the existing extensions will divide into two camps: those that can be re-written in jetpack and those that cannot. Most of the simpler extensions can and probably will be rewritten in jetpack, though mostly I think by new authors. A support or migration story for these extensions within the current extension system hardly makes sense. That is why I suggest focusing on extensions which cannot be supported by jetpack. But many extensions may fall in-between, so an even better strategy is to de-construct jetpack and re-use as much a possible. That gives us a migration story for the extensions or parts of extensions that fit in the middle ground.

  13. jorge | 2010/03/24 at 11:32 AM | Permalink

    @johnjbarton: you’re right about competition, it can be seen as “stealing Jetpack’s thunder”, even if that’s not the intent or the actual outcome.
    If Jetpack succeeds, and this will likely take more than just a few months, you’re correct in thinking that there will be a divide, and most simple extension development will be done with Jetpack, as it probably should. I see how it can be good to focus on advanced development, and maybe the best way to support that is through documentation and developer support channels. However, I still think it’s a good idea to lower the entry barrier for developers, given that nobody is born an advanced extension developer.
    You also mention re-using stuff from Jetpack. Well, I’m partly re-using their packaging system, to bring us up to par with their very low entry barrier.
    I have other ideas I want to share in the future on how we can also bring add-ons closer to be no-restart. But I think packaging is something that would be much simpler to tackle first. If I can get some backing on this idea, then I think that will give me motivation to direct larger goals, like getting to a point where we don’t need restarts and the main difference between extensions and Jetpacks will be the sandboxing and security model.
    I hope we can work together in making all of this happen. All it takes is a couple of patches ;)

  14. johnjbarton | 2010/03/24 at 10:46 PM | Permalink

    Did you have something in mind for “we can work together”?

    A general purpose ‘mindless’ solution to un-loading extensions (a pre-requisite for no-restart on update) is very difficult. jetpack’s approach is to implement undo over every API it offers, so for some time to come its API will be limited.

    I think if you give programmers a little help to implement undo, then unloading is not very hard. Most extensions add XUL, so they know what they added. I guess very few extensions modify browser XUL (themers do that maybe).

    Undoing an overlay may be as easy as walking the overlay DOM tree and removing the matching nodes. If extension developers have to add some ‘id’ attributes or so, its not such a big deal.

  15. jorge | 2010/03/24 at 11:14 PM | Permalink

    Yeah, that’s similar to what I think. I would like to see a general utility library (extension of FUEL, perhaps?) that provides functions to add event listeners, observers, and similar associated with the add-on ID. This would make it possible for the platform to automatically clean up all of them (undo) when the add-on is being removed or disabled. This combined with install / uninstall hooks for extensions so that they can run their custom start up and clean up code more easily would make it reasonable to implement dynamic overlays and completely fix the problem.
    I’d like to tackle all of these issues eventually, but frankly I don’t have the time for it at the moment. I really want to find a way to increase collaboration between add-on developers and platform developers so that the add-ons platform can evolve instead of stagnate or be rethought every few years… This is something I will begin to work on soon, maybe as soon as next month.

  16. rpl | 2010/05/13 at 2:54 PM | Permalink

    Hi jorge,
    I totally agree with your vision and I’m trying to help in getting some of the feature you are requesting
    into the main jetpack package.json:

    * https://bugzilla.mozilla.org/show_bug.cgi?id=564023

    I’m interested to evaluate how I can try to prototype an overlay attribute management into package.json, as in your example… now I know what I will do in the next weekend :-P

  17. jorge | 2010/05/13 at 3:00 PM | Permalink

    @rpl: that’s awesome! I’m very glad to see some collaboration and results so quickly. Keep it up!

Post a Comment

Your email is never published nor shared. Required fields are marked *