Add-on packaging – Manifests revisited and wiki

I’ve been spending some time working on the new wiki for the Add-on Packaging Spec. Here it is.

At the moment I’m writing the complete manifest file specification, basically copying and adapting the current specifications for install manifests and chrome manifests. I’m pretty much done with the former, and will be adding the latter probably tomorrow.

After receiving lots of feedback regarding the manifest format, I agreed to take the manifest file the XML way. I followed JJ Barton’s suggestion of using HTML instead of a custom XML format, which leads to a very simple and familiar manifest format, with some very nice visualization options. This is the first time I’ve used the :before pseudo-class and the CSS content property, and I must say that they blew me away, limited as the latter is. Cool stuff.

If you want to discuss the pros and cons of the HTML manifest format, please fire away in the comments. I will be adding more content to the wiki as the project progresses.

Add-on packaging spec, part 3

This is a specification for a project that will be implemented for the Google Summer of Code, and it can still change as feedback is received. There are no concrete plans of making this a part of Firefox in the near future. Read my previous post for more info.

Part 3 – Installation

Add-on performance has been a hot topic recently, given that add-ons are a significant contributor to slowdowns, specially at startup. One of the reasons I’ve seen mentioned is how extensions are laid out in the file system after installation. The suggestion is to educate developers on how to package add-ons taking performance into account. That’s not a bad approach, but I think it’s easier for developers and more effective in general if the post-install procedure handles performance optimizations. This might even be necessary, considering that different applications and different platforms can have different ways to optimize for performance.

The add-on installer should package most if not all files into a single, uncompressed JAR file, as suggested in this bug report:

• Add-ons in the new format should not have internal chrome JARs.
• All files that can be efficiently read from a JAR file will be moved into a single uncompressed JAR (like a TAR) upon installation. This is a single JAR for all add-ons using the new system. One possible “cheat” to accomplish this is to create a fake add-on install directory, with a single JAR and a generated manifest that includes the information for all add-ons contained in it. This manifest could even be an “old style” chrome.manifest, in case changing chrome resolution rules turns out to be too difficult.
• Files that can’t be put in the JAR (this is mentioned in the bug report, and I don’t know why this is the case), such as binary libraries and components should ideally be grouped together in the same place where the big JAR will be located. One concern here would be file name collisions and the way these files are automagically loaded from their special folders, which may be hard to overcome. In case this is too difficult, it’s probably best to lay them out like it’s done now: on their separate extension directories.

From a performance perspective, this is what makes most sense for all platforms as far as I know, so it would be the default and maybe only behavior. However, if different packaging requirements come up for specific platforms, it can and should be implemented, with the great advantage that it will be possible to do so, at least with add-ons using the new system.

Spec corrections

• By far, the most debated point in this spec has been the format chosen for the manifest. I’m convinced now, JSON was not the right way to go. I really like John J Barton’s idea of an HTML manifest, so I think I will pursue this approach. It makes sense due to familiarity, IDE support and even skinability. Somebody mentioned that I should also cover update.rdf and McCoy. This will be done in the new XML-based spec, and it should be very straightforward.
• In part 2 I mentioned that having internal JARs would still be possible. I’m reconsidering this decision. Since the new format is meant to be a fresh start in several regards, I’m thinking it would be a bad idea to drag these old practices back, considering that they would have a negative effect on the post-install process.

The final project spec will be moved to a wiki, where it makes more sense to have a document that will likely see many iterations and corrections. I will being working on this as soon as possible, and will post a link in this blog once there’s something everybody can look at.

I will also post updates when there’s more progress on the project.

Acknowledgements

• Google, for creating the very cool GSOC and keeping it alive for many years.
• Gerv, choffman, and all GSOC admins.
• Hebert Duarte, for volunteering to implement this project.
• Everyone who has been posting comments and giving me feedback. It’s all very valuable and not a single comment has gone unread.

Thank you all!

Add-on packaging spec, part 2

This is a specification for a project that will be implemented for the Google Summer of Code, and it can still change as feedback is received. There are no concrete plans of making this a part of Firefox in the near future. Read my previous post for more info.

Part 2 – URL resolution

Here’s a simple chrome.manifest file, as exemplified in part 1:

content   xulschoolhello              jar:chrome/xulschoolhello.jar!/content/
skin      xulschoolhello  classic/1.0 jar:chrome/xulschoolhello.jar!/skin/
locale    xulschoolhello  en-US       jar:chrome/xulschoolhello.jar!/locale/en-US/

overlay chrome://browser/content/browser.xul  chrome://xulschoolhello/content/browserOverlay.xul

There are two things I find silly about it:

1. The file and directory structure is too complicated for a simple add-on. It could be simpler, but the form above is the current recommended way of doing things.
2. If this is the default, recommended way of doing things, then why is it necessary to explicitly declare it?

This part of the specification aims to remove as much redundancy as possible from chrome (and other) declarations, as well as provide sensible defaults so that simple add-ons are simple to develop.

The simplest manifest

Let’s begin with the new manifest file introduced in the previous part of the spec:

{
  "id" : "helloworld@xulschool.com",
  "name" : "XUL School Hello World",
  "description" : "Welcome to XUL School!",
  "version" : "0.1",
  "authors" : "Appcoast",
  "homepageURL" : "https://developer.mozilla.org/en/XUL_School",
  "type" : "2",
  "targetApplications" : {
    "id" : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
    "minVersion" : "3.0",
    "maxVersion" : "3.6.*"
  },
  "domains" : "xulschoolhello",
  "overlays" : {
    "source" : "chrome://xulschoolhello/content/overlay.xul",
    "target" : "chrome://browser/content/browser.xul"
  }
}

A couple of notes before diving in:

• There’s an ongoing discussion of what format is best for the manifest, JSON or some form of XML or HTML. Examples are being presented as JSON but could change eventually. If you want to join the discussion, please do so in Part 1.
• The “domains” property is now plural, as it should be. (Thanks Wladimir!)

The highlighted sections are the equivalent of the chrome.manifest file presented above. The “domains” declaration covers the “content”,  “skin” and “locale” declarations in the old format. However, the file layout also needs to change, since the defaults for the new format are much simpler than in the old one. Here are the default resolutions for all URLs and other special folders:

• “content” and “skin” URLs default to the root directory, where the manifest file is located. This means that the simplest of add-ons would consist of the manifest file and an overlay file zipped together. Complex add-ons would of course require directory structures in order to keep everything organized, but that’s left to the authors to determine.
• “locale” URLs are a little more involved. If the author only wishes to ship the add-on in one locale, then the locale files can be located in the root directory, just like “content” and “skin” files. If there are more languages, they can be organized under “/locale/en-US/” and similar. The “locale” directory in the root would have that special meaning, and all directories within it are considered locale folders and are automatically registered. There’s no need for chrome declarations even in this case.
• “resource” URLs will also default to the root directory, and will use the same domain. In the example extension above, “resource://xulschoolhello/hello.jsm” would point to the “hello.jsm” file in the root of the add-on package.
• The “components” and “platform” folders will continue to have their special meaning.
• The default preferences file will be named “defaultPrefs.js” and will also reside in the package root.

Advanced manifests

The defaults defined above should be adequate for most add-ons I’ve ever seen (and I’ve seen a lot!), but we can’t leave out add-ons that require more than this. It’s very important that the new system is as flexible as possible for authors, and that add-ons can be migrated to it without requiring significant changes or refactoring.
First of all, the “domains” attribute can be extended to include declarations similar to old-style manifests. Here’s the equivalent of the first chrome.manifest declarations in the first example:

"domains" : {
  "name" : "xulschoolhello",
  "content" : {
    "location" : "jar:chrome/xulschoolhello.jar!/content/"
  },
  "skin" : {
    "location" : "jar:chrome/xulschoolhello.jar!/skin/"
  },
  "locale" : {
    "locale" : "en-US",
    "location" : "jar:chrome/xulschoolhello.jar!/locale/en-US/"
  }
},

Some notes about this expanded format:

• “domains” can be an array, in case multiple domains need to be defined.
• The name of the skin is still redundant, so there’s no need to specify it.
• “content”, “skin”, “locale” and “resources” can be arrays, to specify multiple entries.
• “content”, “skin”, “locale” and “resources” support a “flags” attribute, in order to support manifest flags. This attribute can also be specified on overlays, overrides and style declarations, which are not exemplified but are the same as the overlay declaration shown before.

Moreover, it should be possible to change the locations of any of our “special” files and folders, in order to give developers maximum flexibility. For this, the manifest can have a “locations” attribute, exemplified here:

"locations" : {
  "defaultPrefs" : "prefs/myPrefs.js",
  "components" : "code/myComponents/",
  "platform" : "code/platform/"
},

The only file that has a mandated name and location would be the manifest file. The rest can be organized to the authors’ preference, provided they don’t like the defaults and they want to tinker with the manifest.

Manifest Localization

Localizing manifests is an arduous task. The most modern way to do it involves adding all the localized information in the manifest file, which isn’t yet supported by the most popular add-on localization community, Babelzilla. Keeping the manifest properly localized becomes a burden to add-on developers, and most don’t even bother.

The suggested solution in the new system is to take advantage of the existing localization infrastructure. The manifest localization file will default to the location “chrome://firstDeclaredDomain/locale/manifest.properties”, or can be explicitly defined (needs to be a chrome locale URL):

"localized" : "chrome://xulschoolhello/locale/manifest.properties",

This localized file would hold strings corresponding to the localizable attributes:

name = Hola Mundo de XUL School
description = Bienvenido a XUL School!
translator.1 = Jorge
translator.2 = Somebody else

The .1 and .2 suffix is just a way to handle multi-valued attributes.

Moving this information out of the manifest has one drawback: the add-on installer needs to do some additional effort to show these values during installation, since it will require to figure out the location of the localized file. This isn’t too hard to figure out using the manifest, though, and the advantages outweigh a few additional ms during installation.

Part 3 of this specification will cover add-on installation and any pending topics. It will also introduce a wiki page where the full spec will be fleshed out and all edge cases will be covered.

Add-on packaging spec, part 1

This is a specification for a project that will be implemented for the Google Summer of Code, and it can still change as feedback is received. There are no concrete plans of making this a part of Firefox in the near future. Read my previous post for more info.

Part 1 – The Manifest File

Most add-ons rely on 2 manifest files: install.rdf and chrome.manifest. The install manifest holds add-on metadata such as its name, description, icon path and compatibility information. The chrome manifest determines where to locate chrome files under different circumstances (OS, application, version, etc.), as well as overlays, skins and locales.

These files are written in very different text formats, and they are not completely independent from each other. For example, the install manifest uses chrome paths to locate the extension icon, preferences window and about window, if defined. Install manifests also hold localized data which hasn’t been integrated properly into the add-on localization system. It’s also worth noting that the average add-on has very short and simple manifest files, including a significant amount of boilerplate code.

In order to simplify the packaging system, both manifest files should be merged into a single file. The file specification should allow for the least amount of unnecessary declarations that is possible, so that simple add-ons are simple to make. It should also be robust enough not to limit complex add-on creation, and it should cover every single non-obsoleted feature of both current manifest files. It should work for all add-ons that currently use the XPI format: extensions, themes, locale, and multi-item packages.

The chosen format for the manifest file is JSON. The reasons behind this decision are:

• JSON is a very compact and easy to read format.
• It’s widely used and well known.
• Firefox and Mozilla-based applications already include a fast and safe JSON parser.
• The Jetpack project uses JSON for its manifest file, and using the same base for all add-ons helps maintain consistency and ease switching between platforms if needed.

The only major disadvantage is not being able to comment in JSON files. There are workarounds, but they’re not very pretty.

The manifest file will be named manifest.json. It’s not package.json like previously speculated because it doesn’t serve the exact same purpose as package manifests in Jetpack, and it would cause confusion about the meaning of packages. It’s not install.json because it could be easily confused with the old install.js manifests.

The presence of manifest.json in the XPI file will indicate this file needs to be handled as a v2 package. If manifest.json isn’t found, the current install system should be used. XPIs should be able to include old and new manifest files in order to be backward-compatible with application versions that don’t support the new system.

A simple manifest file

A very simple Hello World add-on (taken from XUL School) would have the following manifest files.

install.rdf

<?xml version="1.0"?>

<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:em="http://www.mozilla.org/2004/em-rdf#">
  <Description about="urn:mozilla:install-manifest">
    <em:id>helloworld@xulschool.com</em:id>
    <em:name>XUL School Hello World</em:name>
    <em:description>Welcome to XUL School!</em:description>
    <em:version>0.1</em:version>
    <em:creator>Appcoast</em:creator>
    <em:homepageURL>https://developer.mozilla.org/en/XUL_School</em:homepageURL>
    <em:type>2</em:type>
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id>
        <em:minVersion>3.0</em:minVersion>
        <em:maxVersion>3.6.*</em:maxVersion>
      </Description>
    </em:targetApplication>
  </Description>
</RDF>

chrome.manifest

content   xulschoolhello              jar:chrome/xulschoolhello.jar!/content/
skin      xulschoolhello  classic/1.0 jar:chrome/xulschoolhello.jar!/skin/
locale    xulschoolhello  en-US       jar:chrome/xulschoolhello.jar!/locale/en-US/

overlay chrome://browser/content/browser.xul  chrome://xulschoolhello/content/browserOverlay.xul

The equivalent manifest file in the new system would be the following.

manifest.json

{
  id : "helloworld@xulschool.com",
  name : "XUL School Hello World",
  description : "Welcome to XUL School!",
  version : "0.1",
  authors : "Appcoast",
  homepageURL : "https://developer.mozilla.org/en/XUL_School",
  type: "2",
  targetApplications : {
    id : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
    minVersion : "3.0",
    maxVersion : "3.6.*"
  },     
  domain : "xulschoolhello",
  overlays : {
    source : "chrome://xulschoolhello/content/overlay.xul",
    target : "chrome://browser/content/browser.xul"
  }
}

The (not really) missing chrome declarations will be covered in part 2 of this specification.

Plural attributes in the manifest file (authors, targetApplications, overlays, and others) can either be a single value or object, or an array of them. So, if there were multiple authors to the add-on, the following would also work:

authors : [ "Appcoast", "Mozilla" ],

Part 2 of this specification will attempt to cover all possible attributes in the manifest file, as well as attribute localization.

Add-on Packaging Idea now a GSOC project

When I wrote my previous post about making add-on packaging simpler, I hoped but didn’t expect much to come out of it. It was more like taking a peek into the community and see how much interest there was for these kinds of proposal. I have a real desire to see things like this come to fruition, but not enough time or resources to do it myself. And it’s very difficult for a volunteer to handle a project of such magnitude.

But then the Google Summer of Code came to the rescue. I believe Gerv mentioned it in one of our weekly all hands meetings, and I thought “it’s a long short, but what the hell”. I just threw the idea into the candidates list. In a matter of a couple of weeks, the idea went from candidate to proposal, to approved proposal, to approved proposal with an assigned student. So, this summer, Hebert Duarte from Brazil will be working on the add-on packaging idea, and hopefully we will have a working implementation within a couple of months. That’s awesome!

Now it’s my responsibility to guide Hebert in the right direction, and to give him all he needs to get this project going. For that I will be posting a more detailed specification of the new add-on packaging system, to try to cover the quirks and special cases that need to be addressed. Expect it to appear on this blog very soon. Coding begins on May 24th, so I want to have everything figured out by then. Your feedback as usual is welcome.

Note that having a working implementation doesn’t guarantee it will become part of Firefox or other applications any time soon. At the moment there are no plans or expectations of when and if we will be including this into the Mozilla code base. That’s a whole other project