A while ago I suggested a different packaging system for add-ons that would (arguably) make it easier to work with. I proposed it as a project for the Google Summer of Code, and it was accepted. That was kind of unexpected, and I was very excited to be part of that experiment. There was a good student submission for the project and everything looked well underway.

Fast forward a few months, and I was contacted by the student working on the project. He wanted to drop it due to lack of time and progress . After chatting for a while, I agreed with him that dropping it was for the best. In the end the problem was a mixture of poor communication and a scheduling mismatch between the summer break in the US and other countries (Brazil in this case). He just couldn’t focus on the project due to school work.

So, the experimental implementation never happened, but the idea lives on. The project’s approval motivated me to create a pretty decent spec that I’ll update once Firefox 4 is released. Many changes have been introduced in Firefox 4 that affect manifest files, and others that overlap with some of the proposed features. Once Firefox 4 is stable for add-on developers, I’ll update the wiki and probably post an update here. I don’t have any plans to implement this or push the idea too hard, but I still think there’s some value there and it’s worthwhile to keep it alive.

At the moment I’m writing the complete manifest file specification, basically copying and adapting the current specifications for install manifests and chrome manifests. I’m pretty much done with the former, and will be adding the latter probably tomorrow.

After receiving lots of feedback regarding the manifest format, I agreed to take the manifest file the XML way. I followed JJ Barton’s suggestion of using HTML instead of a custom XML format, which leads to a very simple and familiar manifest format, with some very nice visualization options. This is the first time I’ve used the :before pseudo-class and the CSS content property, and I must say that they blew me away, limited as the latter is. Cool stuff.

If you want to discuss the pros and cons of the HTML manifest format, please fire away in the comments. I will be adding more content to the wiki as the project progresses.

Part 3 – Installation

Add-on performance has been a hot topic recently, given that add-ons are a significant contributor to slowdowns, specially at startup. One of the reasons I’ve seen mentioned is how extensions are laid out in the file system after installation. The suggestion is to educate developers on how to package add-ons taking performance into account. That’s not a bad approach, but I think it’s easier for developers and more effective in general if the post-install procedure handles performance optimizations. This might even be necessary, considering that different applications and different platforms can have different ways to optimize for performance.

The add-on installer should package most if not all files into a single, uncompressed JAR file, as suggested in this bug report:

• Add-ons in the new format should not have internal chrome JARs.
• All files that can be efficiently read from a JAR file will be moved into a single uncompressed JAR (like a TAR) upon installation. This is a single JAR for all add-ons using the new system. One possible “cheat” to accomplish this is to create a fake add-on install directory, with a single JAR and a generated manifest that includes the information for all add-ons contained in it. This manifest could even be an “old style” chrome.manifest, in case changing chrome resolution rules turns out to be too difficult.
• Files that can’t be put in the JAR (this is mentioned in the bug report, and I don’t know why this is the case), such as binary libraries and components should ideally be grouped together in the same place where the big JAR will be located. One concern here would be file name collisions and the way these files are automagically loaded from their special folders, which may be hard to overcome. In case this is too difficult, it’s probably best to lay them out like it’s done now: on their separate extension directories.

From a performance perspective, this is what makes most sense for all platforms as far as I know, so it would be the default and maybe only behavior. However, if different packaging requirements come up for specific platforms, it can and should be implemented, with the great advantage that it will be possible to do so, at least with add-ons using the new system.

Spec corrections

• By far, the most debated point in this spec has been the format chosen for the manifest. I’m convinced now, JSON was not the right way to go. I really like John J Barton’s idea of an HTML manifest, so I think I will pursue this approach. It makes sense due to familiarity, IDE support and even skinability. Somebody mentioned that I should also cover update.rdf and McCoy. This will be done in the new XML-based spec, and it should be very straightforward.
• In part 2 I mentioned that having internal JARs would still be possible. I’m reconsidering this decision. Since the new format is meant to be a fresh start in several regards, I’m thinking it would be a bad idea to drag these old practices back, considering that they would have a negative effect on the post-install process.

The final project spec will be moved to a wiki, where it makes more sense to have a document that will likely see many iterations and corrections. I will being working on this as soon as possible, and will post a link in this blog once there’s something everybody can look at.

I will also post updates when there’s more progress on the project.

Acknowledgements

• Google, for creating the very cool GSOC and keeping it alive for many years.
• Gerv, choffman, and all GSOC admins.
• Hebert Duarte, for volunteering to implement this project.
• Everyone who has been posting comments and giving me feedback. It’s all very valuable and not a single comment has gone unread.

Thank you all!

Part 2 – URL resolution

Here’s a simple chrome.manifest file, as exemplified in part 1:

content   xulschoolhello              jar:chrome/xulschoolhello.jar!/content/
skin      xulschoolhello  classic/1.0 jar:chrome/xulschoolhello.jar!/skin/
locale    xulschoolhello  en-US       jar:chrome/xulschoolhello.jar!/locale/en-US/

overlay chrome://browser/content/browser.xul  chrome://xulschoolhello/content/browserOverlay.xul

There are two things I find silly about it:

1. The file and directory structure is too complicated for a simple add-on. It could be simpler, but the form above is the current recommended way of doing things.
2. If this is the default, recommended way of doing things, then why is it necessary to explicitly declare it?

This part of the specification aims to remove as much redundancy as possible from chrome (and other) declarations, as well as provide sensible defaults so that simple add-ons are simple to develop.

The simplest manifest

Let’s begin with the new manifest file introduced in the previous part of the spec:

{
  "id" : "helloworld@xulschool.com",
  "name" : "XUL School Hello World",
  "description" : "Welcome to XUL School!",
  "version" : "0.1",
  "authors" : "Appcoast",
  "homepageURL" : "https://developer.mozilla.org/en/XUL_School",
  "type" : "2",
  "targetApplications" : {
    "id" : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
    "minVersion" : "3.0",
    "maxVersion" : "3.6.*"
  },
  "domains" : "xulschoolhello",
  "overlays" : {
    "source" : "chrome://xulschoolhello/content/overlay.xul",
    "target" : "chrome://browser/content/browser.xul"
  }
}

A couple of notes before diving in:

• There’s an ongoing discussion of what format is best for the manifest, JSON or some form of XML or HTML. Examples are being presented as JSON but could change eventually. If you want to join the discussion, please do so in Part 1.
• The “domains” property is now plural, as it should be. (Thanks Wladimir!)

The highlighted sections are the equivalent of the chrome.manifest file presented above. The “domains” declaration covers the “content”,  “skin” and “locale” declarations in the old format. However, the file layout also needs to change, since the defaults for the new format are much simpler than in the old one. Here are the default resolutions for all URLs and other special folders:

• “content” and “skin” URLs default to the root directory, where the manifest file is located. This means that the simplest of add-ons would consist of the manifest file and an overlay file zipped together. Complex add-ons would of course require directory structures in order to keep everything organized, but that’s left to the authors to determine.
• “locale” URLs are a little more involved. If the author only wishes to ship the add-on in one locale, then the locale files can be located in the root directory, just like “content” and “skin” files. If there are more languages, they can be organized under “/locale/en-US/” and similar. The “locale” directory in the root would have that special meaning, and all directories within it are considered locale folders and are automatically registered. There’s no need for chrome declarations even in this case.
• “resource” URLs will also default to the root directory, and will use the same domain. In the example extension above, “resource://xulschoolhello/hello.jsm” would point to the “hello.jsm” file in the root of the add-on package.
• The “components” and “platform” folders will continue to have their special meaning.
• The default preferences file will be named “defaultPrefs.js” and will also reside in the package root.

Advanced manifests

The defaults defined above should be adequate for most add-ons I’ve ever seen (and I’ve seen a lot!), but we can’t leave out add-ons that require more than this. It’s very important that the new system is as flexible as possible for authors, and that add-ons can be migrated to it without requiring significant changes or refactoring.
First of all, the “domains” attribute can be extended to include declarations similar to old-style manifests. Here’s the equivalent of the first chrome.manifest declarations in the first example:

"domains" : {
  "name" : "xulschoolhello",
  "content" : {
    "location" : "jar:chrome/xulschoolhello.jar!/content/"
  },
  "skin" : {
    "location" : "jar:chrome/xulschoolhello.jar!/skin/"
  },
  "locale" : {
    "locale" : "en-US",
    "location" : "jar:chrome/xulschoolhello.jar!/locale/en-US/"
  }
},

Some notes about this expanded format:

• “domains” can be an array, in case multiple domains need to be defined.
• The name of the skin is still redundant, so there’s no need to specify it.
• “content”, “skin”, “locale” and “resources” can be arrays, to specify multiple entries.
• “content”, “skin”, “locale” and “resources” support a “flags” attribute, in order to support manifest flags. This attribute can also be specified on overlays, overrides and style declarations, which are not exemplified but are the same as the overlay declaration shown before.

Moreover, it should be possible to change the locations of any of our “special” files and folders, in order to give developers maximum flexibility. For this, the manifest can have a “locations” attribute, exemplified here:

"locations" : {
  "defaultPrefs" : "prefs/myPrefs.js",
  "components" : "code/myComponents/",
  "platform" : "code/platform/"
},

The only file that has a mandated name and location would be the manifest file. The rest can be organized to the authors’ preference, provided they don’t like the defaults and they want to tinker with the manifest.

Manifest Localization

Localizing manifests is an arduous task. The most modern way to do it involves adding all the localized information in the manifest file, which isn’t yet supported by the most popular add-on localization community, Babelzilla. Keeping the manifest properly localized becomes a burden to add-on developers, and most don’t even bother.

The suggested solution in the new system is to take advantage of the existing localization infrastructure. The manifest localization file will default to the location “chrome://firstDeclaredDomain/locale/manifest.properties”, or can be explicitly defined (needs to be a chrome locale URL):

"localized" : "chrome://xulschoolhello/locale/manifest.properties",

This localized file would hold strings corresponding to the localizable attributes:

name = Hola Mundo de XUL School
description = Bienvenido a XUL School!
translator.1 = Jorge
translator.2 = Somebody else

The .1 and .2 suffix is just a way to handle multi-valued attributes.

Moving this information out of the manifest has one drawback: the add-on installer needs to do some additional effort to show these values during installation, since it will require to figure out the location of the localized file. This isn’t too hard to figure out using the manifest, though, and the advantages outweigh a few additional ms during installation.

Part 3 of this specification will cover add-on installation and any pending topics. It will also introduce a wiki page where the full spec will be fleshed out and all edge cases will be covered.

Part 1 – The Manifest File

Most add-ons rely on 2 manifest files: install.rdf and chrome.manifest. The install manifest holds add-on metadata such as its name, description, icon path and compatibility information. The chrome manifest determines where to locate chrome files under different circumstances (OS, application, version, etc.), as well as overlays, skins and locales.

These files are written in very different text formats, and they are not completely independent from each other. For example, the install manifest uses chrome paths to locate the extension icon, preferences window and about window, if defined. Install manifests also hold localized data which hasn’t been integrated properly into the add-on localization system. It’s also worth noting that the average add-on has very short and simple manifest files, including a significant amount of boilerplate code.

In order to simplify the packaging system, both manifest files should be merged into a single file. The file specification should allow for the least amount of unnecessary declarations that is possible, so that simple add-ons are simple to make. It should also be robust enough not to limit complex add-on creation, and it should cover every single non-obsoleted feature of both current manifest files. It should work for all add-ons that currently use the XPI format: extensions, themes, locale, and multi-item packages.

The chosen format for the manifest file is JSON. The reasons behind this decision are:

• JSON is a very compact and easy to read format.
• It’s widely used and well known.
• Firefox and Mozilla-based applications already include a fast and safe JSON parser.
• The Jetpack project uses JSON for its manifest file, and using the same base for all add-ons helps maintain consistency and ease switching between platforms if needed.

The only major disadvantage is not being able to comment in JSON files. There are workarounds, but they’re not very pretty.

The manifest file will be named manifest.json. It’s not package.json like previously speculated because it doesn’t serve the exact same purpose as package manifests in Jetpack, and it would cause confusion about the meaning of packages. It’s not install.json because it could be easily confused with the old install.js manifests.

The presence of manifest.json in the XPI file will indicate this file needs to be handled as a v2 package. If manifest.json isn’t found, the current install system should be used. XPIs should be able to include old and new manifest files in order to be backward-compatible with application versions that don’t support the new system.

A simple manifest file

A very simple Hello World add-on (taken from XUL School) would have the following manifest files.

install.rdf

<?xml version="1.0"?>

<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:em="http://www.mozilla.org/2004/em-rdf#">
  <Description about="urn:mozilla:install-manifest">
    <em:id>helloworld@xulschool.com</em:id>
    <em:name>XUL School Hello World</em:name>
    <em:description>Welcome to XUL School!</em:description>
    <em:version>0.1</em:version>
    <em:creator>Appcoast</em:creator>
    <em:homepageURL>https://developer.mozilla.org/en/XUL_School</em:homepageURL>
    <em:type>2</em:type>
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id>
        <em:minVersion>3.0</em:minVersion>
        <em:maxVersion>3.6.*</em:maxVersion>
      </Description>
    </em:targetApplication>
  </Description>
</RDF>

chrome.manifest

content   xulschoolhello              jar:chrome/xulschoolhello.jar!/content/
skin      xulschoolhello  classic/1.0 jar:chrome/xulschoolhello.jar!/skin/
locale    xulschoolhello  en-US       jar:chrome/xulschoolhello.jar!/locale/en-US/

overlay chrome://browser/content/browser.xul  chrome://xulschoolhello/content/browserOverlay.xul

The equivalent manifest file in the new system would be the following.

manifest.json

{
  id : "helloworld@xulschool.com",
  name : "XUL School Hello World",
  description : "Welcome to XUL School!",
  version : "0.1",
  authors : "Appcoast",
  homepageURL : "https://developer.mozilla.org/en/XUL_School",
  type: "2",
  targetApplications : {
    id : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
    minVersion : "3.0",
    maxVersion : "3.6.*"
  },     
  domain : "xulschoolhello",
  overlays : {
    source : "chrome://xulschoolhello/content/overlay.xul",
    target : "chrome://browser/content/browser.xul"
  }
}

The (not really) missing chrome declarations will be covered in part 2 of this specification.

Plural attributes in the manifest file (authors, targetApplications, overlays, and others) can either be a single value or object, or an array of them. So, if there were multiple authors to the add-on, the following would also work:

authors : [ "Appcoast", "Mozilla" ],

Part 2 of this specification will attempt to cover all possible attributes in the manifest file, as well as attribute localization.

But then the Google Summer of Code came to the rescue. I believe Gerv mentioned it in one of our weekly all hands meetings, and I thought “it’s a long short, but what the hell”. I just threw the idea into the candidates list. In a matter of a couple of weeks, the idea went from candidate to proposal, to approved proposal, to approved proposal with an assigned student. So, this summer, Hebert Duarte from Brazil will be working on the add-on packaging idea, and hopefully we will have a working implementation within a couple of months. That’s awesome!

Now it’s my responsibility to guide Hebert in the right direction, and to give him all he needs to get this project going. For that I will be posting a more detailed specification of the new add-on packaging system, to try to cover the quirks and special cases that need to be addressed. Expect it to appear on this blog very soon. Coding begins on May 24th, so I want to have everything figured out by then. Your feedback as usual is welcome.

Note that having a working implementation doesn’t guarantee it will become part of Firefox or other applications any time soon. At the moment there are no plans or expectations of when and if we will be including this into the Mozilla code base. That’s a whole other project

I keep feeling the same about a side project of mine that began shortly before I joined Mozilla. A long time friend of mine has lead a heavy metal band called Pneuma for quite a few years now. They had to break ties with their production company recently because of unacceptable delays on their part, and lots of the typical crap professional musicians have to deal with. They were looking for people to invest in the independent production of their next album, and I guess I was just in the right place at the right time.

Of course I wouldn’t invest in them if I didn’t think they were really good.  I have great expectations for their upcoming album and I’m helping them in every way I can. The best way I can help them is by bringing my knowledge of the web and technology to the table. So far I’ve helped them set up their temporary website and twitter account. I will also be helping them out with their online distribution, because not having your album listed on the iTunes Store is a very new and very big mistake a band can easily make.

But I digress. The point of this post is that I decided to use HTML5 on their site, and provide their first single in open OGG/Vorbis format (falling back to MP3 and Quicktime), with the added bonus that the band agreed to have the track available for download and free distribution. So anybody can listen and download the track in their preferred format, MP3 or OGG. The audio player looks fantastic on the site, by the way. Much better than QuickTime or WMP plugins. The audio quality is also great, and the file is much smaller than its MP3 counterpart.

Adding also to the surrealism of the whole “music producer” side of my life is the fact that Pneuma will be the opening act at the first Metallica concert in Costa Rica, next Sunday. This is the biggest concert in Costa Rican history, and I’m pretty sure I’ll be backstage. I haven’t really taken that in yet, there’s just too much work to do before the concert. But WOW.

If you like thrash / progressive metal, I recommend you visit http://pneumametal.com/ and have a quick listen. I think you’ll be pleasantly surprised.

1. Most of all, I’ve been contributing to the AMO editor team to keep the add-on review queues down to a manageable size. There’s a great deal of new submissions and updates due to the release of Firefox version 3.5. There are hundreds of updates that are minimal to non existent compatibility changes, so I’ve been able to review close to 200 updates on this month alone. I enjoy my work with the editor team, specially because I’ve very fond of doing code review and being ruthlessly critical. It’s in my nature, what can I say… There are plans for removing the AMO sandbox, which I find intriguing, to say the least. I wonder how that will change the editor group and their work.
2. The Extend Firefox 3.5 competition has been officially announced. With an October deadline, it includes several new interesting categories to compete in. Jose and I are already working on Fire.fm 1.3, which will be our entry for best extension update. There’s a great deal of new features we’ll be putting into it, which I think will greatly improve our user experience. We’re also working on a few ideas for new add-ons. I hope to be able to submit a couple of entries to increase our chances of winning.
3. Finally, the Add-ons Contributions pilot was introduced to AMO. This enabled add-on authors to request donations for their project right on AMO, giving our donation links more exposure and possibly allowing a few authors to remove donation requests from their add-on UI, which is always a loss for user experience. We have activated contributions for Fire.fm, and so far the response has been pretty good. We’re very happy about it and hope people will continue helping us out.

I still feel in ‘break mode’, and hope to start working in full gear soon. There’s so much I want to do, and I still need to get everything sorted out. More updates soon!

This project is a set of skins for the Personas extension. Personas is a new approach to developing themes for Firefox and other Mozilla apps. Creating a skin is pretty simple: all you need is a header image and a footer image. Creating a good one is a little harder; you’ll need to fill a very, very large image area, while at the same time keeping in mind that only a tiny fraction of the image will be visible in the majority of browsers. Patterns and other artificial designs are probably easier, but mine are just extracts from my large photo collection, so it’s trickier.

I have a list of the skins I’ve created in the project page, with explanations on why I chose the pictures. So far my designs are doing pretty well, with a few hundred active daily users at this time. You can see the user counts in my designer page.

My skins were all created using The Gimp, and the pictures were taken with 2 different models of the Canon PowerShot (I upgraded recently). Artistic feedback is greatly appreciated.

Anyway, at the time we were pretty excited about it, and we spent a great deal of time and effort making Fire.fm a sure winner. And we won! We won the Best Music Add-on category, and I think we weren’t allowed to win in others, because I think we should have won the general competition as well . Needless to say, we were super happy with this, and we have been actively maintaining the add-on in our spare time.

Fire.fm is, in a nutshell, a Last.fm radio player. It used to be a very easy way to tune into Last.fm radio without having to be on top of the website all day long. Our users loved it, and shortly we were in the AMO Recommended List.

Sadly, Last.fm cut off their free radio service to most of the world shortly afterward, so at least I lost a lot of the motivation I initially had to continue making it grow. Still, I don’t want to abandon a project that still has much to be fixed, so I’ll certainly won’t quit until all the annoying bugs are solved. Most users have a very smooth experience with Fire.fm, but there are a few unlucky ones that have some serious problems, like not being able to listen to stations at all… New features in Fire.fm are probably going to be very rare from version 1.2.3 and later.

Xulforge was already in my mind when we began working on Fire.fm, so I always kinda saw it as a Xulforge project. Xulforge was put on hold until very recently, so it feels weird trying to associate the two. I’ll keep Fire.fm in my project list, though, because I it was one of my first independent projects, and also because it has been very successful and I’m very proud of it.

Fire.fm is open source, BSD-licensed (more on that in another post). So, if you want to take a look into our code, you’re more than welcome. See the Fire.fm Sourceforge Project Page for more information.

Fire.fm Logo