Skip to content

Add-on packaging spec, part 1

This is a specification for a project that will be implemented for the Google Summer of Code, and it can still change as feedback is received. There are no concrete plans of making this a part of Firefox in the near future. Read my previous post for more info.

Part 1 – The Manifest File

Most add-ons rely on 2 manifest files: install.rdf and chrome.manifest. The install manifest holds add-on metadata such as its name, description, icon path and compatibility information. The chrome manifest determines where to locate chrome files under different circumstances (OS, application, version, etc.), as well as overlays, skins and locales.

These files are written in very different text formats, and they are not completely independent from each other. For example, the install manifest uses chrome paths to locate the extension icon, preferences window and about window, if defined. Install manifests also hold localized data which hasn’t been integrated properly into the add-on localization system. It’s also worth noting that the average add-on has very short and simple manifest files, including a significant amount of boilerplate code.

In order to simplify the packaging system, both manifest files should be merged into a single file. The file specification should allow for the least amount of unnecessary declarations that is possible, so that simple add-ons are simple to make. It should also be robust enough not to limit complex add-on creation, and it should cover every single non-obsoleted feature of both current manifest files. It should work for all add-ons that currently use the XPI format: extensions, themes, locale, and multi-item packages.

The chosen format for the manifest file is JSON. The reasons behind this decision are:

  • JSON is a very compact and easy to read format.
  • It’s widely used and well known.
  • Firefox and Mozilla-based applications already include a fast and safe JSON parser.
  • The Jetpack project uses JSON for its manifest file, and using the same base for all add-ons helps maintain consistency and ease switching between platforms if needed.

The only major disadvantage is not being able to comment in JSON files. There are workarounds, but they’re not very pretty.

The manifest file will be named manifest.json. It’s not package.json like previously speculated because it doesn’t serve the exact same purpose as package manifests in Jetpack, and it would cause confusion about the meaning of packages. It’s not install.json because it could be easily confused with the old install.js manifests.

The presence of manifest.json in the XPI file will indicate this file needs to be handled as a v2 package. If manifest.json isn’t found, the current install system should be used. XPIs should be able to include old and new manifest files in order to be backward-compatible with application versions that don’t support the new system.

A simple manifest file

A very simple Hello World add-on (taken from XUL School) would have the following manifest files.

install.rdf

<?xml version="1.0"?>

<RDF xmlns="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:em="http://www.mozilla.org/2004/em-rdf#">
  <Description about="urn:mozilla:install-manifest">
    <em:id>helloworld@xulschool.com</em:id>
    <em:name>XUL School Hello World</em:name>
    <em:description>Welcome to XUL School!</em:description>
    <em:version>0.1</em:version>
    <em:creator>Appcoast</em:creator>
    <em:homepageURL>https://developer.mozilla.org/en/XUL_School</em:homepageURL>
    <em:type>2</em:type>
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id>
        <em:minVersion>3.0</em:minVersion>
        <em:maxVersion>3.6.*</em:maxVersion>
      </Description>
    </em:targetApplication>
  </Description>
</RDF>

chrome.manifest

content   xulschoolhello              jar:chrome/xulschoolhello.jar!/content/
skin      xulschoolhello  classic/1.0 jar:chrome/xulschoolhello.jar!/skin/
locale    xulschoolhello  en-US       jar:chrome/xulschoolhello.jar!/locale/en-US/

overlay chrome://browser/content/browser.xul  chrome://xulschoolhello/content/browserOverlay.xul

The equivalent manifest file in the new system would be the following.

manifest.json

{
  id : "helloworld@xulschool.com",
  name : "XUL School Hello World",
  description : "Welcome to XUL School!",
  version : "0.1",
  authors : "Appcoast",
  homepageURL : "https://developer.mozilla.org/en/XUL_School",
  type: "2",
  targetApplications : { 
    id : "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}",
    minVersion : "3.0",
    maxVersion : "3.6.*"
  },     
  domain : "xulschoolhello",
  overlays : {
    source : "chrome://xulschoolhello/content/overlay.xul",
    target : "chrome://browser/content/browser.xul"
  }
}

The (not really) missing chrome declarations will be covered in part 2 of this specification.

Plural attributes in the manifest file (authors, targetApplications, overlays, and others) can either be a single value or object, or an array of them. So, if there were multiple authors to the add-on, the following would also work:

authors : [ "Appcoast", "Mozilla" ],

Part 2 of this specification will attempt to cover all possible attributes in the manifest file, as well as attribute localization.

{ 17 } Comments

  1. Mossop | 2010/05/13 at 7:15 PM | Permalink

    It’s great to see some resources working on this, I’ve had replacing rdf manifests in the works for a while now, in fact the new add-ons manager was explicitly written to make it easy. This is unfortunately the first I’ve seen of your posts though and I have some suggestions:

    I’m not sure combining the install.rdf and chrome.manifest data into one file is the best choice. Conceptually they do represent different things, metadata about the add-on and chrome mappings that the add-on provides. You can do without chrome.manifest (and indeed it won’t exist for add-ons developed with the Jetpack SDK) but you can’t do without install.rdf. They are also read in quite different ways. install.rdf is only needed at install time and then to all intents and purposes ignored. chrome.manifest on the other hand is read on every startup, I suspect that combining it with data that isn’t will cause a performance hit unless we were to implement some caching for this (mind you many people would like that). Incidentally the chrome registry that reads chrome.manifest files right now can’t parse JSON data since to my knowledge the JSON parser is only available to JS code, not C++.

    I also suspect that JSON isn’t the correct choice of format. JSON does have many benefits however I believe that a simple XML format has more. On the downside XML is more verbose than JSON, however it is also better established which brings better editor support. XML files encoded in any format while JSON must be UTF-8. We would also be able to provide an XML schema for the manifest which would allow many editors to automatically validate and offer inline autocomplete to developers letting them know that their manifests are correct (syntax-wise) without needing to test in an application. I put the choice of JSON and XML to a room full of add-on developers a couple of years ago and at that time they agreed that XML was a better choice.

    Have you had any thoughts on the update.rdf format? I think we really should be replacing that to match whatever the install.rdf format becomes to make it easier for developers to provide their own updates.

  2. jorge | 2010/05/13 at 10:48 PM | Permalink

    Conceptually they do represent different things

    I don’t think they are so different from a developer standpoint.

    chrome.manifest on the other hand is read on every startup, I suspect that combining it with data that isn’t will cause a performance hit

    I doubt that would make a significant difference. If we’re loading a file and parsing it, the performance hit of adding, say, 1kb of data to parse is very small. This is a case where I think we need to think about making the lives easier for developers by having a single manifest file, and maybe make our lives slightly more difficult finding a way to process this “mixed file” efficiently.

    unless we were to implement some caching for this (mind you many people would like that)

    I’d like to hear more about this. You mean merging all chrome manifest data into a single file so that we don’t need to read all individual files at startup? Or is it something more sophisticated than that? This is the time to ask and receive :)

    Incidentally the chrome registry that reads chrome.manifest files right now can’t parse JSON data since to my knowledge the JSON parser is only available to JS code, not C++.

    That’s a limitation that I hadn’t contemplated. However, it does look like the JSON parser is C++: http://mxr.mozilla.org/mozilla-central/source/dom/src/json/nsJSON.h. So maybe it is possible?

    XML … is also better established which brings better editor support

    True, although I wouldn’t say that this helped me in any way making install.rdf files over the years.

    XML files encoded in any format while JSON must be UTF-8.

    I guess this is important for localization, but I’m aiming to keep localized data out of the file. This will be covered on part 2.

    We would also be able to provide an XML schema…

    My experience with this kind of thing is that it never happens. Either the schema is left for the future as a “nice to have”, or it is implemented but never used or kept up to date. Again, this was possible with install.rdf but never happened.

    I put the choice of JSON and XML to a room full of add-on developers a couple of years ago and at that time they agreed that XML was a better choice.

    That I can’t argue with. I’d like to know the reasons behind their decision. I don’t have very strong feelings against an XML manifest file, but I do feel that JSON will bring us greater consistency with the Jetpack platform, and perhaps some performance gains when processing the files. I also think it feels easier to use, even if that is not necessarily true.

    Have you had any thoughts on the update.rdf format? I think we really should be replacing that to match whatever the install.rdf format becomes to make it easier for developers to provide their own updates.

    Thank you! I knew I was missing something. It should definitely be consistent with whatever is chosen for the manifest file. I’ll cover this file on Part 3.

  3. Mook | 2010/05/13 at 11:54 PM | Permalink

    RDF/XML is certainly not the easiest format to work with, so it getting replaced isn’t all that bad (assuming, of course, you could just keep an install.rdf around for older Geckos to build cross-version addons).

    Allowing plain strings in addition to arrays actually seems more confusing to me than just making it always an array (so you’d have to supply ["Appcoast"] instead), but perhaps I’m just odd.

    It may also be useful to think about non-contiguous compatible app versions (3.5 to 3.5.*, 3.6 to 3.6.*, but not 3.6a and friends, etc.), but perhaps that can be tacked on later instead of right now. Yes, sadly I’m actually occasionally hitting things where this might be handy. Same applies for the chrome.manifest equivalent, too (currently, from code inspection, it seems to just take the last appversion on the line…)

    Chrome.manifest was (I believe) deliberately made very easy to parse (essentially just whitespace separated fields similar to a csv) for perf reasons – so caching this will probably be adequate consolation for changing it to a more complex format. (That’s “complex to parse”, not “complex to write a parser for” – JSON works around the second point by having an existing parser.)

    JSON is unusable from C++, because while it’s _implemented_ in C++, it uses the JSAPI to actually return things in. It’s totally designed to be only usable from JS. In return, it’s really nice to use from JS.

    Lacking comments makes me extremely sad :( Not sure if it’s deal-breakingly sad, and I probably don’t matter enough to break any deals anyway.

    I don’t quite understand the “domain” bit; perhaps that’s better left for Part 2. I suspect I’ll have stronger things to say there, but it’s too early right now.

  4. Matthew Wilson | 2010/05/14 at 1:45 AM | Permalink

    Please don’t forget the signing of install.rdf/update.rdf with McCoy, for extensions which are not delivered over HTTPS.

  5. johnjbarton | 2010/05/14 at 9:28 PM | Permalink

    As formats go, the chrome.manifest is awesome. We may not know what the entries mean, but editing and copying them from other files is simple and effective.

    .rdf and .json are another matter. Both of these formats are designed specifically for machine reading. They have no significant concessions for human readability. Pick them only for small projects or hacks.

    In the firebug project I am writing our own extension loader. Our format will be HTML. Not as easy to read plain text but lots of good tools and all of the developers understand it. It shares the advantages of XML but renders it nicely in a browser ;-) .

    BTW our extension loader has collaborative testing as its goal: we want to empower extension collection creators to test a collection of extensions together then certify the results for their users. This creates a new player in the development/distribution chain, a kind of super tester/organizer person who helps a collection of extensions avoid breaking each other. When tests fail this person can help point to the problem; the collection may in some cases form a kind of loosely couple development team. Let me know if any of this interests you.

  6. Dave | 2010/05/14 at 9:28 PM | Permalink

    If you’re creating an XPI v2 format, please consider adding support for a better compression algorithm while you’re at it. (bug 93126) LZMA support in addition to DEFLATE (ZIP) would be great. Not much is required to get a significant reduction in installer size.

  7. harry | 2010/05/15 at 6:24 AM | Permalink

    Having manifest.json, you proposed, is a good goal, but is changing the internal representation a optimal way for achieving it?
    1) it’s hard to implement, one needs to modify lots of things in c++, persuade everyone it’s better way etc, and probably will not be included in the next release.
    2) when completed it will do little in reaching the main goal, which is to make creating and modifying full extensions as easy as jetpacks, since jetpacks will have tools for managing and with extensions one will have to do all the routine manually.

    the problem is that addon manager now doesn’t help one who want to modify his addons to effectively manage them
    of course majority of firefox users doesn’t need this, but those who need could have something like “tinkerers addon manager” addon, which will allow to
    1) treat files in chrome directory the same way as addons
    2) install unpacked addons
    3) unjar addon to make it easy to modify and then repack it
    4) create new addons from templates (like Add-on Builder does), package and upload addon to AMO
    5) be able to change small things in the addon and share your changes with the others (feature like addon variations, small bugfixes or changes, people who do this in mozillazine forums will appreciate a more automated way)
    6) maybe to be able to use diff files for this

    this addon will make a very easy path to implement manifest.json, since only autocompletion file for bespin or codemirror, and simple function to convert file in new format into 2 in old will be needed
    on the positive side also
    we will be able to use comments as we wish
    won’t need to generate more file IO on millions of computers for the sake of only thousands of developers
    will get rid of old style guid for firefox and use new one like songbird
    maybe will have it already in 3.6
    will not depend on changes in internal representation
    this is also more similar to the way jetpac solves the problem

  8. Robert Kaiser | 2010/05/16 at 9:52 AM | Permalink

    For one thing, this JSON is much uglier and too much boilerplate IMHO compared to the simple and nice text files that the chrome manifests are now, and it’s much easier to introduce unrecoverable syntax errors when editing it.

    For the other, I love how chrome manifests used in normal application context (copied from jar.mn) are the same as what add-ons use. You’re diving that as your format doesn’t seem to work on any way for using for normal application chrome.

    Do you have solutions for those things?

  9. Wladimir Palant | 2010/05/17 at 1:54 AM | Permalink

    @jorge:

    If we’re loading a file and parsing it, the performance hit of adding, say, 1kb of data to parse is very small.

    Looking at Adblock Plus, we are talking about more like 10kB – mostly localized descriptions that are absolutely worthless during startup. However, the add-on manager already caches extension paths in extensions.ini. Changing the format of this file and filling it with data previously stored in chrome.manifest would solve this problem. And it should improve startup time because reading in all the individual manifests will become entirely unnecessary.

    Other than that, I would also prefer XML over JSON. Comments are certainly one reason (can you really remember that {ec8030f7-c20a-464f-9b0e-13a3a9e97384} is Firefox? I have nine applications listed in my install manifest). The other is that XML data is more structured, finding matching opening tag is much easier than balancing brackets – better suited for human reading IMO.

    And your chrome.manifest replacement looks like you want to use default paths for chrome locations. I hope you still allow for the more complicated scenarios. E.g. I use a chrome.manifest with absolute file paths as my “development environment” – the extension installed in Firefox profile uses files directly from my source code repository. And Adblock Plus is using three different chrome “domains” right now (granted, the two additional domains are work-arounds for shortcomings of the platform).

  10. jorge | 2010/05/17 at 5:24 PM | Permalink

    @Matthew Wilson: thanks!

    @johnjbarton: I like the idea of (strict) HTML manifests. It removes the burden of yet-another-custom-xml format and brings realistic IDE support to the table. I’d really like to see some examples of what you have in mind. As for the collaborative testing, I don’t understand what the idea is, I’d like to hear more about it. Thanks!

    @Dave: that’s a good idea, but I think it falls outside of the scope of this project. It’s something I would consider if we had time to spare, and I doubt there will be. That bug needs a patch to get things rolling!

  11. jorge | 2010/05/17 at 5:34 PM | Permalink

    @harry: some good points there, but mostly unrelated to this project. Sure, it’s a hard problem to tackle, and it should’ve been tackled years ago, but that doesn’t make it unworthy of our time. Other tools and projects will come, I promise :)

    @Robert Kaiser: can’t help people like JSON, or other formats FWIW. I personally think chrome.manifest is horrible. You have a good point about this format used elsewhere in the application, and that’s something this project won’t tackle, as it’s limited to add-ons. I agree it will be a little inconsistent with chrome registration in Firefox, but that’s something that only very advanced developers discover. Most will never know about it. This is no excuse, I do think that they should eventually use the same format.

    @Wladimir Palant: good points. ABP is in some ways THE add-on, and it will be an ultimate test case for any changes in add-on standards. I’ll make sure the new manifest standard is capable of supporting all current add-ons. Part 2 will cover most of your concerns.

  12. jorge | 2010/05/17 at 5:40 PM | Permalink

    Everyone: thank you for your feedback. Please keep it coming!

    Just to reiterate a point I try to make in several places: this is a specification for the GSOC project, not a final spec for a new add-on packaging system. If I chose JSON and then it turned out that it is necessary to switch to XML to integrate this code into Firefox, it can be changed. I’ll do the coding if necessary. So don’t feel discouraged if I make a decision you don’t agree with. I’m only trying to make it easier for the student that will be working on it. The discussion is ongoing and there is plenty of time to adjust the final implementation before it is considered as a new part of the platform, with more input from the community.

    Thanks again!

  13. sys | 2010/05/19 at 4:45 AM | Permalink

    You want to simplify for developers, but you choose a format without comments. I think comments is useful for developers in many ways. Xml sound better for me.

    I’m agree with Mossop, merging packaging manifest and chrome manifest is meaningless. Deployment artifacts and application behavior should be distinguished. In this case why not merging with pref.js ? If you want simplify, perhaps you could merging default prefs and chrome ?

  14. jorge | 2010/05/19 at 4:54 PM | Permalink

    @sys: I don’t think simplification is meaningless, it does go a long way in making things easier for developers. If you have a look at part 2, most chrome.manifest declarations are removed, making a separate chrome.manifest file almost completely unnecessary.
    Merging default prefs into this manifest is something I didn’t contemplate, and I don’t think it makes sense to do since preferences are tightly integrated with the actual add-on code, while all information in the manifest file can be considered as metadata. I know that last comment wasn’t completely serious, but I wanted to give you a sound reason not to do it.

  15. sys | 2010/05/25 at 12:10 PM | Permalink

    @jorge Yes, merging pref and chrome manifest wasn’t really serious ! More seriously, the good question is “all information in the manifest file can be considered as metadata” ? With “overlay”, “override”, “style” (…) options in chrome manifest, we can use it for customize the application behaviour. In our case we use many chrome manifests (with overlay, override…) in order to build easily different target applications. These chromes can be deployed in Firefox add-ons and in Xulrunner stand-alone applications. Duplicate these settings in each packaging seems bad for us.
    Sorry if I missed something, but with your idea how do we declare chromes in Xulrunner app (not in add-ons) ?

  16. jorge | 2010/05/25 at 3:02 PM | Permalink

    @sys: I consider it metadata because an add-on can’t easily modify overlays, overrides, etc. They define how the add-on is loaded and rarely change dynamically, unlike most add-on code.
    This way of declaring chrome packages is limited to add-ons at the moment. Replacing it all over the Mozilla code base and extending it to XULRunner applications is definitely out of the scope of this project. We’ll first have to see if there’s real acceptance of this implementation once it’s done.
    One of the requirements of this project is that the new packaging system needs to work alongside the old one, since there would be a long adaptation and migration process. It will also be possible to include manifests for the old and new formats, for backward compatibility.

  17. pjdkrunkt | 2010/06/13 at 7:31 PM | Permalink

    I’m trying to wrap my head around how these proposed changes will be more simple in the end for the developers. On one hand you have JSON, which while it looks fairly *clean* it also looks like it will be way too easy to forget a comma or a } and not being able to have comments makes it difficult for mutli-platform add-ons especially. In the case of XML you are talking about a format that’s *basically* identical to RDF in terms of grammar. Unless you are introducing custom commands just for this kind of file it will actually end up being more cumbersome than the RDF format already is. Is there any actual advantage to:

    (li id=”id”)helloworld@xulschool.com(/li)

    over

    (em:id)helloworld@xulschool.com(/em:id)

    ?

Post a Comment

Your email is never published nor shared. Required fields are marked *