Often inconspicuous, the package manager is the workhorse of a thriving open source community. It is the source of truth and an index of reusable software building blocks. When it works, the developer experience is seamless; when it falters, the community palpitates.
Percolate Studio had a chance to sit down with Meteor core developer Ekate Kuznetsova and packaging lead to answer the most pressing questions about the new packaging backend (which provides the API for Atmosphere).
Why use the Meteor package manager (versus NPM)?
Ekate: Biggest distinction: Meteor packages can do a lot more things. Take an easy example, like some accounts packages. There is a server component that works on the server, but there is also the client component (for example, the dialog box you get when click the login button), and the communication between them. This is even more obvious for something like mongo-livedata, redis support, iron:router (which is the router for your apps!) and more. You can’t get that level of client-server integration with just NPM.
There are some other differences as well. We don’t, for now, support multiple loading (partly because of the above: multiple loading on the client would be slow, and multiple loading routers/accounts/database drivers (for example) would be just weird). Instead, we computationally figure out which versions of which packages the app should use. We err on the side of using the versions that packages were tested with and don’t just update to the latest version automatically. Similarly, Meteor packages and apps keep track of package versions, providing repeatable builds. Thus, there is no need for a special ‘shrinkwrap’ command.
Overall, the Meteor package manager is just better integrated into the use of Meteor than npm is, including some of the same offline-workflow support that you may have come to expect from us. You can still use NPM packages in Meteor, by using the Npm.depends clause in package.js, which is the control file for the package.
What is Meteorite?
Ekate: Meteorite has been the amazing solution to the Meteor package distribution problem, written by Percolate Studio. Back when Meteor was announced, there was no support for distributing and managing non-core Meteor packages, but there was the idea of a Meteor ‘unipackage’ and its eventual distribution.
Meteorite took that idea and ran with it. It provided an easy way to use non-core packages, and Percolate Studio ran both the front end and the back end of the Atmosphere package repository, which helped keep track of them. There were nearly 1700 packages published from Meteorite by members of the Meteor community.
Meteorite did an amazing job — thanks again, Percolate Studio, for running Meteorite and Atmosphere! The packaging ecosystem has become one of the most amazing parts of the Meteor community, and it would not have existed without you.
How is the Meteor Package Manager different than Meteorite?
Ekate: Meteor Packaging is the official solution to the problem of Meteor packaging. Unlike Meteorite, it is fully integrated into the Meteor tool, with both immediate and future consequences! Here is a brief, somewhat scattered list of some of them.
Command line integration with meteor: no more need to install a separate program to add bootstrap-3 to your app. In Meteor 0.9.0 and above, adding community packages and core packages is the same (meteor add). It is also easier to deal with version restrictions — ‘meteor add email@example.com’ means never use any version of foo below 1.0.0, and ‘meteor add foo@=1.0.0’ means use 1.0.0 exactly an only that. If you are writing a package, you just need a package.js file with the right information, and then you can hit ‘meteor publish –create’, and then, subsequently, ‘meteor publish’.
Versioned core packages: Prior to Meteor 0.9.0, core packages did not have versions. That meant, that there was no good way to say, as a package maintainer, that this is a version of the package that works with the new UI package (as opposed to the old package), or better yet, the other way around. On the app side, this meant that updating Meteorite packages was often an awkward task, with weird side-effects. The new packaging system’s update command will take that interplay into account, and, for example, prevent from taking the new version of Meteor if it would cause your packages to stop working. (You can override this, if you want).
Weak dependencies on community packages: In the same vein as the last one, prior to Meteor 0.9.0, it was difficult to have a weak dependency (if the user added this package, use it, but don’t otherwise) on most packages. Package authors had to parse internal app configuration files, and use that to figure out what packages the user had in mind. Now, that is not an issue — using a community package is the same as using a core package, including support for weak dependencies, implies, etc.
Better support for different architectures: Some packages have architecture-specific builds. That is, they use C code that needs to be separately compiled on linux32, linux 64 and mac (the three supported architectures for Meteor). In the new Meteor build system, you can publish different builds for the same version of the package from different architectures. Because Meteor links the built package, rather than the source of the package, there are some interesting consequences. For example– let’s say you have an app, and you are developing on a mac, and you have added, through the Meteor build system, a package that contains binary code. Now you want to deploy using Meteor deploy, or even just bundle the app for a linux server with the bundle command. You couldn’t do this before, but now you can just run ‘meteor deploy’ and ‘meteor bundle –architecture’, and Meteor will pull in the precompiled version of that package that works for linux, even though you are using a mac yourself.
There are also other bonus features, but they are probably not as central to most people’s experience. For example, Meteor caches a lot of data on the machine, so, in many cases, you can run commands like ‘meteor add’ while you are offline. Similarly, the new packaging system is very extendable, which means that people can publish their own forks of the Meteor command-line tool, and use them through normal Meteor commands (including updating to new versions) instead of running from a git checkout. There are probably others! These are some of the main ones.
What are some of the technical challenges that went into creating this package manager?
Ekate: Meteorite functioned by, essentially, dropping the source of the package into the app, and proceeding from there. The Meteor packaging system does something slightly more complicated — there is a central repository of packages on disk, and then we choose the version of the package that we are going to use (based on core package versions, user constraints, etc). This works better for a variety of reasons — more efficient, easier to version, etc.
In order to do this, we had to rewrite large portions of the tool that deals with building apps. Actually, to be honest, we rewrote almost everything about building packages, and some of the stuff about building apps, changed our internal release process, and so on. The system is much better now, more robust, more reasonable, more extendable. Right off the bat, it does more things than the old one — but we had to change a bunch of stuff, and that took work.
Additionally, we also wrote this constraint solver package. It takes a set of dependencies and solves them into a compatible set of package-version mappings. (I want version 1.0.0 or above of iron:router, and version 1.5.0 of sparkle:forms, so which versions of iron:sparkles do I get?). That supports our single-loading, makes the system work in predictable and proper ways, etc. It is also extendable, so we are looking forward to adding more things to it in the future.
Those were the two main things, but there was more stuff. It was fun! Glad that 0.9.0 is out, and looking forward to doing more development on packaging in the future. :)
Why do package names include the author?
Ekate: Two reasons! First off, a lot of times, there is a clear demand for a package that does something, and a lot of people set out to write different solutions. For example, there are a lot of packages that deal with forms. It would be kind of weird to have a rush for a good name: you don’t want to explain to all newcomers that ‘forms’ is outdated, and they should use ‘kitten-responder’ instead. That’s just weird. When package names are namespaced, it is much easier for everyone to come up with good and informative names for their packages. We think that contributes, a lot to the usability of the system.
The other part of namespacing is credit. Our community is full of amazing people! We wanted to reflect that in the packaging system. So, packages are namespaced by the author, and if there is a cool package, and you wrote it, people will know who you are.
How do I migrate my app from Meteorite to the Meteor Package System (0.9.0) ?
Ekate: First, you should run the ‘mrt migrate-app’ command. Hopefully, that will do everything! It is super cool, I have played around with it! Go Meteorite, yay!
In case that didn’t work, or you want to do things manually: take a look at your Meteorite packages. For example, let’s say you are using a Meteorite package called ‘meow’. Then, you should find it in the new system — you can use the search command in Meteor, or look it up in the new Atmosphere. In Meteor, ‘meow’ will be prefixed by the name of its maintainer — for example, cheezburger. Once you know what package you want, add it to your app: type in ‘meteor add cheezburger:meow’. Remove ‘meow’ from the list of Meteorite packages that you are using. Repeat for all Meteorite packages.
If you are using local packages, you may also want to make sure that their names are valid in 0.9.0. In 0.9.0, package names must use lowercase letters, – and . characters. You may use a colon to prefix a package with a username. Packages may not begin with a dot. If you have a local package that doesn’t follow these guidelines, you will have to rename it.
What do I do if some of the packages I’m using don’t work in the new system?
Ekate: You should contact the package author, and figure out why. It is possible that the package could not be migrated, in which case, hopefully, MDG is aware of it, and is working on fixing the underlying issue. It is also possible that the package could not be automatically migrated by the Meteorite script, and the author has not gotten around to migrating it manually.
At this point, you can try to migrate a fork of the package yourself, but that can get a bit awkward if you are using other packages that depend on it. (We are working on making forking easier, but not in this release). If you are not using other packages that depend on this one, then you can try to successfully fork it into the new system, by publishing it with your own username prefix (and a note in the description). Then, you can use your fork until the original is migrated.
If neither of those helps you out, you can continue to use Meteorite for a little while longer. Hopefully, most packages have either been automatically migrated prior to 0.9.0, or will be migrated by their owners in the near future.
How do package authors make sure their packages work in the new system?
Ekate: We have automatically migrated a lot of packages from the old system. In most cases, the migration succeeded, and the package works as it did before. Sometimes, though, the migration may seem to have succeeded, but the package might not work as well. This is particularly true for packages that relied on being in the specific ‘packages’ directory of an app. If you suspect that this has happened to your package, you can set up a test app, add your package with the add command, and see what happens. Alternatively, you can just begin by porting your package to the new system, testing it as you normally would, and publish a new version of it into the system. Running the update command will take users to that version.
Unfortunately, It is also possible that some of your packages did not migrate successfully. You can run
meteor search '' --maintainer <your Meteor Developer Account username> --show-old. If you see an ‘INCOMPATIBLE WITH METEOR 0.9.0’ note, that means that the automigration failed. In that case, you are going to have to publish a new version of the package manually. Once you do, users will be able to get to that version with the update command, and the ominous note in search will go away.
How do I publish a brand new package to the Meteor Package System?
Ekate: Let’s start with package creation. You can use ‘meteor create –package
To publish, cd into the package’s directory and run ‘meteor publish –create’.
To publish new versions, increment the version number and just run ‘meteor publish’.
Is it possible to keep using Meteorite with Meteor 0.9.0?
Ekate: Yes, it is still possible. In particular, Meteor does not have support for adding packages via git URLs in a way that Meteorite does, so if you were relying on that feature, you may have to continue using Meteorite for a while longer. Additionally, we have not been able to successfully auto-migrate all of the packages, and if a package that you are using has not been migrated to 0.9.0, you might have to continue using Meteorite with it for a bit.