Take any software project, on its own its probably not very useful. First of all, you probably need a complier or interpreter, something to directly run the software, or convert the source form (preferred form for editing), to a form which can be run by the computer.

In addition to this compiler or interpreter, it's very unusual to have software which does not use other software projects. This might require the availability of these other projects when compiling, or just at runtime.

So say you write some software, the other bits of software that your users must have to build it (generate the useful form of the software, from the source form) are called build dependencies. Any bits of software that are required when your software is run are called runtime dependencies.

This complexity can make trying to use software a bit difficult... You find some software on the web, it sounds good, so you download it. First of all, you need to satisfy all the build dependencies, and their dependencies, and so on... If you manage to make it this far, you can then actually compile/run the software. After this, you then need to install all the runtime dependencies, and their dependencies, and so on... before you can run the software.

This is a rather offputting situation. Making modular software is good practice, but even adding one direct dependency can add many more indirect dependencies.

Now there are systems to help with this, but unfortunately I don't think there is yet a perfect, or even good approach. The above description may make this seem easy to manage, but many of the systems around fall short.

Software Packages

Software packages, or just packages for short is a term describing some software (normally a single software project), in some form (source, binary, or perhaps both), along with some metadata (information about the software, e.g. version or contributors).

Packages are the key component of the (poor) solutions discussed below to the problem of distributing, and using software.

Debian Packages

Debian, "The universal operating system" uses packages (*.deb's). Debian packages are written as source packages, that can be built to create binary packages (one source package can make many binary packages). Debian packages are primarily distributed as binary packages (which means that the user does not have to install the build dependencies, or spend time building the package).

Packaging the operating system from the bottom up has its advantages. This means that Debian can attempt to solve complex issues like bootstrapping (building all packages from scratch), reproducible builds (making sure the build process works exactly the same when the time, system name, or other irrelevant things are different).

Using Debian's packages does have some disadvantages. They only work if you are installing the package into the operating system. This is quite a big deal, especially if you are not the owner of the system which you are using. You can also only install one version of a Debian package on your system. This means that for some software projects, there are different packages for different versions (normally different major versions) of the software.

npm Packages

On the other end of the spectrum, you have package managers like npm. This is a language specific package manager for the JavaScript language. It allows any user to install packages, and you can install one package several times on your system.

However, npm has no concept of source packages, which means its difficult to ensure that the software you are using is secure, and that it does what it says it will. It is also of limited scope (although this is not necessarily bad).

Something better...

I feel that there must be some middle ground between these two situations. Maybe involving, one, two, or more separate or interconnected bits of software that together can provide all the desirable properties.

I think that language specific package managers are only currently good for development, when it comes to deployment, you often need something that can manage more of the system.

Also, language specific package managers do not account for dependencies that cross language boundaries. This means that you cannot really reason about reproducible builds, or bootstrapping with a language specific package manager.

On the other end of the scale, Debian binary packages are effectively just archives that you unpack in to the root directory. They assume absolute and relative paths, which makes them unsuitable for installing elsewhere (e.g. in a users home directory). This means that it is not possible to use them if you do not have root access on the system.

All is not yet lost...

There are some signs of light in the darkness. Debian's reproducible builds initiative is progressing well. In the Debian way, this has ramifications for everyone, as an effort will be made to include any changes made in Debian, in the software projects themselves.

I am also hearing more and more about package managers that seem to be in roughly the right spot. Nix and Guix, although I have used neither both sound enticing, promising "atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments" (from the Nix homepage). Although with great power comes great responsibility, performing security updates in Debian would probably be more complex if there could be multiple installations, of perhaps versions of an insecure piece of software on a system.

Perhaps some semantic web technologies can play a part. URI's could prove useful as unique identifiers for software, and software versions. Basic package descriptions could be written in RDF, using URI's, allowing these to be used by multiple packaging systems (the ability to have sameAs properties in RDF might be useful).

At the moment, I am working on Debian packages. I depend on these for most of my computers. Unfortunately, for some of the software projects I write, it is not really possible to just depend on Debian packages. For some I have managed to get by with git submodules, for others I have entered the insane world of shell scripts which just download the dependencies off the web, sometimes also using Bower and Grunt.

Needless to say I am always on the look out for ways to improve this situation.