Deprecation as product lines

I would like to draw a connection between two lines of research: deprecation, and product lines. The punchline is that my personal view on deprecation could be explained by reference to product lines: deprecation is a product line with just two products. To see how that connection works, first take a look at what each of these terms means.

A product line is a collection of products built from a single shared pool of source code. Some examples of a product line would be:

• The Android, iPhone, Windows, and Macintosh versions of an application.
• The English, Chinese, and Lojban versions of an application.
• The trial, normal, and professional versions of an application.
• The embedded-Java and full-Java versions of a Java library.

There is a rich literature on product lines; an example I am familiar with is the work on CFJ (Colored Featherweight Java). CFJ is Java extended with "color" annotations. You "color" your classes, methods, and fields depending on which product line each part of the program belongs to. A static checker verifies that the colors are consistent with each other, e.g. that the mobile version of your code does not invoke a method that is only present on the desktop version. A build-time tool can build individual products in the product line by extracting just the code that goes with a designated color. To my knowledge, CFJ has not been explicitly used outside of the CIDE tool it was developed with, and CIDE itself does not appear to be widely used. Instead, the widely used tools for product lines don't have a good theoretical grounding.

Deprecation, meanwhile, is the annotation of code that is going away. As with CFJ, deprecation tools are very widely used but not well grounded theoretically. With deprecation, programmers mark chunks of code as deprecated, and a compile time checker emits warnings whenever non-deprecated code accesses deprecated code. I have previously shown that the deprecation checker in Oracle javac has holes; there are cases where removing the deprecated code results in a a program that either does not type check or that does not behave the same.

As much as I enjoyed working on a specific theoretical framework for deprecation, I must now admit that it's really a special case of CFJ. For the simpler version of deprecation checking, choose two colors, non-deprecated and everything, and mark everything with the "everything" color. You then have two products in the product line: one where you leave everything as is, and one where you keep only the non-deprecated code.

There is a lot of potential future work in this area; for this post I just wanted to draw the connection. I believe CFJ would benefit from explicitly claiming that the colored subsets of the program have the same behavior as the full program; I believe it has this property, and I went to the trouble of proving it holds for deprecation checking. Also, I believe there is fruitful work in studying the kinds of colors that are available. With deprecation, there is usually no point in time where you can remove all deprecated code in the entire code base. You want to have a number of colors for the deprecated code, for example different colors for different future versions of the software.

The Web version of interface evolution

Nick Bradbury points out an important issue with web APIs:

I created FeedDemon 1.0 in 2003, and it was the first app I wrote that relied on web APIs. Now those APIs no longer exist, and almost every version of FeedDemon since then has required massive changes due to the shifting sands of the web APIs I've relied on.

There are a tangle of issues involved here.

One is that, for sure, web APIs are not a suitable way to build a self-contained system that will at least remain internally consistent indefinitely. You couldn't build a space probe using any web APIs that you don't control. By the time it got to Jupiter the protocols might well have changed. Even barring an intentional protocol change, the service might upgrade its software and accidentally break you.

Not all software is of this character, however. Most user-facing software is expected to stay compatible with the other software a particular user is taking advantage of. Most user-facing software has a mechanism for being updated after it's deployed.

The thought experiment I use here is to consider why the Lisp Machine is no use nowadays. These systems are very impressive; think Emacs, but with a better programming language, and with multiple for-profit companies writing professional software to run on it. Nonetheless, they are useless nowadays (and Emacs is as awesome as ever). That it is so is obvious, but what is the precise reason for it?

My best answer is that the world changed around them. Even without any explicit API break, Lisp Machines just don't have integration with other forms of software that its users would find important. That whole Internet thing is a simple example.

In general, most software is only useful if it is under active maintenance. The difference between zero maintenance and a little bit of maintenance is huge. If you are considering using software isn't under maintenance, run away! If you continue to use it, you will eventually find that you have become its maintainer yourself.

Which brings me back to APIs. APIs are never perfect, and so they evolve just like any other interface. The only way this is different for web APIs is that the clients do not get to choose when to upgrade. One day, the provider updates their software and it's simply on the new API. Gilad Bracha describes this as "versionless software".

Ideally, this evolution should involve discussion between the service provider and all of the clients. Exactly how those discussions work is a rich question that is similar to any other decision process by a number of stakeholders. For an API like one offered by Google, individual clients have very little influence on the API, so you have to decide whether to take it or leave it; part of that decision involves your expectation that the service provider will treat you well. In other cases, there might be a contractual agreement between a user of the service and its provider; in that case, any API changes could be worked out as part of negotiating the contract. In still other cases, a group of software users might meet in standards committees, as happens to some degree with the HTTP protocol.

In short, I really don't think that constant interface change is a fundamental reason to avoid web APIs. Instead, it's a fundamental part of most software development that you have to keep maintaining, whether or not the APIs you program against are accessed over the web. Instead, you should be choosy about what specific APIs you depend on. Just like you wouldn't want to depend on an ancient unmaintained hunk of software, you also wouldn't want to use a hyperactively maintained hunk of web service that has a completely different API from one week to the next. Think explicitly about the API evolution story for any service you depend on, and use your judgement.

Avoiding big-bang updates

A big-bang update is a system update where the entire system must be shut down, multiple components are upgraded in parallel, and then the whole system is turned back on. If it goes well, then BANG, the whole system is upgraded in one big swoop. If it goes well, you lose a day or two of work from everyone working on that system, which is already pretty bad, but at least it's only a day or two. It often doesn't go well, however. Often, there is some dependency that snuck through early testing, and the one or two days extends into three, four, or five as engineers panic to patch up the lurking problems.

With a little foresight, big-bang updates can usually be avoided. The basic requirement is that system components can be updated one by one, without needing to update multiple components at the same time. That, in turn, implies that the bulk of components need to tolerate both the current versions of all other components as well as the upcoming version. Solve the multi-version dependency problem, and you can avoid big-bang updates.

Depending on multiple versions can be tricky. A common situation that arises, for any kind of system, is a desire to change a name that is shared between two different components. If the components are modules of source code, then the name might be a class name that is exported from one module and imported by others. If the components are hooked up using HTTP, then the names might be URLs. The temptation is to change the name, but that leads to a big-bang update. Every component that uses the name has to be updated at the same time. To avoid the big bang, try to find a way to support both the old name and the new name during a transition period. Then all components can be updated one by one to support both names. Once everything has been updated, it is possible to gradually drop support for the old name. It takes longer but it avoids a big bang.

An instance of the problem in programming languages is that module interfaces often tempt programmers into big-bang updates. For example, experienced Java system developers frequently exhort fellow developers not to change any interface once it's published. If you do change such an interface, then all users and all implementers of the interface have to update simultaneously. Bang. With Java as it stands, a better approach is to define a new interface with the desired changes and use dynamic typing to decide whether to use the old or new interface in any given circumstance. Alternatively, of course, an components-friendly programming language could directly support interface evolution directly in the language.