Computing notes 2015 part two

This document contains only my personal opinions and calls of judgement, and where any comment is made as to the quality of anybody's work, the comment is an opinion, in my judgement.

[file this blog page at: digg Technorati]

151219 Sat: Making systems more fragile

Yesterday I was replacing the one disk storage unit in an Ubuntu based workstation and this was first somewhat complicated because the system boots from a /boot and / filesystem on an LVM2 logical volume using GRUB2. This can be handled by the LVM2 module for GRUB2, but it requires a careful set iof update because the new volume group and logical volume labels changed:

Such are the benefits of having multiple layers of software virtualization.

I did all this by attaching the flash SSD a new SATA target, and then setting up a new volume group and logical volumes on it and filetrees and then copying with RSYNC.

To fix the initrd images, the GRUB2 configuration and the bootloader install it is easiest if the new flash SSD is already the primary/boot unit on the workstation, so I recable it as such, and then used the GRML live image to boot the system and run the relevant commands with a chroot to the flash SSD's new / filetree. It is a bit sad that it is non trivial to do all this except in the default case, but I wanted to take a shortcut.

So far so ordinary. But then I also noticed that the just copied system was way out of date, and I used the aptitude command in the chroot to upgrade it. That failed: because some packages cannot be successfully upgraded (notably the awful udev by the despicable GregKH) unless DBUS is running and Upstart is running or both. Neither is available under GRML or in a GRML chroot, which is an old-style Debian derivative. The resulting system could not boot properly.

Despite some efforts to fix it by hand, I had to restore the / filetree as it was, reapply the configuration updates, and once rebooted, I could upgrade the outdated packages.

All of the above is quite regrettable because the story is that the boot process and the package installation process has become a lot more fragile: the tangle of dependencies, of course mostly undocumented, has become a lot deeper from the traditional UNIX days.

PS While writing this I noticed in an IRC channel another complication:

[16:01]         jena | hello everybody, yesterday I upgraded linux-image deb. 
                       Today when booting i get a lot of repeated message like 
                       "volume group sda5_crypt not found \n skipping volume 
                       group sda5_crypt \n unable to find LVM volume 
                       sda5_crypt/ENCRYPTED_OS" and (initramfs) promt, what 
                       should I do? [PS /dev/sda5 is a luks partition on which 
                       there are lvm for / and /home, /boot is on /dev/sda2] 
                       (should I ask in a different channel?)
151216 Sat: Ever faster PCIe based flash SSDs

A review of a recent fast flash SSD M.2 device is quite interesting and insightful. Some of the more interesting points:

Because of the small size and low power draw it is clearly targeted at laptops, but it seems a bit overkill for a laptop.

Because of the endurance and temperature limits it seems mostly aimed at short-medium bursts of peak performance, but I guess that with suitably cooling it could have a wider applicability, for example for collecting data at fairly high speed.

151114 Sat: Container based deployment has been around a long time

Reading a piece about container based deployment of applications caused me a lot of amusement when reading this astuste observation:

2. Are you operationally prepared to manage multiple languages/libraries/repositories?

Last year, we encountered an organization that developed a modular application while allowing developers to “use what they want” to build individual components. It was a nice concept but a total organizational nightmare — chasing the ideal of modular design without considering the impact of this complexity on their operations.

The organization was then interested in Docker to help facilitate deployments, but we strongly recommended that this organization not use Docker before addressing the root issues. Making it easier to deploy these disparate applications wouldn’t be an antidote to the difficulties of maintaining several different development stacks for long-term maintenance of these apps.

That encapsulates unwittingly the major reason why containerized deployment formats exist: precisely to avoid dealing in the short term with the difficulties of maintaining several different development stacks.

The biggest appeal of containers is that it shifts the burden of doing the latter to the developers of the applications by first letting them “use what they want” and then taking complete responsibility for the maintenance of all that.

But of course developers, once they have delivered the application, are rarely going to be involved in its long term maintenance: many will be moved to different projects, or leave, or will simply not have the incentives to look after the application stacks they have already delivered.

The second biggest appeal of container deployment is that it front-loads the benefits and back-loads the costs: deployment is cheap and easy, as it is maintaining the deployed applications and their stacks that is made harder, because they are isolated in each container.

Containerization is mostly a new memetic wrapper for an old and established technique: DLL-hell, the practice of delivering an application with a whole copy of its development stacks and deploying them in a separate subtree of the filesystem so that each application on the host system has a whole separate environment. This has been the standard deployment technique for a long time on consumer oriented platforms like MS-Windows and Google Android.

It is the DLL-hell technique implicit in containerization that has the effect of shifting environment stack maintenance from the people responsible for the hosting to the people responsible for the development of applications. This works under the following assumptions:

These conditions usually apply to workstations and mobile phones, which are usually replaced as a whole every few years rather than maintained. When however the DLL-hell deployment strategy or its equivalents like containers are deployed in a business environment, the usual result is not maintenance, but its opposite: once deployed, applications and their development stacks get frozen in place, usually because it is so expensive to maintain them, and freezing in an isolated environment each application can reduce the impact of many issues to a single application. For businesses that deploy as commonly happens hundreds of applications that can be a good strategy.

And this is easily observed in practice: because containerization has been happening in business IT for a long time, with physical containers: that is the very common practice of one application deployed per server. It is such a common practice that the success of virtualization has been based on consolidation: turning physical containers, each running one application and its developement stack, into virtual containers as virtual machines. Full virtualization required to avoid any issues in running legacy operating system layers.

The main objective of consolidation into virtual machines of an existing set of deployments onto physical machines was of course to keep those deployment frozen in place, by creating a virtual host as close as possible to the physical one.

Less complete virtualization as in virtual containers like Docker is essentially a reversal of that practice: instead of deploying applications and their development stacks to their own physical container, and then virtualizing it to achieve consolidation, consolidation into virtual containers is the starting point, by delivering applications already frozen in place inside a container with their custom development stacks, with at least the advantage that by targeting a standalone container to start with a more efficient less complete virtualization technique is feasible.

That is, as argued above, the purpose of containers is indeed to ignore the difficulties of maintaining several different development stacks for long-term maintenance of these apps.

Therefore containerization is an advantageous strategy when that long-term maintenance is not important: for example for applications that have a relatively short expected life, or for situations where their long-term maintainability is not otherwise relevant in the short term.

For example in many projects driven by developers long-term maintainability is not that relevant, because the developers are rewarded on speed of delivery and the looks of their applications, and the ability to “use what they want” to build individual components helps them towards that.

151023 Sat: Anisotropy and the difference between performance and speed

I keep seeing incorrect uses of the concept of performance where it is confused with the concept of speed or some other simple ratio, and the two are very different. A contrived example might illustrate:

This illustrates three important points:

Note: performance envelopes with smaller dimensions are often illustrated with what are commonly called diamond diagrams, kiviat or radar charts, or star plots (1, 2).

Because of anisotropy even if the performance envelopes are equivalent in some cases one will allow reaching better results than another, and even when a performance envelope is overall better than another, it may still deliver worse results in some cases where the other has an advantage for which it is specialized.

Performance envelope profiles and their anisotropy are particularly important for computer system performance, because that anisotropy is likely to be significant. To begin with computer systems are designed for anisotropy: they are usually not very good at running, digging, or philosophy compared to human beings, but are much better at storing data and repetitive calculations over masses of data.

Thus computer engineers have found that many computer applications are specialized and benefit from anisotropy, and sometimes these applications are benchmarks which matter for sales even if they are unrepresentative of common uses precisely because they are too anisotropic.

Anisotropy of performance envelopes is particularly common in storage systems, because it seems quite difficult to build storage systems that are fairly good in most respects, but easier to build them so that they be very good at some aspect and quite bad at others.

Storage systems performance anisotropy may be inevitable because they are more or less necessarily based on the replication of multiple identical components, which have anisotropic performance envelopes themselves, and that are connected in parallel in some ways and in serial in others.

For example in disk systems the components are magnetic tracks; there is a single arm with multiple read-write heads, and the organization in concentric cylinders means that all the tracks on a cylinder can be read almost (because of single channel electronics) in parallel, but access to tracks in other cylinders can only be expensively serialized by moving the disk arm.

So a simple metric like speed is not very useful as to evalutating the general value of a computer component, and it is not performance. Unfortunately performance is not a simplistic figure of merit but an often anisotropic volyme of possible tradeoffs, and it takes considerable skill to profile it, and judgement to evaluate it with respect to potential applications.

Recently for example in storage we have flash SSD drives that can deliver high speed in many applications but also high latency in sustained writes and even more recently SMR disk drives that deliver high capacity and low power consumption but also low IOPS-per-TB and high write latency bcause of RMW.

Sometimes it amazes me that some people lack awareness (or pretend to) of the difference between a simplistic metric and a performance envelope, which is usually quite anisotropic. After all the saying choose your poison is ancient, and fast, cheap, reliable: choose any two is more recent but still well known. It is tempting to put the blame on the shift from an engineering mindset to a marketing one, but then that's not exactly new either.

150918 Fri: Canonical's community involvement model seems unwise

Canonical have been funding the development of several interesting tools, for example Mir, Ubuntu Touch, Unity, Juju, MAAS, bzr, Launchpad (plus significant contributions to other projects like Upstart).

While some people think this is because of not-invented-here syndrome the obvious reason is that most of the tools above filled a gap, and anyhow Canonical tranparently wants to build an inventory of intellectual property assets, even if licensed usually as free software.

My impression of most of them is that they are not-quite-finished, proof of concept initial versions, in particular because of lack of documentation, instrumentation and testing, that is overall maintainability.

Canonical's hope is that the community will help, and volunteers will provide the missing documentation, instrumentation and testing, which are very expensive to do, so crowdsourcing would help: according to the author of the book The mythical man-month adding maintainability to a proof of concept project requires twice as much time and effort as writing the proof of concept, and ensuring in addition robustness for system-level software another factor of two. These to me seem plausible guidelines.

The difficulty with that is that a lot of software development has just for fun or scratch your itch or reputation-building motivations. Developing half-finished cool-looking stuff feels like fun and scratches itches for many people, and there is an abundance of half-finished cool-looking pet projects.

Not many people enjoy the often dull investment of time and effort into adding maintainability to the pet projects of other people. That investment in higher engineering quality also has the effect of boosting the reputation of the initial authors of the proof-of-concept pet project, as their name is on the cover page of the project, rather than that of the somewhat invisible contributors that put in most of the hard and skilled effort of improving quality and maintainability of what is often an unfinished cool demo.

As a result the usual pattern is for people in the community to develop cool-looking half-begun pet projects, and then for commercial entities to pay some anonymous gophers to turn that into a maintainable product. Doing the opposite is a much harder sell to the community.

The better option for Canonical would be developing fully finished (that is with good quality documentation and engineering) tools with initially a limited functionality. Because of their good engineering they would be easier (thanks to documentation and clean and robust internal structure) to extend and a good example to potential extenders.

Then the community would have the incentive to enrich them in functionality rather than in quality and community contributors would be able to point at a specific aspect of functionality as its cover-page authors. Something like EMACS, or UNIX, or Perl, or Python, or git; their development largely happened like like that. I hope the reader sees the pattern...

But Canonical, or rather its clever employees, seem to prefer developing applications with half-begun but sprawling functionality, and opaque, messy, unrealiable, codebases. It probably is good for Canonical employee job security but less good for attracting voluntary contributions.

150823 Sun: Still happy with Dell U2412M and Acer B326HUL

I was using my recent 32in monitor B326HUL and I brought it home with some misgivings because my home computer desk is much smaller than the work desk. Indeed the 32in monitor barely fits in it and needs to be pulled a bit nearer to the viwer, but then my home computer desk has an extendible keyboard slat.

It look a bit out of place but it is very usable. Since the desk is small it would not have carried comfortably two sensibly sized monitors side-by-side, so it turned out that this was the best way for me to get lot more screen space.

I am still very happy about this monitor, both the size and the very high quality of the picture from its AMVA display. Some additional notes:

150822 Sat: Fast MacBook Air builtin flash, Corsair USB3 stick

I have just tested the builtin flash SSD of a MacBook Air and I have been astonished to see that it can read and write at 1.2GB/s:

$ time sh -c 'dd bs=1m count=200000 if=/dev/zero of=/var/tmp/TEST; sync'
200000+0 records in
200000+0 records out
209715200000 bytes transferred in 168.928632 secs (1241442599 bytes/sec)

real    2m49.217s
user    0m0.483s
sys     1m58.570s

I also recenty read in CustomPC magazine that some recent desktop motherboards have 4× M.2 slots and therefore can reach 700-800MB/s, while 2× M.2 slots and typical flash SSD SATA drives can reach at around 550MB/s.

The MacBook Air flash SSD considerably exceeds that and the high transfer rate seems made possible by direct connection to the system bus.

Quick testing also confirmed the manufactur's claim for transfer rate of a Corsair USB3 32GiB flash SSD stick at 170MB/s and write at around 70MB/s which is also quite impressive:

# lsscsi | grep Corsair
[4:0:0:0]    disk    Corsair  Voyager SliderX2 000A  /dev/sdb
# sysctl vm/drop_caches=1; dd bs=1M count=2000 if=/dev/sdb iflag=direct of=/dev/zero
vm.drop_caches = 1
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 12.0935 s, 173 MB/s