This document contains only my personal opinions and calls of judgement, and where any comment is made as to the quality of anybody's work, the comment is an opinion, in my judgement.
Today we were given a chance to see the first prototype wafer they will be using for production of 80-core processors. Each CPU like that will have 80 simple floating-point dies and each die is capable of teraflop performance and can transfer terabytes of data per second. They claim that it will be commercially available in a 5-year window and will be ideal for such tasks as real-time speech translation or massive search, for instance.To some extent the ClearSpeed CSX600 is already like that, with 96 FPUs each with around 6KiB of onchip memory for operands. It has a somewhat different usage profile, and I suspect that for now it costs a lot more than what would be the target for regular desktop usage.
automount
I have realized that I sometimes still want to disable it and
mount things manually and permanently. Unfortunately the
format of automounter maps different from that of
/etc/fstab therefore I would have to maintain the
same information twice. Fortunately the automounter can
synthetize map lines on the fly by invoking a script, and it
is pretty easy to select the relevant /etc/fstab
line (to which noauto should be added of course
if necessary) and turn it into an automounter map entry
dynamically, with a script like this:
#!/bin/sh
SP='[:space:]'
: '
We need to look up the key to find type, options and device to
mount. This means we need the name of an "fstab" file and the
prefix under which the key appears in it. These can be different
depending on the "autofs" mount point, under which this script
is run.
'
case "$PWD" in
*) FSTAB='/etc/fstab'; FSPFX="/fs/";;
esac
KEY="$1"
grep '^['"$SP"']*[^\#'"$SP"']\+['"$SP"']\+'"$FSPFX$KEY"'/\?['"$SP"']' \
"$FSTAB" | if read DEV DIR TYPE OPTS REST
then
case "$TYPE" in ?*)
case "$OPTS" in
'') OPTS="fstype=$TYPE";;
?*) OPTS="fstype=$TYPE,$OPTS";;
esac;;
esac
case "$DEV" in *':'*) :;; *) DEV=":$DEV";; esac
: '
We must omit the key in a map program like this.
key -[fstype=TYPE,][OPTION]* [HOST]:RESOURCE
'
echo "-$OPTS $DEV"
fi
and then running the automounter like this from
/etc/inittab:
af:2345:wait:/usr/sbin/automount -t 300 /media program /etc/auto.fstabBy the way, too bad that
automount is designed to
always background itself, because if it did not it that line
could have action respawn instead of
wait.
-t 300
seconds instead of 60 because some server I use intermittently
accesses some directories on some mounted filesystems, which
causes a large number of mounts. This adds pointless lines to
the log, and some filesystems like ext3 impose a
periodic check every so many mounts. To avoid the latter I
have actually disabled the force check every some number of
mounts and retained only the one every some number of days.microarchitecture. The first one definitely supports the AMD dual CPU chips vs. the
I have been able to try out a dual-dual-core Opteron machine, and it flies.which is not unexpected, while the second mentions that for something as highly parallel as a DBMS running many transactions, Intel's Hyper-Threading seemed to work pretty well too:
In fact, it flies so well that we ordered one that day. So, in short £3k's worth of dual-opteron beat the living daylights out of our Xeon monster. I can't praise the Opteron enough, and I've always been a firm Intel pedant - the HyperTransport stuff must really be doing wonders. I typically see 500ms searches on it instead of 1000-2000ms on the Xeon)
Actually, believe it or not, a coworker just saw HT double the performance of pgbench on his desktop machine.This is rather unexpected, as the author hints, because Hyper-Threading tends to result, even on custom coded applications, something like 20-30% better performance (I did some work on that custom coding).
Granted, not really a representative test case, but it still blew my mind. This was with a database that fit in his 1G of memory, and running windows XP. Both cases were newly minted pgbench databases with a scale of 40. Testing was 40 connections and 100 transactions. With HT he saw 47.6 TPS, without it was 21.1.
The machine is to be built entirely from commercially available hardwareso that explains why no custom dual Opteron and Cell BE boards, but the BladeCenter H is commercially available. I suspect that with several thousand racks involved a common physical format is not that important.
Roadrunner's construction will involve the creation of advanced "Hybrid Programming" software which will orchestrate the Cell B.E.-based system and AMD system and will inaugurate a new era of heterogeneous technology designs in supercomputing. These innovations, created collaboratively among IBM and LANL engineers will allow IBM to deploy mixed-technology systems to companies of all sizes, spanning industries such as life sciences, financial services, automotive and aerospace design.which seems to be a remarkably optimistic statement.
only2GiB of main memory, and while many 64b applications tend to run a bit faster than 32b ones on an AMD chip, they take significantly more memory. Not quite a doubling, because in practice only pointers double in size, but still often significantly larger.
proxy
cache, I also had a look at another one that seems especially promising:
Polipo is a small and fast caching web proxy (a web cache, an HTTP proxy, a proxy server) designed to be used by one person or a small group of people. I like to think that is similar in spirit to WWWOFFLE, but the implementation techniques are more like the ones ones used by Squid.The attention to detail in the features listed above is notable. I am still using Apache as a proxy cache though, because I have to run it anyhow as a web server, and seems to be adequate.
Polipo has some features that are, as far as I know, unique among currently available proxies:In short, Polipo uses a plethora of techniques to make web browsing (seem) faster.
- Polipo will use HTTP/1.1 pipelining if it believes that the remote server supports it, whether the incoming requests are pipelined or come in simultaneously on multiple connections (this is more than the simple usage of persistent connections, which is done by e.g. Squid);
- Polipo will cache the initial segment of an instance if the download has been interrupted, and, if necessary, complete it later using Range requests;
- Polipo will upgrade client requests to HTTP/1.1 even if they come in as HTTP/1.0, and up- or downgrade server replies to the client's capabilities (this may involve conversion to or from the HTTP/1.1 chunked encoding);
- Polipo has complete support for IPv6 (except for scoped (link-local) addresses).
- Polipo can optionally use a technique known as Poor Man's Multiplexing to reduce latency even further.
proxy
cachebut also occasionally Apache which has proxying and caching modules (they are distinct: Apache can proxy without caching and cache non-proxy requests, and both modules must be enabled to provide proxy caching like Squid).
<IfModule mod_proxy.c>
ProxyRequests on
Listen *:3128
NoProxy 127.0.0.1
NoProxy 192.0.2.0/24
NoProxy .sabi.co.UK
ProxyDomain .sabi.co.UK
ProxyVia on
ProxyTimeout 50
<Proxy *>
Satisfy any
Order deny,allow
Deny from all
Allow from 127.0.0.1
Allow from 192.0.2.
</Proxy>
</IfModule>
The supposed effect of the above is to allow proxying
(on port 3128 too, but then Apache does both local
service and proxy service on all ports); servers with
local addresses are not proxied, and only clients with
local addresses are allowed to proxy (unrestricted
proxying is a very bad idea). Even without
caching Apache proxying has a value in a mixed IPv4
and IPv6 environment, because Apache has full support
for both protocols, and it can proxy between them, so
that IPv6-only clients can access IPv4-only servers
and viceversa.
<IfModule mod_cache.c>
CacheDefaultExpire 3600
CacheMaxExpire 86400
CacheLastModifiedFactor 0.1
CacheIgnoreNoLastMod on
CacheIgnoreCacheControl off
<IfModule mod_disk_cache.c>
CacheEnable disk http://
CacheRoot /var/cache/httpd
CacheDirLength 1
CacheDirLevels 2
CacheMinFileSize 1
CacheMaxFileSize 900000
</IfModule>
<IfModule mod_mem_cache.c>
CacheEnable mem /
MCacheSize 10000
MCacheMinObjectSize 1
MCacheMaxObjectSize 200000
MCacheMaxObjectCount 2000
</IfModule>
CacheDisable http://[::1]/
CacheDisable http://127.0.0.1/
CacheDisable http://localhost/
CacheDisable http://ip6-localhost/
CacheDisable http://example.com/
CacheDisable http://.example.com/
</IfModule>
The supposed effect of the above is to allow disk
caching of proxy requests involving
HTTP
URIs (FTP
could also be cached, but I don't have a use for it) and memory
caching for content served by Apache itself; and to permit no
caching of any content on the local server or any server within
my own domain.
CacheEnable and
CacheDisable directives:
/ pattern would match both
local and proxy URIs.http:// would match no
URIs.they are stable (because they don't depend on your ISP),Here the argument is that in IPv6 all global unicast address range allocated to users are not
portable, because while IPv4 routed each subnet independently, IPv6 only allows routing by
aggregation, that is hierarchically. The argument above, expanded, looks like:
prefixfrom an ISP, assigns addresses under it to nodes, and then changes ISP, one gets a different prefix, and has to change the address of all nodes.
leakonto the Internet they are not routable because no Internet router (hopefully) will accept routes for those well known address ranges, and most will simply drop packets to or from such addresses (which are listed in well known bogon list).
intranetssometimes) with one way address translation to the global Internet.
Note: it is of course possible to create a separate intranet which reuses the full 32 bit address space of IPv4 instead of just the private address ranges. But while the latter only requires NAT one way (because the intranet and the Internet address sets then do not overlap, only the intranet addresses at different sites overlap), a full separate internet would require two way NAT, mapping in-use Internet addresses to a subset of the intranet's address space, as well as in-use intranet addresses to a subset of the Internet's address space, and addresses in the DNS protocol would have to be mapped too.
The reason why unicast site local addresses have been deprecated was not that they were ambiguous, but that they were ambiguous and without NAT: because if there had been NAT then nodes could not be really belong to multiple sites ambiguously. The zone suffixes were an attempt to disambiguate the site locale addresses, but using relative instead of absolute addressing, and as Robert Stroud's thesis showed if absolute addresses work but do not scale, relative address scale but do not work.The only alternative is to sacrifice a deterministic notion of identity by using random identifiers to approximate global uniqueness with a known probability of failure (which can be made arbitrarily small if the overall size of the system is known in advance).But why bother for IPv6? Why not allocate a proper prefix? There is very little difference between that and a (partially) randomly generated one. The control (and weak security) of IPv4 private addresses is derived mostly from the necessity to use a NAT proxy to gain any access to the proper Internet, not from ambiguity in itself.
2001:0630:0050::/48, and they could
have been given 2001:0630:0051::/48 as
well, with the idea that they would never publish any routing
table for the second prefix; or been given something like
fc00:0630:0050::/48 as well, with the idea that
no Internet router would route datagrams with addresses under
fc00::/16. This would force any node with addresses
under 2001:0630:0051::/48 or
fc00:0630:0050::/48 prefixes to route via a proxy
under 2001:0630:0050::/48 to communicate
with the Internet, with the proxy having full flexibility on
how to remap (NAT) either internal prefix to the externally
routable one.
social definitionof
works, and I made some examples of long standing bugs in software that I use:
clean all command, sometimes leaving hundreds
of MiBs of data in its cache.path MTU discovery, does not work well in IPv4, (largely because it was a later extension to IPv4 and many routers are misconfigured for it) and the other is that IP datagram size in IPv4 is limited to 64KiB. These issues have become increasingly frustrating and are discussed forcefully at the large MTU advocacy site in which MTU and datagram sizes of several hundred KiB are advocated for 10gb/s links, with a target of around 500 microsecond per packet, or 2,000 packets per second.
blackholeDNS servers:
The blackhole servers generally answer thousands of queries per second. In the past couple of years the number of queries to the blackhole servers has increased dramatically.Usual problem here: the average network administrator will not waste his precious time figuring out how to setup reverse mappings for the private address range he uses, especially if someone else pays the cost. Also, this reminds me of an article on Internet root DNS servers reporting that one of the most common queries to them was for the top level domain
It is believed that the large majority of those queries occur because of "leakage" from intranets that are using the RFC 1918 private addresses. This can happen if the private intranet is internally using services that automatically do reverse queries, and the local DNS resolver needs to go outside the intranet to resolve these names.
For well-configured intranets, this shouldn't happen.
WORKGROUP (never mind all
those desktops with MS Windows 2000 or later trying to
register themselves via DNS in the root servers). The lesson
learned by Michael Stonebraker long ago about
Ingres catalog indices:
Users are not always able to make crucial performance decisions correctly. For example, the INGRES system catalogs are accessed very frequently and in a predictable way.has been lost in time, like so many other others.
There are clear instructions concerning how the system catalogs should be physically structured (they begin as heaps and should be hashed when their size becomes somewhat stable). Even so, some users fail to hash them appropriately.
Of course, the system continues to run; it just gets slower and slower. We have finally removed this particular decision from the user's domain entirely. It makes me a believer in automatic database design (e.g., [11]).
For an example of that on the customer side, we want to improve performance. The engineers said: "Rather than guessing what's bottlenecking performance, let's go and measure what's actually going on." We instrumented all the Steam clients, and the answer was surprising. We thought that we should go and build a deferred level-loader, so that levels would swap in. It turned out that the real issue was that gamers' hard drives were really fragmented, and all of the technology we wanted wouldn't have made a difference, as we were spending all our time waiting on the disk-heads spinning round.Note also the implication: installing Valve games (at least via their Steam service) means installing some software that does an exhaustive scan of the hard disk and reports its contents to Valve. This is most likely well advertised in the license.
The solution that we're trying is to break things into smaller chunks and to do them more regularly. So far, it seems to be working. When we look at how long it took us to build a minute of gameplay for Half-Life 2, versus how many man-months it takes us to build a minute of gameplay for Episode One or Episode Two, we seem to be about four times as productive. But we'll go through all three episodes to see... We sort of made a commitment to do it three times and then assess.That
four times more productiveis a big thing. It reminds me claims by Mark Rein of Epic that games based on their Unreal engine development tools only require about 15 developers. My impression is that the case they are making is not about episodic content or development tools, it is that games that are really
mods, reusing a lot of the engine and art of previous games, are much cheaper to develop than games developed from scratch. Which probably is possible because most platforms have sort of leveled in terms of functionality, so game engines don't have to be rewritten from scratch frequently, and thus can become somewhat stable platforms. Never mind
middleware, whether third party libraries or tools: a game engine is in effect its own middleware, when used as a modding platform.
modscan be very profitable. Especially when they release them via Steam: they get the whole price, instead of a fraction of it when released retail. No surprise that Valve is rather cautious when talking about Steam:
What is the split between sales of Episode One via Steam and boxed sales?Fabulously disingenuous argument about promotion: the goal of online game delivery is not to promote games, for which one can do web sites and demos; it is is to cut out the middleman as much as possible, and that is also why massive online games like World of Warcraft are so enormously profitable for game developers (and most are not available for consoles, where the console brand owners really insist on getting a large cut). If online content becomes even more popular then games publishers will be reduced to the role of marketing and PR agencies, not resellers, and the other role that they perform, project venture capitalists, will probably split off into independent entities.
Gabe Newell: That isn't something that we've talked about. It's something we're keeping to ourselves.
So, how do you manage your relationship with EA when you're selling games via Steam?
Gabe Newell: Our relationship with EA is fine. I think that retailers are really frightened of these kinds of changes in the industry, and I think that we're learning stuff that is going to be very important for them. For example, Steam enables new ways of doing promotion: [ ... ] I think retailers are starting to understand that communicating more efficiently with customers is a way, not of taking money away from them, but of driving people into stores. It's not a way of cutting them out of the equation.
Right now, I think the benchmark game in the industry is World of Warcraft, and every platform could be measured against its ability to give advantage, or fail to give advantage, to building a better World of Warcraft.a notion that I have previously discussed as that game probably explains a lot of the falling sales of other PC games. As to that, I supect that if World of Warcraft were available on consoles, the sales of other console games would be impacted too.
One of the primary features is that the moving parts were all in the *plug* (all connectors have moving parts ... those little springy bits that apply pressure and make sure the connection is good and tight). This way the part of the connector/socket system that wears out is in the cable. When something goes wrong (as it will in any mechanical system), you throw out the cheap, easily replaceable component ... i.e., the cable.I was talking about this recently again in the context of a large scientific facility, which expects a lot of visiting scholars for relatively short periods of time. My argument was that for the user facing part of the network wireless network is a good idea (security can be sorted out) simply because it avoids a lot of plugging and unplugging of cabling by hasty and not very careful people, which is especially damaging as the insertion cycles to which many sockets are designed are pretty low.
This feature can also be used to debug a machine whose operating system has crashed, and in some systems for remote-console operations. On FreeBSD, the dcons driver provides both, with using gdb as debugger. Under Linux, firescope and fireproxy exist.These are classic examples of the
if life gives you lemons, make lemonadeprinciple.
2002 prefix encapsulation
for IPv4 addresses and then it is usually
autorouted
or explicitly configured and routed via a
tunnel broker.
Except in rare cases it is not supported by consumer grade
ADSL/cable gateways, but can be used if one connects to
the Internet via a simple
PtP connection
via a modem (either a phone or ADSL one). Autorouting
usually results in poor performance, so registering with a
6in4 tunnel broker is usually
better.sabishape
I was interested to find a nice detailed discussion about
shaping and QoS.
The author makes some useful points, for example about jitter
limitation requiring a far higher rate of examination
on low bandwidth links, and it is somewhat surprising how high
a rate is needed:
The only timer source of use at SME bandwidths are the high performance timers available in post-PentiumPro CPUs. These allow bandwidth estimation and policying at speeds lower than 64K on FreeBSD (with the HZ raised beyond 2KHz) and lower than 128K on Linux (at 1KHz HZ).and that high overhead, interrupt-per-packet cards are better for that purpose:
At the same time, cards that are considered "horrid" like Realtek 8139 (rl) provide much more interrupts and much more scheduling opportunities. As a result they provide considerably better estimator precision and policy performance. The difference is especially obvious on speeds under 2MBit. It is nearly impossible to achieve a working ALTQ hierarchy where some classes are in the sub-64Kbit range for a 2MB (E1) using Intel EtherExpress Pro (fxp). It is hard, but possible on a Tulip (dc). It is trivial to do this using Realtek (rl).The reason is that if we want to be sure that the maximum bandwidth used is under a certain level, the more variable it is, the lower the average must be to ensure the limit is not crossed. The overall discussion is quite agreeable, even if I think that:
Linux estimator entry points differ from BSD. As a result, the effects of hardware are less pronounced and system is more dependant on the precision of the timer source. Still, the same rules are valid for Linux as well. QoS on low bandwidth links (sub-2MB) cannot be performed on server class network hardware.
In reality the diagram is likely to contain 16-20 classes for an average company or 5-10 classes for an average home office network.exaggerates a bit. Because really one can do one class per source of traffic (e.g. sharer of the link) and three classes per type of traffic, like low, medium, high. More classes just make the situation more complex and don't really work that well especially if there is little bandwidth to share.
HTB is not a good choice for an SME or hobby network. Bandwidth will not be utilised fully and the link efficiency is considerably worse. Its only advantage is that its "ease of understanding" and "predictability" are easier to express in a subletting agreement.HTB is a a looser policy than CBQ (both also described in this Linux specific HOWTO) but it should not lead necessarily to lower utilization. If one specifies the ceiling for each class as the limit for the whole link.
ingress discipline. The goal of dropping packets
on incoming is in effect to simulate a congested link and thus
trigger quenching at the source; dropping packets regularly as
ingress does simulates a constant, steady
bandwidth limit. But letting incoming queue up and then drop
them as the queue becomes full simulates a full bandwidth link
that occasionally becomes very congested, and congestion
control algorithms don't react well to that. I'll ask the
author of the site about his reasons for that.getaddrinfo.sparse capabilitiesagainst networking scanning attacks.
init dæmon
it is pleasing to see that the
Ubuntu project
is replacing the standard System V style one with a
redesigned one called upstart
for which they have written a
neat paper describing it and comparing it
with similar projects like
OpenSolaris SMF,
initng
or
Apple's launchd.
init,
cron and inetd (and more). Never
mind that launchd uses XML for configuration
files...
inetd functionality, because it has been
traditional to allow a UNIX like
system to startup without any networking, for various reasons.
init (almost)
just runs enough scripts to reach some state of readiness and
back at shutdown, cron just runs
commands, not dæmons, and inetd
really manages sockets, not services.
upstart:
In fact, any process on the system may send events to the init daemon over its control socket (subject to security restrictions, of course) so there is no limit.which seems to indicate that it is yet another mess like
udev. And it was pretty scary to read that some
people wanted to integrate upstart with
D-Bus, another mess.
Overall I think initng is the more UNIX like
solution, simple and dependency based, but event based
upstart may be not too bad, even if it is already
decided that it will use /etc/event.d as its
configuration directory (UNIX style is not to have a
.d suffix on directories). Let's hope that the
socket is actually a named pipe, at least.coprocessors, network oriented or otherwise. Many years ago for example Weitek had line of extra-performance x87 socket compatible floating point coprocessors. One can now imagine putting all sort of things like that in the HyperTransport capable, or more probably HTX capable, AMD64 motherboards.
Direct Connect Architecturebus give it a considerable advantage for 4-chip and 8-chip systems, which do not therefore require the SMP chipsets that Intel CPUs need.
high memory supportis enabled, and then real memory above 896MiB is mapped temporarily in a subwindow of the per-process kernel memory area of 128MiB, which involves a modest slowdown and some complications.
CONFIG_PAGE_OFFSET setting in the
.config file when building the kernel, or for
older kernels in redefining the macro __PAGE_OFFSET
in the kernel header include/asm-i386/page.h
before building it. The default is 0xC0000000
and the values that I think are sensible are:
| Boundary | Process space |
Kernel space |
Real memory window |
|---|---|---|---|
0xB8000000 |
2944MiB 3GiB-128MiB |
128MiB | 1GiB |
0xC0000000 |
3GiB | 128MiB | 896MiB 1GiB-128MiB |
0x98000000 |
2432MiB 2.5GiB-128MiB |
128MiB | 1.5GiB |
0x78000000 |
1920MiB 2GiB-128MiB |
128MiB | 2GiB |
0x38000000 |
896MiB 1GiB-128MiB |
128MiB | 3GiB |
0x78000000, where the real memory map window is
2GiB and the per process address space is just under that, as
this allows direct mapping of most desktop real memory sizes,
and the 1920MiB per process address space is still pretty
large, and it still allows the full amount of real memory to
be used up by a single process.
0xC0000000,
because unless high memory is enabled it wastes 128MiB of a by
now common 1MiB real memory endowment, for the dubious benefit
of a 3GiB per process address space. More useful would have
been 0xB8000000 as at least it allows full
mapping of the 1GiB, at the insignificant cost of 128MiB less
of per process address space.BCI's gCORE processors employ a new Grid on Chip architecture which arranges system elements such as processor cores, memory, and peripherals on an internal "grid" network. gCORE is one of the first commercial applications of "network on chip" technology, that has been at the forefront of research at leading universities in recent years. It is widely accepted that traditional bus architectures are no longer valid for large scale system on chip implementations in the sub 90 nanometer geometries. Traditional buses become too large, and too slow to support 16 or more processor cores. Ease of use has been one of the biggest obstacles for the widespread adoption of multi-core processors. BCI has taken an unique approach of incorporating a "Time Machine" module in the chip to dynamically assign tasks to each of the processor cores. By alleviating the need to explicitly program each core, this approach greatly simplifies the software development process.This chip is also designed to run Linux. Not surprising: just as the ready and cheap availability of UNIX source licenses considerably reduced initial cost to develop new minicomputers and workstation systems in the 1980s, Linux has done the same for small servers and PCs and even desktop boxes and ADSL routers in the 1990s.
equivalentperformance to a 5GHz processor, which is pretty remarkable.
partitionsper hosting PC, and with an OCTEON (or UltraSPARC-T1) CPU they can provide a real CPU per user, which solves a number of issues, and for a lot less watts than an equivalent 3-5GHz x86 style single-CPU chip.
microformatsare used to tag data for automatic processing (as the Semantic Web is not quite here yet), I feel like confessions that I use the sensible alternative to microformats, which is finer resolution text tagging. The reason for this is that often there is a need to put in evidence both some
parts of speechand point out
types of discourse(usually in the special case, but not only, of
levels of discourse, where the type is the degree of abstraction).
abbr and acronym.
proper names, because some proper names are not easy to distinguish from ordinary words (many are derived from them). In many languages there is some convention to indicate a proper name, and the one I use is to capitalize the first letter of each word of a proper name. I also reckon that the
cite tag of HTML is the neatest way of
tagging proper names in general. So I would write:
<cite>Smith</cite> is an engineer, not a smith.Then however there are different types of proper names, and I wish often to be able to differentiate them. This is particularly valuable in talking about computer and business related matters, because often different entities, like companies and their products, have the same proper name; and many companies and products have all-lower case or mixed case names, and first letter capitalization cannot be used to indicate a proper name role for the word. For this I use not just the
cite tag but
also the class attribute to indicate what
type of proper name is being tagged, and then a slightly
different
CSS rule
to give them slightly different renderings. For example,
I would write:
<cite class="corp">Oracle</cite>'s main product is <cite class="thing">Oracle</cite>, and only an oracle can predict whether they will release a similarly name Linux distribution.because Oracle is the name of a corporate person and Oracle that of a thing. I have tried not to define too many classes of proper names, and currently the values I use for
class of a cite element are
corp for corporate persons),
thing for objects, place for
locations, and uom for units of measure.
cite
to tag bibliographic citations, not just citations of
proper names, thus I also have classes
author for proper names of authors,
title for names of the article or book,
part for names of the specific part of a
book, which for the name of the issue,
publ for publisher names, and
date for the date of publication.
blockquote for large textual quotations and
q for smaller quotations, and
code for quoting code and a few others.
q to
indicate the type of discourse of single words or longer
sequences, for example fl for foreign
language, nlq for non-literal quotes, and
toa for terms of art.
A <q class="toa">type of discourse</q> can be recognized because it must be read in a non-<q class="nlq">plain</q> way to get the correct meaning: in <cite class="thing">French</cite> the word <q class="fl">chair</q> does not mean <q>chair</q>, but <q>flesh</q>.Apart from being useful to remove ambiguities, the use of finer pseudo-tagging of text has other advantages: it becomes a lot easier to search for stuff in text. For example, using tools like
sgrep,
a version of grep
which has
matching operators
specifically for
SGML
style syntax.-dpi dpi in
the X server's command line (usually X server command lines
are specified in the display manager's configuration file);
Monitor section's DisplaySize
directive.microformats:
Every once in a long while, I read about an idea that is a stroke of brilliance, and I think to myself, "I wish I had thought of that, it's genius!" Microformats are just that kind of idea. You see, for a while now, people have tried to extract structured data from the unstructured Web. You hear glimmers of these when people talk about the "semantic Web," a Web in which data is separated from formatting. But for whatever reason, the semantic Web hasn't taken off, and the problem of finding structured data in an unstructured world remains. Until now.Ahhhhh the pain the pain, the memories :-). In a discussion a long time ago I was not happy with the tendency to use SGML or XML to define not markup languages but data description languages:
Microformats are one small step forward toward exporting structured data on the Web. The idea is simple. Take a page that has some event information on it -- start time, end time, location, subject, Web page, and so on. Rather than put that information into the Hypertext Markup Language (HTML) of the page in any old way, add some standardized HTML tags and Cascading Style Sheet (CSS) class names. The page can still look any way you choose, but to a browser looking for one of these formatted -- or should I say, microformatted -- pieces of HTML, the difference is night and day.
and here I find a variant of that monstrosity. The examples of microformats provided are also particularly repulsive because of less than optimal choice of the HTMLI'll spare myself the DTDs, but consider two instances/examples of two hypothetical SGML architectural forms; the first is called MarkDown: <ELEMENT TAG=html> <ELEMENT TAG=head> <ELEMENT TAG=title TEXT="A sample MarkDown document"></> </> <ELEMENT TAG=body ATTRS="bgcolor" ATTVALS="#ffffff"> <ELEMENT TAG=h1 ATTRS="center" ATTVALS="yes" TEXT="What is MarkDown?"></> <ELEMENT TAG=p> MarkDown is a caricature of SGML; it is an imaginary architectural form whose semantics are document markup, where the <ELEMENT TAG=code TEXT="tag"></> attribute is the one to which the MarkDown semantics are attached. </> </> </> Now this is a monstrosity, but I hope the analogy is clear, even if a bit forced in some respects.
tagsto pervert into data descriptions:
<div class="vevent">
<a class="url" href="http://myevent.com">
<abbr class="dtstart" title="20060501">May 1</abbr> -
<abbr class="dtend" title="20060502">02, 2006</abbr>
<span class="summary">My Conference opening</span> - at
<span class="location">Hollywood, CA</span>
</a>
<div class="description">The opening days of the conference</div>
</div>
as there is too much use of generic tags like div
and span, and the abuse of abbr,
where the data is in the title attribute and its
verbose description as the body of the abbr
element.
Now trying to imagine the above data as text the second
example might be less objectionably marked up as:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<style><!--
dt { font-weight: bold; }
cite.title:before { content: open-quote; }
cite.title:after { content: close-quote; }
cite.location:before { content: "["; }
cite.location: { font-style: normal; }
cite.location:after { content: "]"; }
q.abstract:before { content: ""; }
q.abstract { display: block; font-size:90%; }
q.abstract:after { content: "."; }
--></style>
</head>
<body>
<dl class="vevent">
<dt id="20060501-20060502">
<a href="http://WWW.MyEvent/#opening">
<abbr class="dtstart" title="May 1 2006">20060501</abbr> to
<abbr class="dtend" title="May 2 2006">20060502</abbr></a></dt>
<dd><cite class="title">My Conference opening</cite>
<cite class="location">Hollywood, CA</cite>:
<q class="abstract">The opening days of the conference.</q></dd>
<dt id="20060503-20060504">
<a href="http://WWW.MyEvent/#closing">
<abbr class="dtstart" title="May 3 2006">20060503</abbr> to
<abbr class="dtend" title="May 4 2006">20060504</abbr></a></dt>
<dd><cite class="title">My Conference closing</cite>
<cite class="location">Hollywood, CA</cite>:
<q class="abstract">The closing days of the conference.</q></dd>
</dl>
</body>
</html>
It might seem similar, but it isn't: because my version is
just text with text markup, structured as text, not data (and
never mind the use of SGML and XML instances that contain only
markup, with no data or text). The idea of microformats is to
decorate structured data with pseudo-markup so programs can
extract individual data elements more easily. As the already
mentioned by
this blogger
this is an abuse of HTML, which is about text, not data,
markup, where some XML instance would be more appropriate.
However there is a decent case to be made for finer
resolution markup of text so that parts of it may be
easier to identify and extract, and doing this by some
finer grain HTML markup is not too bad. Using the
class attribute to indicate finer classes of
semantics is a bit of an abuse, as they are meant to
indicate finer classes of rendering, but one can
make the case that different classes of rendering do
relate to finer classes of meaning, at least in the eye
of the average beholder.RSS column in
the output of ps on Linux and I said that in theory
is the number of resident pages the process has, but that is
somewhat unsatisfactory.
RSS, and one is incidental and the other
deeper. The incidental one is that shared library (or other
mappings) resident pages are accounted for in every process
using them, with the result that the total sum of the
RSS figures is larger than the memory actually
used.
exmap
which accounts for share mapping resident pages in proportion
to how many processes share them:
Exmap is a memory analysis tool which allows you to accurately determine how much physical memory and swap is used by individual processes and shared libraries on a running system. In particular, it accounts for the sharing of memory and swap between different processes.Now this useful tool accounts for shared pages equally among processe, but there might be other ways of accounting, like a more dynamic usage based count. But it already needs a kernel extension just to collect per-process ownership data for pages, because the Linux kernel does not:
To my knowledge, other tools can determine that some memory is shared, but can't determine how many processes are making use of that memory and so fairly apportion the cost between the processes making use of it.
Exmap uses a loadable kernel module to assign a unique id to each physical or swap page in use by each process. This information is then collated and 'effective' usage numbers calculated.That the Linux kernel does not already keep track of this leads to the deeper issue with determining how much memory a process is actually using: that Linux uses a global replacement policy for memory management, that is it treats all processes together, in other words pages from different processes compete as to residency.
working setof pages that are most active in that process, and they try to estimate only which pages should be part of that working set. The reason why that is better done per process than globally is that the working set of a process is supposed to change with time but slowly, under the
phase behaviourhypothesis: that processes execution is in distinct phases, and each phase usually has a different working set, which is something that a global policy does not take advantage of.
/etc/fstab to
an automounter map
for most of my filesystem I have been disappointed to see that
I missed a subtle detail that means that the locate
database created by updatedb only lists files in
filesystems mounted for other reasons.
-g
(ghosting) option to
automount
to create directories to act as virtual mount points, which
updatedb would then descend, triggering the
mounting of the relevant filesystem. But currently I
use Fedora
on my desktop PC, and the updatedb in it is from
the highly optimized
mlocate variant,
which checks whether a directory is empty before descending in
it, and unfortunately the mere check does not trigger the
mounting of the filesystem, and the mountpoint directory
created by the -g option is empty before the
filesystem is mounted. Indeed I just checked and the
stat64 system call does not trigger mounting, but
getdents or getxattr trigger the
mounting. Which is incorrect, because mounting should be
triggered by any access to the inode or its data contents, not
just the data contents (the extended attributes read by
getxattr are in not in the inode).
updatedb in
mlocate to scan the relevant directories explicitly,
which triggers the mount, but then the locate
databases gets overwritten unless I create a separate database
file. But well, I can do so indeed, and simplify several
issues, by creating a separate mlocate database
for each filesystem in a given list. The database by default
will be in the top directory of the filesystem, but optionally
the list will have a second field for an explicit name (to
cater to the case where the filesystem is read-only).
/etc/cron.daily/mlocate.cron file now
contains:
/usr/bin/updatedb
DBDIRS='/etc/updatedb.dirs'
if test -e "$DBDIRS"
then
grep -v '^[ \t]*#\|^[ \t]*$' "$DBDIRS" \
| while read DIR DB
do : ${DB:="$DIR/mlocate.db"}
/usr/bin/updatedb -U "$DIR" -o "$DB"
done
fi
and my profile initializes the
LOCATE_PATH environment variable like this:
if test -e "$DBDIRS"
then
export LOCATE_PATH
dbdirspath() {
DBPATH="$1"; DBDIRS="$2"
{ grep -v '^[ \t]*#\|^[ \t]*$' "$DBDIRS"; echo ''; } \
| while read DIR DB
do
case "$DIR" in
'') echo "$DBPATH";;
?*) : ${DB:="$DIR/mlocate.db"}
case "$DBPATH" in
'') DBPATH="$DB";;
?*) DBPATH="$DBPATH:$DB";;
esac;;
esac
done
unset DBPATH
unset DIR
unset DB
}
LOCATE_PATH="`dbdirspath \"$LOCATE_PATH\" \"$DBDIRS\"`"
fi
/etc/inittab
perhaps some discussion of init is useful to
justify that.
init was a simple thing that just
run some script chain, and the script chain would define a few
nested run levelscorresponding usually to
modes, for example single user mode, multi user mode, where the system would pass through each lower level to the target level on startup, and viceversa on shutdown.
init would execute first
thing something like a /etc/singleuser script,
which would contain single user initialization commands, a call
to /etc/multiuser and single user termination
commands; /etc/multiuser would contain multiuser
initialization code, and a call to an upper level or just the
spawning of getty, and multiuser termination
commands.
levels), in the sense that states are not ordered and the system can go from any state to any other state. To support this they added a configuration file
/etc/inittab where each line is tagged by the
states in which it is valid.
init can
monitor it and restart it if it terminates.
init program reads the configuration file,
and when requested to switch run state, lists all commands that
are unique to the source and destination state, and the common
ones, dividing the latter in simple and dæmons, and does the
following:
/etc/inittab.
This creates a problem if one has to goal to allow package
installation without manual intervention, because the package
may be a dæmon
init, only much worse. Because it is
based on having nested run levels by convention (and many
distributions use different conventions, even if there is a
nominal Linux standard)
where however is not checked because the scripts run at each
level are put in a separate per-level directory. Even more
insanely, which script is run in which level is not configured
in a file, but by the mere absence or presence of a script (or
a symbolic link to one) determines what is run.
init model, with
variants, so for example to enable/disable individual services
RedHat uses files in /etc/sysconfig, SUSE first
used the file /etc/rc.config and then this was
split into separate files under etc/rc.config.d,
and Debian goes for gold by having only two levels, and which
scripts run in the upper one is determined solely by
configuration in various random files.
init, for example by:
once, wait).include all the files in
a directory.init redesigns. There are however very small
chances that any of them will be used in a production
distribution, as the inertia of historical accidents probably
will prove too strong. However it may be still interesting to
have a look at some, for example
LFSinit,
boot-scripts,
the NetBSD 1.5 rcorder system. for (unit = 0; unit < MAX_DRIVES; ++unit) {
ide_drive_t *drive = &hwif->drives[unit];
if (hwif->no_io_32bit)
drive->no_io_32bit = 1;
else
drive->no_io_32bit = drive->id->dword_io ? 1 : 0;
}
The perhaps inadvertent double obfuscation of the logical
condition is all too typical. I would have written:
{
const ide_drive_t *const end = &hwif->drives[MAX_DRIVES];
ide_drive_t *drive;
for (drive = &hwif->drives[0]; drive < end; drive++)
drive->no_io_32bit = hwif->no_io_32bit || drive->id->dword_io;
}
I also wonder whether scanning all drives up to
MAX_DRIVES is correct, without checking whether
the drive is actually present. But probably harmless. As a
final note, the code above is not, by far, the worst I have
seen recently; it is just obviously lame./etc/fstab and the boot scripts. The two are
mostly equivalent, but the automounter maps are used by a
dæmon to mount filesystems dynamically instead of
statically and on usage; mounted filesystems are then unmounted
after they haven't been accessed for a while.
mount command before accessing a filesystem, which
is trifling issue, but that they do so dynamically,
that is filesystems only stay mounted for as long as they are
used. This has the not inconsiderable advantage that most
filesystems will stay unmounted most of the time, and an
unmounted filesystem is cleanand does not need to be checked if there is a crash, and this is particularly useful if one does system development and these crashes occur during debugging. Sure, most file system types have journaling and they recover fairly quickly, but because of my PC has MS Windows dual boot, I also have a few (for historical reasons) FAT32 and
ext2 filesystems
that don't have journaling. Moreover keeping filesystems
inactive and unmounted reduces other chances of accidental
damage.
/fs, called /etc/auto.fs
and it has two sections, one for mounting filesystem (mostly
from removable media) that might exist on any system, and
another for mounting PC specific filesystems, and it looks
like:
# vim:nowrap:ts=8:sw=8:noet:ft=conf #MOUNTP -fstype=TYPE[,OPTION]* :RESOURCE # Host independent ################## 0 -fstype=ext2,user,rw,defaults,noatime,nosuid,nodev :/dev/fd0 a -fstype=vfat,user,rw,nocase,showexec,noatime,umask=077 :/dev/fd0 A -fstype=msdos,user,rw,noatime,umask=077 :/dev/fd0 1 -fstype=ext2,user,rw,defaults,noatime,nosuid,nodev :/dev/fd1 b -fstype=vfat,user,rw,nocase,showexec,noatime,umask=077 :/dev/fd1 B -fstype=msdos,user,rw,noatime,umask=077 :/dev/fd1 sda1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/sda1 sdb1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/sdb1 sdc1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/sdc1 sdd1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/sdd1 sde1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/sde1 sdf1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/sdf1 uba1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/uba1 ubb1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/ubb1 ubc1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/ubc1 ubd1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/ubd1 ube1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/ube1 ubf1 -fstype=vfat,user,ro,nocase,showexec,noatime,umask=077 :/dev/ubf1 pkt -fstype=udf,user,rw :/dev/pktcdvd/0 udf -fstype=udf,user,ro :/dev/cdrom cd -fstype=cdfs,user,ro,mode=0775,exec :/dev/cdrom iso -fstype=iso9660,user,ro,mode=0775,exec :/dev/cdrom r -fstype=iso9660,user,ro,mode=0774,norock :/dev/cdrom pkt1 -fstype=udf,user,rw :/dev/pktcdvd/1 udf1 -fstype=udf,user,ro :/dev/cdrom1 cd1 -fstype=cdfs,user,ro,mode=0775,exec :/dev/cdrom1 iso1 -fstype=iso9660,user,ro,mode=0775,exec :/dev/cdrom1 s -fstype=iso9660,user,ro,mode=0774,norock :/dev/cdrom1 # Host dependent ################ home -fstype=jfs,defaults,noatime :/dev/hda2 c -fstype=vfat,rw,nocase,showexec,noatime,umask=02 :/dev/hda3 d -fstype=ext3,rw,defaults,noatime :/dev/hda6 e -fstype=ext3,rw,defaults,noatime :/dev/hda7Interestingly
autofs maps can be scripts
that given the mountpoint print the options and resource to
mount on it, but for my simple purposes of /etc/fstab
replacement that is not necessary, a static map is good enough.
I have of course kept an /etc/fstab, but it now
contains only the indispensable static mounts, which are about
virtual filesystems and swap:
# vim:nowrap:ts=8:sw=8:noet: #DEVICE MOUNT TYPE OPTIONS DUMP PASS # Host independent ################## none /proc proc auto,defaults 0 0 none /sys sysfs auto,defaults 0 0 none /dev/pts devpts auto,mode=0620,gid=tty 0 0 none /proc/bus/usb usbfs auto,defaults 0 0 none /proc/nfs/nfsd nfsd noauto 0 0 none /var/lib/nfs/rpc_pipefs pipefs noauto 0 0 # Host dependent ################ /dev/hda1 / jfs defaults,errors=remount-ro 4 1 none /tmp tmpfs auto,mode=0777,size=700m,exec 0 0 none /dev/shm tmpfs auto,mode=0777,size=100m,exec 0 0 /dev/loop7 swap swap noauto,swap,pri=8 0 0Instead of using the
autofs rcscript to start the
automount dæmon, I have
chosen to start it directly in /etc/inittab,
because that is where it should be, and thus I have added these
lines to it:
# Use 'automount' instead of '/etc/fstab'. Local and remote maps. af:2345:wait:/usr/sbin/automount -g -t 60 /fs file /etc/auto.fs aa:345:wait:/usr/sbin/automount -g -t 180 /am file /etc/auto.amNote that I have specified the
ghostingoption
-g to make the mountpoints visible under
the main mountpoint, and slightly different and somewhat
longish autounmounting timeouts. One can always issue
umount explicitly if one wants a quick unmount,
for example for a removable medium.
/usr and
/var if separate) should in any case be specified
in /etc/fstab and be mounted statically at boot.
Surely nowhere as fragile as udev. Another
possible downside is that mounting is slow for some file system
types, most notable ReiserFS, because some extended consistency
checks are performed. In such a case perhaps lengthening the
default mount timeout is a palliative.
autofs
I might have used
AMD.
AMD is a similar dæmon which is system independent
(instead of using the special autofs module of
Linux it pretends to be an NFS server) and rather more
sophisticated. AMD is more suitable for large networked
installations, where it has considerably simplified my work a
few times, and I just wanted a dynamic extension to
/etc/fstab and for that autofs is
quite decent.
autofs in effect is a subset of the ability in
Plan 9
to
mount processes in the filesystem name space,
and vaguely similar to the
BSD
portalfeature for example as in
mount_portalfs.