Automation and its Limits

What we do in IT is programming and implementing and maintaining automatism. The general usefulness is not really challenged these days, but it was quite a big deal a few decades ago.
Many routine tasks can be done more efficiently and more reliably or become feasible at all with IT…

Just think of getting money from your bank account. Banks typically offer the possibility to do that at the bank counter without extra charges or any questions. But such a trivial operation can be done at the ATM, even while the counter is closed and I guess it is quite uncommon to go to the bank counter for that. There are more complex issues in banking that do require human interaction and they can be addressed with a human.

Buying a railroad ticket can be as trivial as getting money from the ATM, if the software of the ticket vending machine is really good, the tariff system is somewhat transparent, understandable and simple or at least the complexities can be covered well by the software. Most tickets are bought from the ticket vending machine, but the human at the ticket office in the railroad station is still important for more complex tickets that are not covered at all by the machine or that are too complex to buy there. It is interesting to see what can work and what not, maybe in another article in the future…

Phone lines are more and more answered by a machine, which has been given a beautiful voice by a human speaker… „For ABC press 1, for DEF press 2,….“ and so on. That works well for the trivial and most common cases. No matter how good the software is, usually at some point a limit is reached and we do want to find a human. I wanted to find out if there is a packaging service at some airport. It was possible to call, but there was no possibility at all or at least not within one hour of trying, to get to a human through these menus. So in the end the question remained unanswered. For the railroad ticket this can be an issue, if a more complex ticket is needed in a minor railroad station, but that can be solved nicely by providing a phone number (with real humans in the call center) that can be called and getting the ticket printed in ticket vending machine. This is possible in Germany and quite useful and basically covering most needs, even if talking with a real person is maybe nicer than a phone call with a dying battery… In Switzerland the ticket vending machines have a VOIP connection to the call center, so you do not even need to use the phone, but I think they are constrained to what can be done on the ticket vending machine GUI, so it does not help to solve the more difficult cases… Maybe in the future as well.

More delicate issues are the nanosecond trading. Trading (they do not like to call it gambling) on the stock exchange, with foreign currencies and all kinds of derivatives is done by algorithms automatically. Using software that is faulty, using the same software in many places, more or less. And also the risk management is done like that. It could be seen positive, because some traders tend to lose the sense of reasonable risk and machines kind of eliminate the emotions that can be counter productive in trading. But the risk takers are still there, setting out more automated trading systems… Will these systems one day create the real stock market crash by running into weird directions because of having the same software bug or because a situation not anticipated when the software was programmed occurs? Is it good to take such high risks without human control? Or it is even more delicate, if algorithms decide who is a terrorist and send drones, off course without a court order…

An issue that is mentioned quite a lot these days is motor vehicles operated automatically. While I do question the excessive use of cars of our time, I do think that that is an improvement, because human drivers are not very good and too risky. For rail transport this is much easier to achieve and many modern subway system operate automatically without any known problems.

We should always ask the question, does this IT solution and the automation it provides serve the humans and provide an advantage. That is the stuff we should build and operate.

Share Button

MS-Windows-Encodings with CMD: Bug or Feature?


Whoever is working with MS-Windows, should know these black windows with CMD running in them, even though they are not really popular. The Unix and Linux guys hate them, because they are really primitive compared to their shells. Windows guys like to work graphically. Or they prefer powershell or bash with cygwin. Linux and Unix have the equivalent of these windows, but usually they are white. Being able to configure the colors on both systems in any way this is of no relevance.

NT-based MS-Windows systems (NT 3.x, 4.x, 2000, XP, Vista, 7, 8, 10) have several subsystems and programs are running in them, for example Win64, Win32 (or Wow64 on 64-bit-systems), Win16, cygwin (if installed), DOS… Because programs for the DOS subsystem are typically started in a CMD window, and because some of the DOS commands have equally named and similarly operating pendents in the CMD window, the CMD window is sometimes called DOS-window, which is just incorrect. Actually this black window comes into existence in many situations. Whenever a program is started that has input or output (stdin, stdout, stderr), a black window is provided aroudn, if no redirection is in place. This applies for CMD. Under Linux (and Unix) with X11 it is the other way round. You start the program that provides the window and it automatically starts the default shell within that window, unless something else is stated.

Now I recommend an experiment. You just need an MS-Windows installation with any graphical editor like emacs, gvim, ultraedit, textpad, scite, or even notepad. And a cmd-window.

  • Please type these commands, do not use copy/paste
  • In the cmd-window cd into a directory you may write in.
  • echo "xäöüx" > filec.txt. Yes, there are ways to type these letters even with an American keyboard. :-)
  • Open the file with a graphical editor. How do the Umlauts look?
  • Use the editor to create a second file in the same directory with contents like this: yäöüy.
  • view it in CMD:
  • type fileg.txt
  • How do the Umlauts look?

It is a feature or bug, that all common MS-Windows versions are putting the umlauts to different positions then the graphical editors. If you know how to fix this, let me know.

What has happened? In the early 80es MS-DOS came into existence. By that time standards for character encoding were not very good. Only ASCII or ISO-646-IRV existed, which was at least a big step ahead of EBCDIC. But this standardized only the lower 128 characters (7 Bit) and lacked at some characters for almost any language other than English. It was tried to put a small number of these additional letters into the positions of irrelevant characters like „@“, „[„, „~“, „$“ etc. And software vendors started to make use of the upper 128 characters. Commodore, Atari, MS-DOS, Apple, NeXT, TeX and „any“ software came up with a specific way for that, often specific for a language region.

These solutions where incompatible with each other between different software systems, sometimes even between versions or language versions of the same software. Remember that at that time networks were unusual and when they existed, they were proprietary to the operating system with bridge solutions being extremely difficult to implement. Even formats for floppy disks (the three-dimensional incarnations of the save button) had proprietary formats. So it did not hurt so much to have incompatible encodings.

But relatively early X11 which became the typical graphical system for Unix and later Linux started to use standard encodings like the ISO-8859-x family, UTF-8 and UTF-16. Linux was already on ISO-8859-1 in version 0.99 in the early 90es and never tried to invent its own character encoding. Thank god for that….

Today all relevant systems have moved to Unicode standard and standardized encodings like ISO-8869-x, UTF-8, UTF-16… But MS-Windows has done that only partially. The graphical system is using modern encodings or at leas Cp1252, which is a decent approximation. But the text based system with the black window, in which CMD is running, is still using encodings from the MS-DOS times more than 30 years ago, like Cp850. This results in a break within the system, which is at least very annoying, when working with cygwin or CMD-windows.

Those who have a lot of courage can change this in the registry. Just change the entries for OEMCP and OEMHAL in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage simultaneously. One of them is for input, the other one for output. So if you change only one, you will even get inconsistencies within the window… Sleep well with these night mares. :-)
Research in the internet has revealed that some have tried to change to utf-8 (CP65001) and got a system that could not even boot as a result. Try it with a copy of a virtual system without too much risk, if you like… I have not verified this, so maybe it is just bad rumors to create damage for a great company that has brought is this interesting zoo of encodings within the same system. But anyway, try it at your own risk.
Maybe something like chcp and chhal can work as well. I have not tried that either…

It is up to you if you consider this whole issue a bug or a feature.

Share Button

Code examples in WordPress

When writing word press articles and including example code with <code>, the leading spaces are discarded when the article is displayed and without indention code looks weird and unreadable.

A way to bypass this is to transform tabs to spaces, and then replace all spaces by no-break-spaces (unicode 0x00A0). Then the indention remains intact. Direct copy paste from the blog to an editor or IDE will not be resulting in runnable code. But for non trivial examples I would recommend to put the code on github and add a link to the article. Code sniplets in articles should be mostly for human readers.

Share Button

Design Patterns: Singleton


This Singleton Pattern has the advantage to be easy to memorize.

The only really interesting aspect of it is the issue of initialization („lazy or „eager“) and maybe the dependencies between multiple singletons.

But I would like to mention two generalizations.

A Singleton exists once in the whole program. Generalizations can address the uniqueness in two ways. Either the number of instances can be increased from one to a small number or the universe („whole program“) can be changed. What is the universe in a big software system that is running on multiple servers with many processes and components?

Changing from one to a small fixed finite number is quite a routine thing. Java developers are calling this enum, but other programming language often have similar structures or at least allow to build them easily by restricting the generation of objects and creating the whole set of instances statically and making them accessible. I am not talking about the enum of C, C++ and C#.

The other, slightly more complex generalization looks at the „universe“. An application can be distributed and have several parallel processes, not only threads. This would mean dealing with a larger universe, which can become slightly tricky, but some examples of that are actually routine. Think of a database that exists once in the application to allow operating on the same data and keeping them consistent. Frameworks sometimes allow for an application server wide more or less classical singleton that exists once in the whole server landscape but can be accessed more or less transparently. An interesting case that is needed quite often is a global counter. It can be done using a DB-sequence. Or giving disjoint subsets of the numbers that could be generated to different processes and server. Or by using something like a UUID and praying that no collisions will occur. But an application wide singleton might be useful in some cases. Some frameworks are talking about a „bean“ with „application scope“. On the other hand smaller universes are even more useful. A typical example is the „session scope“.

Maybe just understanding these as a proper generalization of the singleton pattern will help in making these frameworks more understandable and more useful. The investment of having started with the most simple design pattern might actually pay off. Looking into some of the other patterns could be a worth while task for the future… 😉

Share Button

Using Collections

When Java came out about 20 years, it was great to have a decent and quite extensive collection library available as part of the standard setup and ready to use.

Before that we often had to develop our own or find one of many interesting collection libraries and when writing and using APIs it was not a good idea to rely on them as part of the API.

Since Java 2 (technical name „Java 1.2“) collection interfaces have been added and now the implementation is kind of detached, because we should use the interfaces as much as possible and make the implementation exchangeable.

An interesting question arose in conjunction with concurrency. The early Java 1 default collections where synchronized all the way. In Java 2 non synchronized variants were added and became the default. Synchronization can be achieved by wrapping them or by using the old collections (that do implement the interfaces as well since Java 2).

This was a performance improvement, because most of the time it is expensive and unnecessary overhead to synchronize collections. As a matter of fact special care should be used anyway to know who is accessing a collection in what way. Even if the collection itself does not get broken by simultaneous access, your application most likely is, unless you really know what you are doing. Are you?

Now it is usually a good idea to control changes of a collection. This is achieved by wrapping it with some Collections.umodifyableXXX-method. The result is that accessing the wrapped collection with set or put will cause an exception. It was a good approach, as a first shot, but not where we want to be now.

Of the wrapped collection still references to the inner, non-wrapped collection can be around, so it can still change while being accessed. If you can easily afford it, just copy collections when taking them in or giving them out. Or go immutable all the way and wrap your own in an umnodifiable-wrapper, if that works.

What I would like to see is something along the following lines:

  • We have two kinds of collection interfaces, those that are immutable and those that are mutable.
  • The immutable should be the default.
  • We have implementations of the collections and construction facilities for the immutable collections
  • The immutable implementation is off course the default.

I do not want to advocate going immutable collections only, because that does come at a high price in terms of efficiency. The usual pattern is to still have methods that modify a collection, but these leave the original collection as it is and just create a modified copy. Usually these implementations have been done in such a smart way that they share a lot, which is no pain, because they are all immutable. No matter how smart and admirable these tricks are, I strongly doubt that they can reach the performance of modifiable collections, if modifications are actually used a lot, at least in a purely single threaded environment.

Ruby has taken an interesting approach. Collections have a method freeze that can be called to make them immutable. That is adding runtime checks, which is a good match for Ruby. Java should check this at compile time, because it is so important. Having different interfaces would do that.

I recommend checking out the guava-collection library from google. It does come with most of the issues described here addressed and I think it is the best bet at the moment for that purpose. There are some other collection libraries to explore. Maybe one is actually better then guava.

Share Button

Indexing of Database Tables II (additional indices)

Additional indices („indexes“ in Oracle’s English) apart from the primary key are extremely important for the performance of databases.
There is this magic behind it, when a complicated query with multiple joins runs slowly and magically becomes faster when adding the right index and the right hint or does not become faster at all, even though the index being added could be so useful.. I leave this to the DBAs.

Some thoughts about indexing in general. Each index costs a little bit of storage and a little bit of performance when writing to the table, especially when inserting or when updating columns that are included in the index. This is more, if the amount of data per row in the index is more. And the helpfulness of the index is more, if it allows drilling down to a much smaller number of rows, not necessarily one. I would call this selectiveness of the index.

An interesting issue arises when combining multiple indices and there is interest in selecting by any one of them or their conjunction. This leads to Z-curve based indices, an issue worth an article by itself, maybe in the future.

Generally it is desirable to only include fields in the index that contribute to selecting the data. That means that adding this column to the where criteria of a select significantly reduces the number of records found. If that is not the case, the overhead of maintaining this field as part of the index and blowing up the index operations for both read and write access is probably not worth it. Sometimes a field with very little selectiveness is included in an index that is supporting a unique constraint. That may be ok, if the table is small, but for larger tables, rethinking the database design should be considered. I have seen this happening when several quite different kinds of records where stored in the same table. Splitting up to different tables would have resolved this. Other cases require different answers.

An interesting issue is also a situation, where many selects contain columns A and B in the WHERE-clause. By itself column A as well as column B are quite selective. But they are quite strongly correlated, so adding the other of the two only marginally contributes to the selectiveness of the index. In this case it should be considered, which of the three options, having A, B or both in the index is best. Try it out, discuss it with a good DBA. Yes, I have seen many people calling themselves DBA who were not really good in their area and I have learned what really good DBAs can do. Get one for a few days, when designing a serious database…

Share Button

PDF and PDF/A formats

The PDF format has experienced a success story on its way from being a quasi proprietary format that could only be dealt with using Adobe tools to a format that is specified and standardized and can be dealt with using open source tools and tools from different vendors. It has become accepted that PDF is primarily a print format and that for web content HTML is the better choice, which was not clear 15 years ago, when people coming from print layout who just considered themselves trivially capable of adding web to their portfolio just wanted to build whole web pages by just using PDF instead of HTML.

Now the format did change over time and there are always PDF files that use specific features that do not work in certain PDF viewers.

But there are requirements for maintaining documents over a long period of time. Just consider long term contracts that have a duration of 50-100 years. The associated documents usually need to be retained for that duration plus ten years. Alone the issue of storing data for such a long time and being able to read it physically is a challenge, but assuming that this issue is addressed and files can still be read in 110 years, the file format should be readable.

Now companies disappear. A lot of them in 100 years, probably even big ones like Adobe, Apple, Microsoft, Oracle and others. We do not know which companies will disappear, only that it is very likely that some companies that are big now will disappear. Proprietary software may make it to another vendor when shutting down the company, to pay the salaries of the former employees for some more days. But it might eventually disappear as well. Open source software has a better chance of being available in 100 years, but that cannot be absolutely guaranteed either, unless special attention is given to that software over such a long time. And if software is not maintained, it is highly unlikely that it will be able to run on the platforms that are common in 100 years.

So it is good to create a stable and simplified standard for long term archiving. Software for accessing that can be written from scratch based on that specification. And it is more likely to remain available, if continuous need for it can be seen.

The idea is a format called PDF/A, where A stands for „archive“, which is an option for storing PDF files over a very long period of time. Many cool features of PDF have been removed from PDF/A and make it more robust and easy to use. Important is also not to rely on additional data sources, for example for passwords of encrypted PDF files or for fonts. Encryption with password protection is a bad thing because it is quite likely that the password is gone in 100 years. Fonts need to be included, because finding them in 100 years might not be trivial. This usually means that proprietary fonts have to be avoided, unless the licensing allows inclusion of the fonts into the PDF file and unlimited reading. Including JavaScript, Video, Audio or Forms is also a bad idea. Video should be archived separately and it has the same issues as PDF for long term archiving.

Share Button

Indexing of Database Tables I (Primary Keys)

Any useful databases with non trivial amounts of data have some indexing mechanism in place that helps finding data based on some key values, because a full table scane or something like that is too expensive and too slow to do it often.

The most typical situation is having a key and one or more values behind that. This applies to relational and NoSQL databases.
Typical database access layers for programming languages like ActiveRecord for Ruby or JPA/Hibernate for Java favor the pattern of having one index column called „id“ that is numeric and gets successive values assigned when new entries are created.

This is good, because the id is always unique and it can be handled and stored efficiently, especially when being used for relationships between different tables. It is off course a good idea to think about the size of integers to use for this, so that the number range is not exhausted. Depending on the DB product this can produced automatically by the DB when inserting if the column is declared as identity column or autoincrement or something like that (MySQL, MS-SQLServer) or it can be created by using a sequence (Oracle, PostgreSQL). It could be considered to use just one global sequence instead of one per table, so the id is unique across tables, which might have some advantages, but it kind of unusual.

Another common approach is to use UUIDs instead of such numeric ids. They have the advantage of being kind of globally unique. They have 128 bits and there are two approaches for obtaining these 128 bit numbers. Some bits are used to express version and type of UUID, but most of them can be used for differentiating from other UUIDs. They can be generated using random numbers or using some identity of the current host combined with time stamp, counter and possibly thread and process numbers and filling some digits with random numbers… When having around a billion (German „Milliarde“, 10^9) entries, the probability of having a collision with a single additional randomly generated UUID is less than 10^{-27}. So the risk of being struck by a lightning is higher than the risk of getting a collision by creating the same random UUID twice, if the random number generator is really good. It is quite possible to build big systems based on that and run them for a long time without problems by duplicate UUIDs. If that is good enough or not is a philosophical question, but I prefer the other way and finding a way to generate a UUID that is unique across hosts in the current and expected future system environment. Quite honestly, I do not see the benefit of using UUIDs as ID fields and primary keys in databases, because the numeric IDs can be handled more efficiently and mechanisms of generating them uniquely are reasonably easy to implement.

It is often forgotten that data actually might contain internal unique keys that can be used. Often a autoincrement numeric ID is used as a primary key and these semantic keys are declared as unique constraint or at least with a non unique index to allow searching. This has the advantage that changing these values is possible by accessing only one table and without breaking referential consistency. But for large tables I do recommend to consider using these natural keys instead of the ID as primary key for the sake of efficiency. Serious databases can also deal with combined keys consisting of several columns that are unique only in their combination.

An obvious stupidity that I am only mentioning because I have actually seen it, is the concatenation of all the fields forming the logical primary key to some column called „HANDLE“ and using that as a primary key. A 40 year old junior developer with the authority of a senior lead architect could impose such a thing on the database of a project that I have worked in in the past and he could degrade the whole software by this, because it just wasted half of the database space and a lot of performance for a product that should have performed really well to be useful to its customers.

I would also resist the temptation to use something like oid in PostgreSQL or the ROW_ID of Oracle as key. These values can change when moving data to other table spaces or restoring a backup or even upgrading to another version of the DBMS product. They might be good for temporary use, probably within a transaction or within a set of closely related transactions occurring within a few minutes or hours.

Additional indexes are an interesting issue, which I might deal with in another article in the future.

Share Button

Microsoft laying off 18000 employees after announced loss


It looks like Stephen Elop’s und Steve Ballmer’s idea to take over the mobile phone branch of Nokia in order to bring MS-Windows as mobile operating system on the success road has failed and will be stopped now.
For the mobile phone branch that is now part of Microsoft, Microsoft-CEO Satya Nadella announced, that billions are deprecated and up to 18000 employees will be layed off. This will be more than half of the remaining former Nokia-employees, but also other parts of Microsoft will experience major lay offs. As an irony these layoffs will also include Stephen Elop.
On the other hand it looks like Nokia will get into mobile phone development again. The deal with Microsoft forbids that for a certain time, but that will end in 2016. Some expect that Jolla might be bought by Nokia in the future. Nokia will become a niche supplier, not more. The market leading position they had before Elop will not be regained and it is also unlikely that they become one of several major suppliers, which might have been a reasonable outcome without Elop. Their own production won’t come back to Finland, but will be given to the well known Asian manufacturing companies.
Such fusions and takeovers of departments always have their challenges. Company cultures tend to be incompatible and markets are used to a brand and will not easily switch to the new supplier. Taking over of Pentax by Ricoh has been relatively successful, as it seems, but in that case they have carefully worked on mainting the brand in an area where consumers are kind of sticky to their traditional brand. Microsoft did not go that road, but they deliberately changed a lot and thus lost customers who might have been loyal to the Nokia-brand. The strategy in these two takeovers was completely different.
Probably Steve Ballmer and Stephen Elop will be remembered as incapable managers for a long time, for first destroying Nokias core business and then losing billions on the takeover that was obviously the original intention behind the whole game.


Share Button

Microsoft entlässt bis zu 18000 Mitarbeiter nach Milliardenverlust


Wie es aussieht, ist die Idee von Stephen Elop und Steve Ballmer, die Mobilfunksparte von Nokia auszuschlachten und damit für MS-Windows als Mobiltelefonbetriebssystem einen Fuß in die Tür zu bekommen, nicht aufgegangen und wird jetzt gestoppt.
Für die durch Microsoft von Nokia übernommene Mobiltelefonsparte kündigte Microsoft-CEO Satya Nadella an, dass Milliarden abgeschrieben werden und dass bis zu 18000 Mitarbeiter entlassen werden sollen. Dies wird über die Hälfte der zu Microsoft gekommenen ehemaligen Nokia-Mitarbeiter betreffen und auch in anderen Teilen von Microsoft findet ein größerer Personalabbau statt. Auch Stephen Elop gehört übrigens zu den entlassenen Mitarbeitern.
Andererseits sieht es danach aus, dass Nokia wieder in die Mobiltelefon-Entwicklung einsteigen wird, zu dem Zeitpunkt, zu dem der mit Microsoft geschlossene Vertrag das erlaubt. Viele vermuten, dass irgendwann Jolla von Nokia übernommen werden könnte, wo viele ehemalige Nokia-Mitarbeiter gelandet sind und wo man für eine Nischenmarkt aussichtsreiche Ideen aufgegriffen und weiterentwickelt hat, die bei Nokia unter Stephen Elop abgebrochen wurden. Nokia wird sich als Nischenanbieter wahrscheinlich wieder einen kleinen Marktanteil sichern können, aber die Stellung als Marktführer, wie sie vor Elop bestand, oder auch nur als einer von vielen größeren Anbietern, wie sie vielleicht ohne Elop realistischerweise herausgekommen wäre, wird nicht wieder erreicht werden. Auch die eigene Produktion wird zugunsten einer Auslagerung an die üblichen ostasiatischen Elektronik-Fertiger kein Thema werden.
Problematisch bei solchen Fusionen oder Übernahmen von Abteilungen, ist dass sie oft nicht gut funktionieren, weil einerseits die Firmenkulturen verschieden sind und es so zu Reibungsverlusten kommt, die man sich in einem engen Markt nicht unbedingt leisten kann und andererseits ein Anbieter, der im Markt etabliert ist, nicht so einfach durch die übernehmende Firma ersetzt werden kann. Dafür muss man die Kunden erst einmal gewinnen und entsprechende Glaubwürdigkeit aufbauen.
Es hat ja solche Fälle gegeben, z.B. bei Kameraherstellern. Die Minolta Kamerasektion wurde von Sony übernommen und auch relativ bald aggressiv als Sony vermarktet. Sony hatte keine Glaubwürdigkeit für Kameras, außer natürlich für Video-Kameras, aber sie haben es sich mit interessanten Produkten und einer speziellen Nische erarbeitet. Für Spiegelreflexkameras hat Sony nicht den Namen wie Minolta, aber für kleine, leichte Digitalkameras, die zwischen Mobiltelefon und Spiegelreflexkamera positioniert sind, haben sie eine gute Stellung.
Pentax wurde von Ricoh übernommen, aber die Marke wird konsequent erhalten und gepflegt und noch als solche wahrgenommen. Auch da ist man nicht mehr Marktführer wie in den besten Zeiten, sondern ein stabiler Nischenanbieter, sofern man in diesem Markt so etwas überhaupt sagen kann.
Offensichtlich sind hier durch Missmanagement erst einmal viele Milliarden an Wert auf Seite von Nokia versenkt worden (und viele Mitarbeiter entlassen worden), dann ist aber die ganze Rechnung nicht aufgegangen und letztlich sind auf Microsofts Seite auch viele Milliarden versenkt worden, so viele, dass es auch in einer relativ großen Firma sichtbar und schmerzhaft ist. Steve Ballmer und Stephen Elop haben sich sicherlich ihren Ehrenplatz in der Liste der „Nieten in Nadelstreifen“ verdient.


Share Button