Zukunft für IT-Freiberufler

English

Einige von uns arbeiten in der Informatik.
Und dort gibt es doch einige Modelle. Man kann bei der Firma, der man die Arbeit erledigt, direkt angestellt sein. Man kann als Freiberufler, also entweder mit einem speziellen Freiberufler-Status oder mit einer eigenen Ein-Personen-Firma für seinen Kunden arbeiten. Oder man kann als Angestellter einer größeren Firma für deren Kunde arbeiten. Vielleicht gibt es noch ein paar Hybridmodelle, auf die ich hier aber nicht eingehen möchte.

Ich habe alle drei Modelle kennengelernt und bevorzuge es, als Freiberufler mit meiner eigenen Firma zu arbeiten. Ich denke, dass die Menschen verschieden sind und so verschiedene Präferenzen zwischen diesen Modellen haben. Und für ein IT-Projekt ist es gut, Mitarbeiter mit verschiedenen Stärken und Erfahrungen zu haben und diese bekommt man am besten, wenn man mindestens Freiberufler und eigene festangestellte Mitarbeiter kombiniert. Diese interessieren sich in der Regel am meisten für den Erfolg des Projekts, während bei externen Mitarbeitern, die bei einer großen Firma angestellt sind, tendenziell noch eine eigene, oft nicht mit den Projektinteressen kongruente Eigendynamik des Arbeitgebers ins Spiel kommen kann, aber nicht muss. Ihr kennt sicher positive und negative Beispiele aus eigener Erfahrung: Z.B. werden in einer frühen Projektphase externe Mitarbeiter gebracht, die sehr viel können, vielleicht aber auch gute Verkäufer sind oder sich irgendwo in der Struktur des Auftraggebers etablieren. Wenn dann der große Auftrag unter Dach und Fach ist, werden überwiegend Berufsanfänger gebracht. Oder generell Personen, die eine enorme Loyalität mit ihrem Arbeitgeber mitbringen, weil das Modell, bei dem man dauernd unterwegs zu Kunden ist, bei dem dem Kunden ein hoher Stundensatz verrechnet wird und bei dem man selbst gemessen an diesem Stundensatz nur ein sehr mittelmäßiges Gehalt bekommt, nur für eine längere Zeit funktioniert, wenn diese Mitarbeiter eine sehr große Loyalität zu ihrem Arbeitgeber oder eine Scheu vor dem Risiko haben, selbst als Freiberufler oder Selbständige zu arbeiten. Genau das kann auch einmal gut sein, um ein Team mit verschiedenen Leuten zu haben, die sich gegenseitig ergänzen, wenn es das Team vielfältiger macht. Je nach Lieferant kann man bei diesem Modell auch Leute bekommen, deren technisches und fachliches Wissen sehr gut ist. Auch das habe ich gesehen und man sollte sich als Auftraggeber genau anschauen, mit wem man zusammenarbeitet, damit sowohl die Firma als auch die Leute, die tatsächlich im Projekt mitarbeiten, gut sind. Mit kleineren Firmen macht man oft bessere Erfahrungen als mit internationalen Großkonzernen, aber es gibt gute und schlechte Beispiele in allen Kombinationen. Um aber ein gutes Mischungsverhältnis von verschiedenen Charakteren zu haben, wie es dem Projekt dienlich ist, ist es gut, bei den „externen“ Mitarbeitern mindestens teilweise auch auf Freiberufler zu setzen, die selbständig arbeiten und nicht bloß Kapazitätsplanung und Checklisten für Qualifikationen anzuschauen.

Eine gute Mischung aus internen und externen Mitarbeitern, gerne auch mit einer großen Mehrheit von „internen“, ist auch für ein Projekt vorteilhaft, weil es einerseits die Flexibilität gibt, die Teamgröße zu variieren, ohne Entlassungen vornehmen zu müssen. Es sollte auch für die Loyalität der internen Mitarbeiter förderlich sein, wenn Entlassungen eher die Ausnahme als die Regel sind. So wie interne Mitarbeiter die Strukturen, Anforderungen, Kunden, IT-Systeme und die verwendeten Werkzeuge über die Jahre sehr gut kennengelernt haben, ist es auch gut, dass „externe“ relative viele Firmen von innen gesehen haben und vielleicht Dinge sehen, die einem nicht mehr auffallen, wenn man sie seit Jahren gewohnt ist, die aber doch Verbesserungspotential bieten. So kommen auch neue Ideen ins Team.

Nun sehe ich, wie die Entwicklung in der Schweiz läuft und höre davon, wie sie in Deutschland läuft.

In der Schweiz wird es auch immer schwieriger. Es gab einige Schikanen, die man als „Kollateralschaden“, als „Kleinigkeiten“ oder auch „als gezielte Verhinderung der Konkurrenz durch Kleinfirmen seitens im Parlament gute verankerter Großfirmen“ interpretieren kann. Ich lasse das einmal offen.

Man muss in der Schweiz die Mehrwertsteuerfrage und die Frage der Sozialversicherungen regeln. Das sind Henne-Ei-Probleme. Um als Firma Mehrwertsteuer abrechnen zu können, muss man Kunden haben. Den Kunden muss man eine Rechnung schreiben und darauf muss entweder eine Mehrwertsteuer mit Mehrwertsteuernummer ausgewiesen sein, die dem Kunden nicht wehtut, wenn er selbst eine Firma ist, weil er sie weiterverrechnen kann, oder man weist sie nicht aus und hat dann das Problem, wenn man nachträglich mehrwertsteuerpflichtig wird. Das lässt sich lösen, wenn man der entsprechenden Stelle schreibt, dass man Kunden oder auch einen großen Kunden hat und gerne die Mehrwertsteuer bezahlen möchte, denn letztlich wollen die ja gerne das Geld einsammeln. Bei der Rentenversicherung und den anderen Sozialversicherungen ist es schwieriger. Die bekommen das Geld sowieso. Wenn man für die selbständig ist, dann zahlt man Arbeitgeber- und Arbeitnehmeranteile. Wenn nicht, zahlt der Kunde die Arbeitgeberanteile. Für die Abrechnung mit einem Kunden ist es schlicht unmöglich, eine Rechnung zu stellen und den Hinweis zu geben, dass man noch nicht weiß, ob irgendeine andere Stelle später noch Arbeitgeberanteile von Sozialversicherung einsammeln wird. Die Kunden gibt es dann schlicht und einfach nicht. Als Bäcker oder Maler hat man in der Regel schnell so viele Kunden, dass einem die Selbständigkeit abgenommen wird und keine Sozialversicherung wird beim Bäcker herausfinden, wer dort alles Brot gekauft hat, um den Kunden jeweils ein paar Franken Arbeitgeberanteile abzuknöpfen. Als Maler sollte man im ersten Jahr einfach ein paar kleine Kunden haben und nicht einen riesigen Millionenauftrag wie das Anstreichen eines Hochhauses, dann ist das auch gut. Als IT-Freiberufler hat man aber eher Projekte, wie das Anstreichen eines Hochhauses, wo viele Leute, interne Mitarbeiter des Kunden und man selbst und vielleicht auch andere „externe“ mehrere Monate oder gar Jahre an einem Projekt arbeiten. Es gibt Leute, die es geschafft haben, mit einer Einzelfirma als Freiberufler bei der Sozialversicherung als Selbständige anerkannt zu werden, aber das ist die Ausnahme. In der Praxis hat man zwei Wege, um diese Situation aufzulösen. Entweder lässt man sich bei einer Vermittlerfirma oder bei einer anerkannten Firma eines Kollegen temporär zu Konditionen anstellen, die komplementär zu dem Auftrag sind und umgeht das Problem auf diesem Weg. Leider bestehen einige Auftraggeber sogar auf diesem Modell. Das andere Modell, das letztlich interessanter ist, ist es eine eigene GmbH oder AG zu gründen und bei dieser angestellt zu sein. Dann ist auch alles sauber geregelt. Anders als in Deutschland kann man in der Schweiz kaum der Rentenversicherung entgehen, sie ist aber auch nicht so exzessiv ausgelegt wie in Deutschland, sondern zahlt eher eine Minimalrente, was die Höhe der Beiträge weniger schmerzhaft ausfallen lässt. Die mit dem Einkommen korrelierte Rente bekommt man in der Schweiz durch die sogenannte zweite Säule, auch Pensionskasse genannt, wo man ein Guthaben anspart, aus dem dieser zweite Teil der Rente gespeist wird.

Nun muss eine GmbH in der Schweiz ein Eigenkapital von mindestens 20’000 CHF haben, aber man musste nur die Hälfte einbringen. Die andere Hälfte musste man als Gründer nur garantieren, das heißt im Falle einer Insolvenz bis zu 10’000 CHF aus dem Privatvermögen aufbringen, um Schuldner zu bedienen. Später kam eine Änderung, dass man die 20’000 CHF von Anfang an komplett liefern musste, aber auch alle, die mit 10’000 gegründet hatten, die anderen 10’000 nachliefern mussten. Das klingt nicht so schlimm, letztlich war aber der bürokratische Aufwand recht hoch und erforderte ca. 1-2 Tage Arbeit und Gebühren, die mehr als ein Drittel der 10’000 CHF auffraßen. Letztlich ließ sich das mit Geld und Zeit bewerfen und war gelöst. Ebenso eine andere Änderung, die sogar nachvollziehbar war. Alle GmbHs und AGs brauchen eine Buchhaltung, die von jemandem mit entsprechendem Knowhow gemacht werden muss. Das ist normal und in allen Ländern so, die eine seriöse Wirtschaftsordnung haben. Große Firmen müssen diese Buchhaltung einer Revision durch eine weiter Buchhaltungsfirma unterziehen, bei kleineren ist das nicht nötig. Auch das ist sinnvoll. Früher war es so, dass man bei GmbHs grundsätzlich annahm, dass sie „klein“ sind und bei AGs, dass sie „groß“ sind. Das wurde dann korrigiert und Größe wurde nach rationaleren Kriterien gemessen. Nun musste man als GmbH aber aus der Revisionspflicht, die man erstmal aufs Auge gedrückt bekam, aussteigen, was wiederum ein größerer bürokratischer und finanzieller Akt war.

Die nächste Hürde war die Personalverleihbewilligung. Dafür muss man einen Antrag stellen, der etwa 46 Seiten lang war, und 50’000 CHF als Sicherheit hinterlegen, die man erst ein Jahr nach Verzicht auf die Personalverleihbewilligung zurückerhalten kann. Größere Firmen müssen die gleichen 50’000 CHF hinterlegen wie kleinere, nur wird sie bei größeren Firmen durch eine Garantie ersetzt, weil man diesen ohne weiteres glaubt, dass sie 50’000 CHF jederzeit aus dem Hut zaubern können. Auch diese Hürde lies sich mit Geld und Zeit bewerfen und überwinden. Und sie wurde nachträglich sogar wieder etwas abgeschwächt, allerdings in der Form, dass große Kunden trotzdem auf der Personalverleihbewilligung bestehen und dass manche Kantone sie nicht mehr gewähren, da sie ja nicht mehr nötig sei.

Jetzt bleibt zu hoffen, dass nicht eines Tages eine neue Schikane auftaucht, die sich schlicht und einfach nicht mehr realistisch überwinden lässt. Das faktische Berufsverbot für freiberufliche Informatiker. Ich möchte gerne viele Projekte sehen und die Flexibilität des Freiberuflers haben. Und ich kann mir nicht wirklich vorstellen, bei einer großen Firma angestellt und als „externer“ für Kunden zu arbeiten und mit dem Interessenkonflikt zu leben, loyal zu meinem Arbeitgeber zu sein und dem Interesse des Kunden und des Projekts zu dienen.

In dem Fall werde ich mir ein anderes Land in Europa suchen, wo ich weiterhin in der von mir bevorzugten Form arbeiten kann. In der Hinsicht bin ich flexibel und ich werde dann auch schnell die Sprache lernen, wenn ich sie noch nicht kann.

Share Button

Future for IT-Contractors

Deutsch

Some of us are working in IT.
And there you can finde some working models. You can work directly as employee of the company whose work you are doing. You can work as contractor (freelancer), with one of several slightly different setups, depending on the country. Or you can work as an employee of a larger company that lets you work for its customers. I am just briefly mentioning that some hybrid models exist.

I have worked all three models and I prefer the second one, working as contractor via my own Ltd-company. I think that people are different and have different preferences. And for staffing IT-projects it is good to have team members with different strengths and experiences that form a great team and not just resources that add up to the estimated number of person days during the project duration. It is good to combine permanent employees of the company doing the project and contractors. They are usually interested in the success of the project, while external team members employed in larger organizations often, but not always, bring in some own dynamic about interests of that organization that are not necessarily so congruent with the interests of the project. I do know positive and negative examples of this. For example have I seen that they bring in highly qualified people in an early phase, negotiate high rates and later replace them with juniors. I even heard of a case where such a company had to provide a couple of hundred people and brought a team consisting partly of office workers who had never done software development and who just edited program files in a way they were told by their more experienced team members by email. They did learn how to use the editor. Often they also bring people with good marketing skills to do the further acquisition during time paid by the customer. Generally working for this model requires an unusually high level of loyalty to the employer, so that working in another team, seeing the company getting relatively high rates, getting relatively low salaries (depending on the company, off course) will be continued for a longer period of time. It can be a reluctance to take risks or it can be loyalty or conservativeness or some combination. This is not a bad thing by itself, but building a team of people who are all too similar in this aspect could be suboptimal. Some organizations do send technically highly skilled people who are there employees. I think it is good to see who is actually joining the team and to make sure that the company sending them and the actual team members are good and trustworthy. Usually smaller companies tend to be better than international corporations for this, but good and bad examples exist in all combinations. To have a good mix in the team it is useful to have at least some contractors working for themselves among the external team members. Going through checklists of qualifications and capacity planning against some estimations is not enough.

A good mixture of internal and external team members, which can very well be with a majority of internal employees, is a great chance. It helps being flexible about the team size without having to do layoffs, which are neither common nor popular in Europe. For the loyalty of the permanent employees it is good if layoffs are a rare exception. Long term employees usually know the structures, the requirements, the customers, the IT systems and the tools very well because they have been around for several years. External team members have seen relatively many teams and companies and projects and have seen things that might not ring a bell for long term team members who have gotten used to certain things. So they can bring in new ideas for improvement and new knowledge.

I do observe how things in Switzerland are evolving and I am hearing some things about Germany. The problem is always that in other areas a contractor status is supposedly used to give employees suboptimal conditions and the government tries to stop this. In IT the contractor status is a good thing and works well and there are enough opportunities to become a permanent employee for those who prefer that. But if rules are formulated in a wrong way they make it very hard to be working as contractor.

In Switzerland it is getting more complicated. Some changes can be seen as „collateral damage“ or even „minor annoyance“ or as „effort by larger companies with good political connections to eliminate competition by smaller companies“. I leave that open for interpretation.

The VAT (sales tax) and the social insurance (retirement, unemployment etc.) needs to be addressed. This can be a chicken-or-egg-problem. To be able to pay VAT and to have a VAT number it is necessary to have some customers that justify the effort to provide a VAT number. Sending invoices with VAT usually does not hurt customers who are companies themselves, because they can pass it to their customers. Doing invoices without VAT is ok too, but becoming obliged to pay VAT later creates a problem because no customer wants to pay VAT on invoices that have already been settled. And nobody likes invoices with a text that says that the VAT-issue is still open and the amount to be paid might change later… Such customers simply do not exist. This is quite easy to overcome by just estimating the total amount of the invoices and writing a letter to the VAT organization prior to sending the first invoice. If the money in the start phase is not too scarce, the first invoice might even be delayed by a few weeks for this. In the end of the day the VAT-organization loves money and opportunities to get more of it.

The social security is a harder one. In Germany almost all contractors leave the social security system which is not very attractive and not very fair. In Switzerland at least the retirement system is slightly better and harder to leave, so the issue is more about finding a way to be part of this system and to work legally. Being an employee, the company and the employee pay half of the social security, respectively. Having ones own company implies paying both parts. If this company is not accepted by the social security organization, the customer is considered to be the employer and has to pay that half. This is a bad basis for sending invoices and if the status is not clear it is again almost impossible to have customers. Nobody likes working with a contractor if there is a possibility that the social security organization with come and collect some employer-half for some payments in an unpredictable future. This kind of customer simply has not been invented yet. Being a baker or a painter is no problem. Enough customers exist anyway, unless the business is a total failure. Nobody will find out who has bought bread and collect employer-parts for social security from the customer basis of the baker. If the painter does some smaller projects and not the really big million dollar contract for painting a skyscraper in the first year, things should be fine too.

Being an IT contractor the typical projects are more like painting a skyscraper. Many people are typically working on IT projects for many years and the contract might be extended for a year or two or in some rare cases even longer than that. So showing multiple customers in the first year can be hard, even though some smaller side projects, like trainings for other customers for a week or so, can help. Some people have managed to get this company status from the social security organization without incorporating their company. But in Switzerland that is an exception. In practice two ways are really working. Either you have to incorporate your company and get the social security issue resolved automatically by that or you have to become temporary employee of some intermediate company, usually the one who helped to acquire the contract. Conditions can be complementary to the contract and transparent.

There are basically two common ways to build a corporation in Switzerland: GmbH and AG. The GmbH requires a capital of 20’000 CHF (just think of around 18’000 EUR or 25’000 USD, but rates may change drastically). It was only necessary to bring have of that. The other half was only guaranteed by the founder, who had to provide another 10’000 from his money to pay debts of the company in case of bankruptcy. Later this was changed and the other 10’000 had to be added, which was reasonable by itself. But the bureaucratic effort was around one or two days and the fees ate up more than a third of the money. Was that necessary? Another change, that was in principal reasonable, was to change the size rule for more complex book keeping. Large AGs need another organization to check their book keeping while small GmbHs need only a professional book keeping. Small AGs where treated like big AGs and big GmbHs like small GmbHs. This was changed and the criteria became some metric about the size instead of the AG vs. GmbH-status, which makes sense. But it required some effort to opt out of the „big company pattern“, which was again expensive and time consuming.

The next step, a few years later, was the necessity to get a „Personalverleihbewilligung“. The paper work was around 46 pages and 50’000 CHF needed to be deposited as guarantee. Big companies have to deposite the exact same amount of money. It is for paying the salaries of the employees in case of failure. But this could be solved with time and money. Now it is hard to get the „Personalverleihbewilligung“ at all in some cantons, but in Switzerland companies play the game of moving to the most favorable canton.

There is always some risk that another problem is invented for small companies that cannot reasonably be dealt with. The factual prohibition for IT consultants working as contractors to do their job. I want to see many projects. I want to have the flexibility. I am willing to take the risk. But I do not want to become employee of a large organization and deal with the conflict of interest when loyalties with my employer and my projects conflict. I want to help the projects succeed.

In that case I will look for another country in Europe that allows me to work in the way I prefer. I am flexible with this and I will learn the language, if I do not already know it.

Share Button

Scala Days 2015

I have visited Scala Days 2015 in Amsterdam.
As usual, I took the night train to get there and back.

Creative Scala

Prior to the conference we did a training session for Scala beginners together with underscore. Some volunteers, including myself, joined the effort and so we were actually more teachers than students. It was fun.

Talks I listend to:

Monday 2015-06-08

* Monday-Keynote: Scala – where it came from, where it’s going / Martin Odersky

Tuesday 2015-06-09

* Tuesday-Keynote: Life Beyond the Illusion of Present / Jonas Bonér
* GraphX: Graph analytics for insights about developer communities / Paco Nathan @pacoid
* Meerkat parsers: a general parser combinator library for real programming languages / Ali Afroozeh @afruze & Anastasia Izmaylova @IAnastassija
* Fixing Reactive Code at 100 Miles per Hour: Five Techniques to Improve How You Debug Scala and Akka / Tal Weiss @weisstal
* Options in Futures, how to unsuck them / Erik Bakker @eamelink
* State of the Meta, Summer 2015 / Eugene Burmako @xeno_by
* Function-Passing Style, A New Model for Asynchronous and Distributed Programming / Philipp Haller @philippkhaller & Heather Miller @heathercmiller
* Essential Scala: Six Core Principles for Learning Scala / Noel Welsh @noelwelsh
* Project Gålbma: Actors vs. Types / Roland Kuhn @rolandkuhn

Wednesday 2015-06-10

* Wednesday-keynote: The Future of AI in Scala, and on the JVM / Adam Gibson
* Why Spark Is the Next Top (Compute) Model / Dean Wampler @deanwampler
* The Reactive Streams Implementation Landscape / Mathias Doenitz @sirthias
* Reactive Slick for Database Programming / Stefan Zeiger @StefanZeiger
* So how do I do a 2-phase-commit with Akka then? / Lutz Huehnken @lutzhuehnken
* Functional Data Validation (or How to Think Functionally) / Dave Gurnell @davegurnell
* Don’t Block Yourself / Flavio Brasil
* Closing Panel and Thank You

More details will come…

Share Button

Update of WordPress

Deutsch

The update of WordPress and its Plugins did not really work.

A small change helped a lot:

In .htaccess put the following line to the beginning of the file:

AddHandler php56-cgi .php

The all updates seemed to work fine.

Maybe this helps others observing the same problem.

Share Button

Aktualisierung von WordPress

English

Die Aktualisierung von WordPress und Plugins hat hier nicht so gut funktioniert.

Es hat eine kleine Änderung geholfen, jetzt ist alles gut:

In .htaccess am Anfang diese Zeile

AddHandler php56-cgi .php

einfügen…
Danach funktionierten alle Updates gut.

Vielleicht hilft das auch anderen, die damit Probleme hatten….

Share Button

Find the next entry in a sequence

In Facebook, Xing, Google+, Vk.com, Linkedin and other of these social media networks we are often encountered with a trivial question like this:

1->2
2->8
3->18
4->32
5->50
6->72
7->?

There are some easy patterns. Either it is some polynomial formula or some trick with the digits.
But the point is, that any such sequence can easily be fullfilled by a polynomial formula. That means we can put any value for 7 and make it work. Or any answer is correct. So what would probably be the real question is the most simple function to full-fill the given constraints. Simplicity can be measured in some way… If the solution is unique is unclear, but let us just look at the polynomial solution.

A function is needed that takes as parameter a list of key-value-pairs (or a hash map) and that yields a function such that the function of any of the key is the associated value.

Assuming a polynomial function in one variable we can make use of the chinese remainder theorem, which can be applied to univariate polynomials over a field F as well as to integral numbers. For a polynomial p(X) we have

    \[p(x) \equiv p(X) \mod X-x\]

where X is the polynomial variable and x\in F is a concrete value.

We are looking for a polynomial p(X) such that for given values x_0,\ldots x_{n-1}, y_0,\ldots,y_{n-1} \in F we have

    \[\bigwedge_{i=0}^{n-1} p(x_i) = y_i\]

or in another way

    \[\bigwedge_{i=0}^{n-1} p(X) \equiv y_i \mod X-x_i\]

which is exactly the Chinese remainder theorem.
Let

    \[I=\{0,\ldots,n-1\}\]

and

    \[\bigwedge_{j=0}^{n-1} I_j = I \setminus \{j\}\]

We can see that for all i \in I the polynomials

    \[e_i = \prod_{j \in I_j} \frac{X-x_j}{x_i-x_j}\]

have the properties

    \[e_i(x_i)=1\]

    \[\bigwedge_{j \in I_i} e_i(x_j)=0\]

or

    \[\bigwedge_{i \in I}\bigwedge_{j \in J} e_i(x_j)=\delta_{i,j}\]

where \delta_{i,j} is the Kronecker symbol, which is 0 if the two indices differ and 1 if they are equal.
Or as congruence:

    \[\bigwedge_{i \in I}\bigwedge_{j \in J} e_i(X)\equiv \delta_{i,j} \mod X-x_j\]

Then we can just combine this and use

    \[p(X) =\sum_{i \in I} y_i e_i(X)\]

This can easily be written as a Ruby function

def fun_calc(pairs)
  n = pairs.size
  result = lambda do |x|
    y = 0
    n.times do |i|
      p_i = pairs[i]
      x_i = p_i[0].to_r
      y_i = p_i[1].to_r
      z = y_i
      n.times do |j|
        if (j != i)
          p_j = pairs[j]
          x_j = p_j[0]
          z *= (x - x_j) / (x_i - x_j)
        end
      end
      y += z
    end
    y
  end
  result
end

This takes a list of pairs as a parameter and returns the polynomial function als lambda.
It can be used like this:

lop = [[0, 0], [1, 1], [2, 4], [3, 9], [4, 16], [5, 25], [6, 36], [7, 64]]

f = fun_calc(lop)

20.times do |x|
  y = f.call(x)
  puts sprintf("%6d -> %6d", x, y)
end

Put this together into a ruby program and add some parsing for the list of pairs or change the program each time you use it and all these „difficult“ questions „that 99.9% fail to solve“ are not just easy, but actually soluble automatically.

This is interesting for more useful applications. I assume that there will always be situations where a function is needed that meets certain exact values a certain inputs and is an interpolation or extrapolation of this.

Please observe that there are other interesting and useful ways to approach this:

  • Use a „best“ approximation from a set of functions, for example polynomials with a given maximum degree
  • use cubic splines, which are cubic polynomials within each section between two neighboring input values such that at the input values the two adjacent functions have the same value (y_i, off course), the same first derivative and the same second derivative.

For highway and railroad construction other curves are used, because the splines are making an assumption on what is the x-axis and what is the y-axis, which does not make sense for transport facilities. They are using a curve called Clothoid.

Use Java, C, Perl, Scala, F# or the programming language of your choice to do this. You only need Closures, which are available in Java 8, F#, Scala, Perl, Ruby and any decent Lisp dialect. In Java 7 they can be done with an additional interface as anonymous inner classes. And for C it has been described in this blog how to do closures.

Share Button

Thread local variables

The idea of threads is to share almost all information between the threads of the same process. This implies more effort to avoid creating inconsistencies by simultaneous access to the same data. But sometimes it is necessary to have data private to one thread. This is usually easily accomplished by just using local variables and method or function parameters that are simply not naturally available to the other threads, unless they are referenced by some global structures or by structures shared with other threads. But there are some cases when it is useful to have thread local variables.

The general idea is to call a function or method several times and to retain intermediate results somewhere. In object oriented programming attributes may be used for this, in procedural programming global variables or a structure that is passed around. In functional programming this is just forbidden and a workaround is always possible but this may be painful, complicated, inefficient or impossible due to the usage of existing functionality written by others.

Now the global variable fails miserably, one of the reasons, it should not be used. And its object-oriented blessed brother, like a static attribute in Java or C++, as well. Multiple threads may run simultanous sessions with these functionalities. So thread local variables are the better approach.

They can easily be built manually. You just need to use a hash map, make sure its access is thread safe by using RWLocks in C or ConcurrentHashMap or some synchronized HashMap in Java. Then use the thread ID as key to the hash map and store some structure in it where all the bad stuff is stored. Or use multiple hash maps.

But both languages provide better facilities for this purpose. In newer Java versions ThreadLocal should be used. This needs an initializer. Then it provides a get method that accesses the thread local copy of the variable and creates it, if it is not present. This approach is strongly recommended when dealing the DateFormat and NumberFormat, which are not thread safe in Java. Using synchronized on the format or creating it on demand each time works as well, but the thread local approach seems to be the most strait-forward and most efficient way. But these special examples should lose their importance. Using Joda-Time or the new Java-8 functionality for Date and Time should make SimpleDateFormat, DateFormat, Calendar and java.util.Date obsolete. And NumberFormat is still useful, but in many cases replaceable by String.format() and the C-like functionality.

In C a function pthread_key_create can be used to create a thread local veriable, pthread_setspecific() and pthread_getspecific() allow to store and retrieve a thread local veriable and hence thread local data. Please read the man pages instead of memorizing the exact API.

I might provide or link example programs for these concepts in some future posting.

Share Button

O’Reilly schließt Aktivitäten in Deutschland

This article is of minor relevance to English speaking readers, because it concerns the German O’Reilly books, while the English O’Reilly books continue to be available. So I will write in German.

Der O’Reilly-Verlag hatte jahrelang eine deutsche Niederlassung in Köln, wo deutschsprachige Versionen der beliebten Informatik-Fachbücher herausgegeben wurden. Dies waren oft Übersetzungen, aber in einigen Fällen auch exklusiv in deutscher Sprache erscheinende Titel oder doch Titel, die später vom Deutschen ins Englische übersetzt wurden. Der d-Punkt-Verlag wird die Aktivitäten übernehmen und auch unter dem Name O’Reilly weiterführen.

Mehr dazu unter
Informatik Aktuell.

Share Button

How to multiply long numbers

Assuming we have a multiplication of n-bit-numbers already available.
Other than typical compiled languages, which assume that the multiplication result fits into the same size as the two factors, we should take a closer look.
Unsigned n-bit-numbers as factors x and y fullfill

    \[0 \le x \le 2^n-1\]

    \[0 \le y \le 2^n-1\]

So we have for the product z=x*y

    \[0 \le z \le (2^n-1)^2 \le 2^{2n}-1\]

So we should be prepared for a 2n-bit long result. Assembly language provides for this. In Java we are kind of running out of luck, because we have the 64-bit-long as the largest possible value, so we cannot make use of the 64-bit capabilities of our CPU by getting a 128-bit-result. Even worse we do not have unsigned primitive types. For addition and subtraction it is possible to cheat by assuming that the longs are unsigned, because the addition for signed and unsigned numbers is almost the same, but for multiplication this is slightly more complex.

In C we do have better chances, but we need to go compiler and OS-specific to some extent. gcc on 64-bit platforms has a type __uint128_t and we can hope that something like this

__uint128_t mul(unsigned long long x, unsigned long long y) {
__uint128_t xx = (__uint128_t) x;
__uint128_t yy = (__uint128_t) y;
__uint128_t z = xx*yy;
return z;
}

is optimized by the compiler to one assembly language instruction that multiplies two unsigned 64-bit-integers into one unsigned 128-bit integer.

Multiplication of longer numbers is achieved by just assuming

    \[ x= \sum_{j=0}^{m-1}2^{nj}x_j, y= \sum_{j=0}^{m-1}2^{nj}y_j\]

with

    \[ \bigwedge_{j=0}^{m-1} 0\le x_j, y_j <2^n.\]

Now the product is

    \[ z = \sum_{j=0}^{m-1}\sum_{k=0}^{m-1}2^{n(j+k)}x_j y_k \]

where each x_j y_k summand needs to be split into a lower and an upper part such that

    \[x_j y_k = 2^n h(x_j y_k) + l(x_j y_k).\]

All these half products need to be added with propagating the carry bits to the next higher level.
This can be sorted out and done with a lot of additions with carry-bit, finally giving the desired result.

For reasonably short numbers this is a good approach, if it is done well, but it can be replaced by better algorithms for larger numbers.

The most obvious next step is to move to the Karatsuba algorithm.
Assuming we have two factors x=x_l + 2^m x_h and y = y_l + 2^m y_h the naïve approach of having to calculate the four products
x_h y_h, x_h y_l, x_l y_h, x_l y_l can be replaced by calculation of only three products x_h y_h, (x_l + x_h)(y_l+ y_h), x_l y_l, because what is actually needed is the sum x_h y_l+x_l y_h which can be obtained by

    \[(x_l + x_h)(y_l+ y_h)-x_l y_l-x_h y_h=x_h y_l+x_l y_h.\]

That can be iterated by subdividing the numbers subsequently into halves until a size is reached that can be handled well enough by the naïve approach.
An issue that needs to be observed is that x_l + x_h and y_l+ y_h need one bit more than their summands.

For even larger numbers the so called Toom-Cook algorithm, an extension of the idea of the Karatsuba-algorithm for more than two summands, can be useful and for numbers larger than that the Schönhage-Strassen multiplication proves to be faster. It is implemented in some libraries for long integer arithmetic, like GMP. Since around 2007 this can be improved by using the Fürer algorithm which is assymptotically faster, but I do not know of any useful implementations and for practical purposes the improvement does not seem to be very relevant.

Share Button

Testing Java- and C-programs with Ruby and Perl

It is very important to write good unit tests for software that is non-trivial and that is relied on by other pieces of software.
Often the logic of the software can easily be covered by the native testing facilities of the programming language, like JUnit for Java or, much less well known but available, CUnit for C. When a lot of framework code is involved or third party libraries are used heavily, there is almost no other way for certain tests, because setting up the environment cannot easily be achieved elsewhere.

But we also encounter cases where writing good unit tests in the same language as the library itself becomes a pain. We procrastinate the issue of writing them and end up with way too little or no unit tests at all. A lot of software deals with doing some calculations or transformations of numbers and strings, usually a lot of numbers and a lot of strings. Now Strings are the strength of Perl and are not really implemented very well or at least not very powerful or easy to use in most other languages. Specifically the String facilities of Perl are much superior to those of Java and C. Off course our software needs to perform well, it needs to integrate into the environment and follow the global corporate software standards, so Java or C or some other programming language is the choice that should not be challenged here for the productive software. But some tests of the functionality can more easily be achieved by iterating over some input data and creating output of the input data combined with the results. This can be perl code already or something really easy to parse. Perl is really the tool that can parse almost anything, but we do not really want to be distracted by unnecessary work but get our job done. So something like generating Perl code or CSV or YAML or JSON, but please not XML if not really needed, should do. Then we can pipe the output to perl or to a perl script and this will tell us, if everything is ok. When we know our platform, it can even be done that the Java- or C-Unit-test stores the output to a file or pipe and calls the Perl script on it and fails or succeeds depending on its output.

When it comes to numeric types, Ruby is very strong. It has unlimited size integers by default, which can be casted to n-bit-integers using constructions like
xx=x&(1<<n)-1,
it has Rational, LongDecimal (as an external gem) and Complex and is easily extendible.

Usually we can expect that corporate constraints on which tools and programming languages may be used are less restrictive when it comes to unit tests. Integrating this on a continuous integration platform is a job that needs to be addressed but it is worth the effort, if a lot of tests become easier with this approach. And doing tests in another language makes tests more credible.

Off course the general idea is applicable for other combinations. Look into Scala, F#, Clojure, JavaScript, Python and some others as well, if they seem to be more helpful than Ruby or Perl for your unit testing automation. But this does indeed raise the question if a world where corporate policies allowed Scala and Closure instead of Java, F# instead of C# and Elixir instead of Erlang and PL/I instead of Cobol would be better.

Share Button