# Scala Exchange 2017

I have visited Scala Exchange („#ScalaX“) in London on 2017-12-14 and 2017-12-15. It was great, better than 2015 in my opinion. In 2016 I missed Scala Exchange in favor of Clojure Exchange.

This time there were really many talks about category theory and of course its application to Scala. Spark, Big Data and Slick were less heavily covered this time. Lightbend (former Typesafe), the company behind Scala, did show some presence, but less than in other years. But 800 attendees are a number by itself and some talks about category theory were really great.

While I have always had a hard time accepting why we need this „Über-Mathematics“ like category theory for such a finite task as programming, I start seeing its point and usefulness. While functors and categories provide a meta layer that is actually accessible in Scala there are actually quite rich theories that can even be useful when constrained to a less infinite universe. This helps understanding things in Java. I will leave details to another post. Or forget about it until we have the next Scala conference.

So the talks that I visited were:

• Keynote: The Maths Behind Types [Bartosz Milewski]
• Free Monad or Tagless Final? How Not to Commit to a Monad Too Early [Adam Warski]
• A Pragmatic Introduction to Category Theory [Daniela Sfregola]
• Keynote: Architectural patterns in Building Modular Domain Models [Debasish Ghosh]
• Automatic Parallelisation and Batching of Scala Code [James Belsey and Gjeta Gjyshinca]
• The Path to Generic Endpoints Using Shapeless [Maria-Livia Chiorean]
• Lightning talk – Optic Algebras: Beyond Immutable Data Structures [Jesus Lopez Gonzalez]
• Lightning Talk – Exploring Phantom Types: Compile-Time Checking of Resource Patterns [Joey Capper]
• Lightning Talk – Leave Jala Behind: Better Exception Handling in Just 15 Mins [Netta Doron]
• Keynote: The Magic Behind Spark [Holden Karau]
• A Practical Introduction to Reactive Streams with Monix [Jacek Kunicki]
• Building Scalable, Back Pressured Services with Akka [Christopher Batey]
• Deep Learning data pipeline with TensorFlow, Apache Beam and Scio [Vincent Van Steenbergen]
• Serialization Protocols in Scala: a Shootout [Christian Uhl]
• Don’t Call Me Frontend Framework! A Quick Ride on Akka.Js [Andrea Peruffo]
• Keynote: Composing Programs [Rúnar Bjarnason]

# Devoxx 2017

I was lucky to get a chance to visit Devoxx in Antwerp the sixth time in a row. As always there were interesting talks to listen to. Some issues that were visible across different talks:

Java 9 is now out and Java 8 will soon go into the first steps of deprecation. The step of moving to Java 9 is probably the hardest in the history of Java. There were features in the past that brought very significant changes, but they were usually kind of optional, so adoption could be avoided or delayed. Java 9 brings the module system and a new level of abstraction in that classes of modules can be made public to other modules selectively or globally. Otherwise they can be by themselves declared as public, but only be visible within the module. This actually applies to classes of the standard library, that were always declared as being private, but that could not be efficiently hidden away from external usage. Now they suddenly do not work any more and a lot of software has some difficulty and needs to be adjusted to avoid these internal classes. Beyond that a lot of talks were about Java 9, for example also covering somewhat convenient methods for writing constant collections in code. Future releases will follow a path that is somewhat similar to that of Perl 5. Releases will be created roughly every half year and will include whatever is ready for inclusion at that time. Some releases will be supported for a longer time than others.

Then there was of course security security security… Even more important than in the past.

And a lot more…

I listened to the following talks:
Wednesday, 2017-11-08

Thursday, 2017-11-09

Friday, 2017-11-10

Previous years:

Btw. I always came to Devoxx by bicycle, at least part of the way…

# ScalaUA

About a month ago I visted the conference ScalaUA in Kiew.

This was the schedule.

It was a great conference and I really enjoyed everything, including the food, which is quite unusual for an IT-conference.. 🙂

I listened to the following talks:
First day:

• Kappa Architecture, Juantomás García Molina
• 50 shades of Scala Compiler, Krzysztof Romanowski
• Functional programming techniques in real world microservices, András Papp
• Scala Refactoring: The Good the Bad and the Ugly, Matthias Langer
• ScalaMeta and the Future of Scala, Alexander Nemish
• ScalaMeta semantics API, Eugene Burmako

I gave these talks:

• Some thoughts about immutability, exemplified by sorting large amounts of data
• Lightning talK: Rounding

Day 2:

• Mastering Optics in Scala with Monocle, Shimi Bandiel
• Demystifying type-class derivation in Shapeless, Yurii Ostapchuk
• Reactive Programming in the Browser with Scala.js and Rx, Luka Jacobowitz
• Don’t call me frontend framework! A quick ride on Akka.Js, Andrea Peruffo
• Flawors of streaming, Ruslan Shevchenko
• Rewriting Engine for Process Algebra, Anatolii Kmetiuk

Find recording of all the talks here:
https://www.scalaua.com/speakers-speeches-at-scalaua2017/

# Slick

Almost every serious programming language has to deal with database access, if not out of love, then at least out of practical necessity.

The theoretical background of a functional programming language is somewhat hostile to this, because pure functional langauges tend to dislike state and a database has the exact purpose to preserve state for a long time. Treating the database as somewhat external and describing its access with monads will at least please theoretical purists to some extent.

In case of Scala things are not so strict, it is allowed to leave the purely functional path where adequate. But there are reasons to follow it and to leave it only in well defined, well known, well understood and well contained exceptions… So the database itself may be acceptable.

Another intellectual and practical challenge is the fact that modern Scala architectures like to be reactive, like to uncouple things, become asynchronous. This is possible with databases, but it is very unusual for database drivers to support this. And it is actually quite contrary to the philosophy of database transactions. It can be addressed with future database drivers, but I do not expect this to be easy to use properly in conjunction with transactions.

It is worthwhile to think about
immutability and databases.
For now we can assume that the database is there and can be used.

So the question is, why not integrate existing and established persistence frameworks like JPA, Hibernate, Eclipslink and other OR-mapping-based systems into Scala, with Scala language binding?

Actually I am quite happy they did not go this path. Hibernate is broken and buggy, but the generic concept behind it is broken in my opinion. I do not think that it is theoretically correct, especially if we should program with waterproof transactions and actually defeat them by JPA-caching without even being aware of it? But for practical purposes, I actually doubt that taming JPA for non trivial applications is really easier than good old JDBC.

Anyway, Scala took the approach of building something in the lines of JDBC, but making it nicer. It is called Slick, currently Slick 3. Some links:
* Slick
* Slick 3.1.1
* Documentation
* Github
* Assynchronous DB access
* Recording of the Slick Workshop at ScalaX

What Slick basically does is building queries using Scala language. It does pack the results into Scala structures. Important is, that the DB is not hidden. So we just do the same as with JDBC, but in a more advanced way.

# How to create ISO Date String

It is a more and more common task that we need to have a date or maybe date with time as String.

There are two reasonable ways to do this:
* We may want the date formatted in the users Locale, whatever that is.
* We want to use a generic date format, that is for a broader audience or for usage in data exchange formats, log files etc.

The first issue is interesting, because it is not always trivial to teach the software to get the right locale and to use it properly… The mechanisms are there and they are often used correctly, but more often this is just working fine for the locale that the software developers where asked to support.

So now the question is, how do we get the ISO-date of today in different environments.

date "+%F"

## TeX/LaTeX

\def\dayiso{\ifcase\day \or
01\or 02\or 03\or 04\or 05\or 06\or 07\or 08\or 09\or 10\or% 1..10
11\or 12\or 13\or 14\or 15\or 16\or 17\or 18\or 19\or 20\or% 11..20
21\or 22\or 23\or 24\or 25\or 26\or 27\or 28\or 29\or 30\or% 21..30
31\fi}
\def\monthiso{\ifcase\month \or
01\or 02\or 03\or 04\or 05\or 06\or 07\or 08\or 09\or 10\or 11\or 12\fi}
\def\dateiso{\def\today{\number\year-\monthiso-\dayiso}}
\def\todayiso{\number\year-\monthiso-\dayiso}

This can go into a file isodate.sty which can then be included by \include or \input Then using \todayiso in your TeX document will use the current date. To be more precise, it is the date when TeX or LaTeX is called to process the file. This is what I use for my paper letters.

## LaTeX

(From Fritz Zaucker, see his comment below):

\isodate % switch to ISO format
\today % print date according to current format

## Oracle

SELECT TO_CHAR(SYSDATE, 'YYYY-MM-DD') FROM DUAL;

On Oracle Docs this function is documented.
It can be chosen as a default using ALTER SESSION for the whole session. Or in SQL-developer it can be configured. Then it is ok to just call

SELECT SYSDATE FROM DUAL;

Btw. Oracle allows to add numbers to dates. These are days. Use fractions of a day to add hours or minutes.

## PostreSQL

(From Fritz Zaucker, see his comment):

select current_date;
—> 2016-01-08

select now();
—> 2016-01-08 14:37:55.701079+01

## Emacs

In Emacs I like to have the current Date immediately:

(defun insert-current-date ()
"inserts the current date"
(interactive)
(insert
(let ((x (current-time-string)))
(concat (substring x 20 24)
"-"
(cdr (assoc (substring x 4 7)
cmode-month-alist))
"-"
(let ((y (substring x 8 9)))
(if (string= y " ") "0" y))
(substring x 9 10)))))
(global-set-key [S-f5] 'insert-current-date)

Pressing Shift-F5 will put the current date into the cursor position, mostly as if it had been typed.

## Emacs (better Variant)

(From Thomas, see his comment below):

(defun insert-current-date ()
"Insert current date."
(interactive)
(insert (format-time-string "%Y-%m-%d")))

## Perl

In the Perl programming language we can use a command line call

perl -e 'use POSIX qw/strftime/;print strftime("%F", localtime()), "\n"'

or to use it in larger programms

use POSIX qw/strftime/;

# Slick: Bringing Scala’s Powerful Features to Your Database Access

## Rebecca Grenier

Slick is a library that generates and executes SQL queries. The conceptional weaknesses of JPA and Hibernate are avoided.
It has drivers for the five major DBs (PostgreSQL, mySQL/mariaDB, Oracle, MS-SQL-Server and DB2) and some minor DBs, but it is not free for the three commercial DBs.
Inner and outer joins are possible and can be written in a decent way.
With database dictionaries slick is now even able to generate code. Which I have done, btw. a lot using Perl scripts running on the DDL-SQL-script. But this is better, off course…

# Upcoming in Slick 2.2

## Jan Christopher Vogt

Monads haunt us everywhere, even here. Time to learn what they are. I will be giving a talk in the Ruby-on-Rails user group in Zürich, which will force me to learn it.
Sessions in conjunction with futures are the best guarantee for all kinds of nightmares, because the SQL is sometimes executed when the session is no longer valid or the transaction is already over. When dealing with transactional databases a lot of cool programming patterns become harder. Just think of the cool java guys who execute stuff by letting a EJB-method return an instance of an inner class with the DB-session implicitely included there and calling a method which via JPA indirectly and implicitely does DB-access long after the EJB-method is over. Have fun when debugging this stuff. But we know about it and address it here.
At least Slick is theoretically correct, other than JPA which I conject to be theoretically incorrect, apart from the shortcomings of the concrete implementations.
Several statements can be combined with andThen or a for-comprehension. Be careful, by default this uses separate sessions and transactions, with very funny effects. But threads and sessions are expensive and must not be withheld during non-running non-SQL-activities by default. Reactiveness is so important. Futures and thread pools look promising, but this fails miserably when we have blocking operations involved, that will occupy our threads for nothing.
We would like to have assynchronous SQL-access, which can be done on DB- and OS-level, but JDBC cannot. So we have to work around on top of JDBC. Apart from using a reasonably low number of additional threads this approach seems to be viable.
Statically type checked SQL becomes possible in the future.

# No more Regular Expressions

## Phil Wills

I love the regex of Perl. Really. So why do effort to give up something so cool, even in non-perl-languages?
It is not as bad as it sounds. We retain regular expressions as a concept, just do not call them like that (for marketing reasons I assume) and write them differently. Writing them as strings between // is very natural in Perl, but it breaks the flow of the language in Scala. A typical programmatical scala-like approach is more natural and more type safe. And more robust in case of errors. org.paraboiled2 can be used. Capture is unfortunately positional, unlike in newer Perl-regex, where captures can be named. But it hurts less here.

# Scala eXchange – Q&A Panel

## Jon Pretty, Kingsley Davies, Lars Hupel, Martin Odersky, and Miles Sabin

Interesting discussions…

# Why Scala is Taking Over the Big Data World

## Dean Wampler

‚“Hadoop“ is the „EJB“ of our time.‘
MapReduce is conceptionally already functional programming. So why use Java and not Scala?
Some keywords: „Scalding“, „Storm“, „Summing bird“, „Spark“.
Scala can be more performant than python, which is so popular in this area, but migration has to be done carefully.

# Case classes a la carte with shapeless, now!

## Miles Sabin

Example: tree structure with backlinks. Hard to do in strict immutabilty. Shapeless helps.

# Reactive Programming with Algebra

## André Van Delft

Algebra can be fun and useful for programming. Algebraic interpretations were introduced.
Page is subscript-lang.org.
Algebra of communicationg processes. It is very powerful and can even be applied to other targets, for example operation of railroad systems.
Every program that deals with inpout is in its way a parser. So ideas from yacc and bison apply to them.

# High Performance Linear Algebra in Scala

## Sam Halliday

Lineare Algebra has been addressed extremely well already, so the wheel should not be reinvented.
TL;D, Netflix and Breeze.
Example for usage of that stuff: Kalman Filter.
netlib has reference implementation in Fortran, a well defined interface and a reliable set of automatic tests. How do we take this into the scala world?
Fortran with C-Wrapper for JNI. (cblas)
compile Fortran to Java. really.
Alternate implementations of the same test suite in C.
High-Performance is not only about speed and memory alone, but about those under the given requirements concerning correctness, precision and stability.
Hardware is very interesting. The CPU-manufacturers are talking with the netlib team.
Can GPU be used? Off course, but some difficulties are involved, for example transfer of data.
FPGA? maybe soon? Or something like GPU, without graphics and operating on the normal RAM?
We will see such stuff working in the future.

# An invitation to functional programming

## Rúnar Bjarnason

Referential transparency, something like compatibility with the memoize pattern.
Pure functions…
Parallelization..
Comprehensiveness.. The all time promise, will it be kept this time?
Functional programming is not good for the trenches of real life project, but for getting out of the trenches. This should be our dream. Make love not war, get rid of trenches…

# Building a Secure Distributed Social Web using Scala & Scala-JS

## Henry Story

Spargl is like SQL for the „semantic web“.
Developed with support from Oracle.
We can have relativiity of of truth while retaining the absolute truth. The speech was quite philosophical, in a good way.
Graphs can be isomorphic, but have a different level of trust, depending on where the copy lies.
How is spam and abuse avoided? WebID?
We are not dealing with „Big Data“ but with massively parallel and distributed „small data“.

# TableDiff – a library for showing the difference between the data in 2 tables

## Sue Carter

What is the right semantics for a diff?
What do we want to see? How are numbers and strings compared when occurring in fields?
Leave timestamps that obviously differ but do not carry much information out.

# Evolving Identifiers and Total Maps

## Patrick Premont

Idea is to have a map where get always returns something non-null. Smart type concepts avoid the compilation of a get call that would not return something.
Very interesting idea, but I find it more useful as theoretical concept rather than for practical purposes. The overhead seems to be big.

# Summary

Overall it was a great conference.

# Visit to Scala Days in Berlin 2014

Deutsch

Around mid of June 2014 I have been visiting the Scala Days in Berlin. Like usual these events contain a lot of speeches, which were distributed in four tracks, apart from the key notes. The event location was a cinama, like the Devoxx in Antwerp, but this time one that has been transformed to something else many years ago, but good projectors were available. Major topics where issues about compiler construction which is a hard task, but looked at with the right functional perspective it can be derived from the simple task of writing an interpreter. This helps understanding language constructs in Scala, but the idea was applied to many other areas as well, for example for compiling and optimizing SQL queries and for analyzing source code.

Another major topic was „streams“, which can be useful for web services. Other than traditional web services which usually receive the whole request before starting to process it, concepts where discussed for dealing with large requests whose size is painful or impossible to keep at once in memory or who are even unlimited in size. These can also be applied to websockets. This demands processing data as soon as useful parts have arrived.

Another minor, but very interesting topic was development of Android apps with Scala. The commonly known approach is off course Scaloid, but an alternative, Macroid, was presented. It looked quite promising, because it allows to write nice Android apps with less code. A major worry is that scala apps consume too much memory due to their additional libraries. Because Scala uses its own libraries on top of the usual preinstalled Java libraries which are about 5 MB in size, this can easily anihilate the attractiveness of Scala for modern smart phone development, unless we assume rooted devices which have the Scala libs preinstalled. But that would seriously limit the scope of the app. This is not as bad as it sounds, because the build process contains one step in which unnecessary classes are removed, so that we only install what is really needed. When going as crazy as running Akka on the cell phone it becomes quite a challenge to configure this step because Akka uses a lot of reflection and so all these reflective entry points need to be configured, leaving a wide door open for bugs that occur at runtime when some class is not found.

API design was another interesting talk. Many ideas were quite similar to what I have heard in a API design traing for the Perl programming language by Damian Conway a couple of years ago, but off course there are many interesting Scala specific aspects to the topic. It is surprisingly hard to obtain binary compatibility of classes and this even forces to ugly compromises. So always recompiling everything looks tempting, but is not always reasonable. So ugly compromises remain part of our world, even when we are working with Scala.

# RISC und CISC

Vor 20 Jahren gab es einen starken Trend, Mikroprossoren mit RISC-Technologie (reduced instruction set computer), zu bauen. Jeder größere Hersteller wollte so etwas bauen, Sun mit Sparc, Silicon Graphics mit MIPS, HP mit PA-Risc, IBM schon sehr früh mit RS6000, was später, als man auf die Idee kam, das zu vermarkten, als PowerPC rebranded wurde, DEC mit Alpha u.s.w. Zu einer Zeit, als man zumindest in Gedanken noch für möglich hielt Assembler zu programmieren (auch wenn man es kaum noch tat), tat das noch richtig weh. Und Software war doch sehr CPU-abhängig, weil plattformunabhägige Sprache, die es damals selbstverständlich schon lange vor Java gab, einfach wegen des Overheads der Interpretation für viele Zwecke zu langsam waren. So behalf man sich mit C-Programmen mit wahren Orgien an ifdfefs und konnte die mit etwas Glück für die jeweilige Plattform aus CPU und einem UNIX-Derivat kompilieren. Der Wechsel der CPU-Architektur eines Herstellers war damals eine große Sache, z.B. bei Sun von Motorola 680×0 zu Sparc. Und die Assemblerprogrammierung der RISC-CPUs war ein Albtraum, an den sich auch erfahrene Assemblerprogrammierer kaum herangewagt haben. Zum Glück gab es damals schon sehr gut optimierende Compiler für C und Fortran und so konnte man das Thema einem ganz kleinen Personenkreis für die Entwicklung von kleinen Teilen des Betriebssytemkerns und kleinen hochperformanten Teilen großer Libraries überlassen.

Eigentlich sollte RISC ermöglichen, mit derselben Menge an Silizium mehr Rechenleistung zu erzielen, insgesamt also Geld und vielleicht sogar Strom zu sparen. Vor allem wollte man für die richtig coole Server-Applikation, die leider immer etwas zu ressourchenhungrig für reale Hardware war, endlich die richtige Maschine kaufen können. RISC war der richtige Weg, denn so hat man einen Optmierungsschritt beim Compiler, der alles optimal auf die Maschinensprache abbildet, die optimiert dafür ist, schnell zu laufen. Ich wüsste nicht, was daran falsch ist, wenn auch diese Theorie vorübergehend nicht zum Zuge kam. Intel konnte das Problem mit so viel Geld bewerfen, dass sie trotz der ungünstigeren CISC-Architektur immer noch schneller waren. Natürlich wurde dann intern RISC benutzt und das irgendwie transparent zur Laufzeit übersetzt, statt zur Compilezeit, wie es eigentlich besser wäre. Tatsache ist aber, dass die CISC-CPUs von Intel, AMD und Co. letztlich den RISC-CPUs überlegen waren und so haben sie sich weitgehend durchgesetzt.

Dabei hat sich die CPU-Abhängigkeit inzwischen stark abgemildert. Man braucht heute kaum noch Assembler. Die Plattformen haben sich zumindest auf OS-Ebene zwischen den Unix-Varianten angeglichen, so dass C-Programme leichter überall kompilierbar sind als vor 20 Jahren, mit cygwin sogar oft unter MS-Windows, wenn man darauf Wert legt. Applikationsentwicklung findet heute tatsächlich weitgehend mit Programmiersprachen wie Java, C#, Scala, F#, Ruby, Perl, Python u.ä. statt, die plattformunabhängig sind, mittels Mono sogar F# und C#. Und ein Wechsel der CPU-Architektur für eine Hardware-Hersteller ist heute keine große Sache mehr, wie man beim Wechsel eines Herstellers von PowerPC zur Intel-Architektur sehen konnte. Man kann sogar mit Linux dasselbe Betriebssystem auf einer unglaublichen Vielfalt von Hardware haben. Die große Mehrheit der schnellsten Supercomputer, die allermeisten neu vekrauften Smartphones und alle möglichen CPU-Architekturen laufen mit demselben Betriebssystemkern, kompiliert für die jeweilige Hardware. Sogar Microsoft scheint langsam fähig zu sein, verschiedene CPU-Architekturen gleichzeitig zu unterstützen, was lange Zeit außerhalb der Fähigkeiten dieser Firma zu liegen schien, wie man an NT für Alpha sehen konnte, was ohne große finanzielle Zuwendungen seitens DEC nicht aufrechterhalten werden konnte.

Aber nun, in einer Zeit, in der die CPU-Architektur eigentlich keine Rolle mehr spielen sollte, scheint alles auf Intel zu setzen, zum Glück nicht nur auf Intel, sondern auch auf einige konkurrierende Anbieter derselben Architektur wie z.B. AMD. Ein genauerer Blick zeigt aber, das RISC nicht tot ist, sonder klammheimlich als ARM-CPU seinen Siegeszug feiert. Mobiltelefone habe häufig ARM-CPUs, wobei das aus den oben genannten Gründen heute fast niemanden interessiert, weil die Apps ja darauf laufen. Tablet-Computer und Netbooks und Laptops sieht man auch vermehrt mit ARM-CPUs. Der Vorteil der RISC-Architektur manifestiert sich dort nicht in höherer Rechenleistung, sondern in niedrigerem Stromverbrauch.

Ist die Zeit reif und CISC wird in den nächsten zehn Jahren wirklich durch RISC verdrängt?

Oder bleibt RISC in der Nische der portablen stromsparendenen Geräte stark, während CISC auf Server und leistungsfähigen Arbeitsplatzrechnern dominierend bleibt? Wir werden es sehen. Ich denke, dass früher oder später der Vorteil der RISC-Architektur an Relevanz auf leistungsfähigen Servern und Arbeitsplatzrechnern gewinnen wird, weil die Möglichkeiten der Leistungssteigerung von CPUs durch mehr elektronischen Elementen pro Quadratmeter Chipfläche und pro Chip an physikalische Grenzen stoßen werden. Dann die bestmögliche Hardwarearchitektur mit guten Compilern zu kombinieren scheint ein vielversprechender Ansatz.

Die weniger technisch interessierten Nutzer wird diese Entwicklung aber kaum tangieren, denn wie Mobiltelefone mit Android werden Arbeitsplatzrechner mit welcher CPU-Architektur auch immer funktionieren und die Applikationen ausführen, die man dort installiert. Ob das nun plattformunabhängig ist, ob man bei kompilierten Programmen ein Binärformat hat, das mehrere CPUs unterstützt, indem einfach mehrere Varianten enthalten sind oder ob der Installer das richtige installiert, interessiert die meisten Benutzer nicht, solange es funktioniert.