Hello folks! I have already told you that I am currently working on a set of interviews with some of the speakers you will find at Erlang Factory SF Bay Area 2013. Today I have the pleasure to interview Knut Nesheim. Knut is widely known for some applications he created while working at Wooga (e.g., Elli and Locker). In this interview we will ask him something more about his Erlang experience, his famous Erlang applications and the talk he will give at Erlang Factory SF Bay.
Rock ‘n Roll? No, lock and rule!
Paolo – Hi Knut! Thanks for making yourself avaiable for an interview. Please, describe yourself in few words.
Knut - Hey Paolo, thanks for having me! I’ve been developing software for the last five years, in Norway where I’m originally from, Sweden and now Germany. I’ve always had an urge to understand how things work and computers provide endless opportunities for tinkering, hacking and exploring.
Paolo – You started as with an educational background in music and then moved to computers right? How come did you switch?
Knut - After computers, music is my second big interest in life. Music obeys “laws”, many quite intricate. It’s interesting to take apart a piece of music and understand how the parts work together to form something that can create emotion, give energy and inspire in people without them maybe even noticing, like in film. I find music for film to be very interesting.
It became apparent pretty quickly that I was much better at computers than music. I still play guitar and I regularly visit the Berlin Philharmonie. I just bought an electric piano and I want to start taking lessons again. It’s a nice hobby.
Paolo – I do believe that your main language before Erlang was Python. In your opinion what is the most difficult concept of Erlang for a developer coming from Python?
Knut - I was using Python quite a lot, yes. I was doing smaller web applications with lots of business logic.
The hardest part about learning Erlang for me was immutable state. I was used to thinking of data being stored in places and having different parts of the program storing new values in that place. Understanding that after I modify something, there are now two versions of it took some time. It’s a bit sad that it’s mostly inside of our programs we can have the immutable state. I find Datomic, the database from Rich Hickey to be very interesting.
Paolo – You are the author of Elli, an Erlang application I like very much. Can you tell us somethink about it?
Knut - Good to hear someone likes it! Elli is a special purpose webserver you can run inside your Erlang application to let your code speak HTTP with your clients. It’s not general purpose in that it lacks features you would take for granted if you’re for example making a web application. It’s a mashup of ideas from different projects, both within the Erlang community and outside, specifically it takes a lot of inspiration from Rack.
It has been a fun project to work on and I have received 34 pull requests to date, which isn’t much compared to other bigger projects, but I think it’s great to see so users contributing their improvements. It’s used in production in Wooga. Upcoming games are using it in very interesting ways. The traffic is absolutely crazy, with each online user making a request every 1-5 seconds.
Paolo – Why should we take a look at Elli? I mean, in the Erlang community we have already many Web Servers; when should we prefer Elli to them?
Knut - It’s true that there’s a bunch of projects already there, overlapping in quite big areas resulting in duplicated efforts.
I wrote Elli for the needs of Wooga, which was difficult to meet with the existing projects. Now, that’s not because they are bad in any way, lack features or are buggy. They are quite successful at making developers more productive.
When the volumes of traffic grow, so does the amount of weird input and amount of unexpected interactions in the implementation of the webserver. If you have an error that statistically speaking happens every 10,000 requests, at Wooga that error happens more than once a second. We found that developer productivity wasn’t all that important for us, it’s more important to focus on performance, robustness and operations.
A webserver that did less, we thought, would be less likely to have features interact in unexpected ways and would be faster from less code (and fewer processes!). Using the Rack-style of request-response where the handler function returns the response (as opposed to sending it directly on the socket), allows us to create middlewares. In the “web world”, middlewares are a common way to plug in third-party functionality, like an authentication middleware inspecting and injecting headers, a middleware implementing cookies or a middleware collecting statistics and exposing it over HTTP by overriding a url. It doesn’t fit everything, you can’t for example postprocess a streamed response.
If you are able to judge the trade-offs made by the different projects and you find Elli fits your needs better, I would encourage you to use it. I’m happy to help and committed to maintaining the project. If you’re not sure which one to pick, go for Yaws.
Paolo – During your career you have been working in many companies as for example Wooga and Klarna. How does it feel to work in some of the biggest Erlang companies out there?
Knut – Pretty cool! I really enjoyed my time at Wooga. I joined January 2011 as a “gun for hire” to help out with a experimental Erlang project, started by Paolo Negri. Together we built the first backend using Erlang, paving the way for adoption within the company. They now have 14 developers using Erlang with almost all working on upcoming games.
Now I’m heading up a very interesting project at Unity, the makers of the 3d game engine. It’s too early to talk publicly about the projects we are building, but I’m very excited about this opportunity. The team is starting to form and by the end of the year, hopefully we have something really cool to show.
Paolo – During the Erlang Factory in San Francisco you will give the following talk “Locker: Consistent Distributed Locking”. Would you like to tell us something about it?
Knut - Locker is a distributed locking service, implemented in Erlang and uses the Erlang cluster for messaging. It’s a multi-master key-value in-memory database, where a key expires if the lease is not renewed periodically. In terms of CAP, it will sacrifice availability when a quorum can not be made
It’s a hack, in the sense that it’s not an implementation of a “proper” algorithm from a paper. For example, coordination of writes happens in two phases, where if a quorum can be made in the first phase, the second phase writes the value on all masters. Some people properly educated in distributed systems tell me it’s good stuff, others tell me it ignores too many hard problems to be useful, like group membership. I’m looking forward to hopefully getting more feedback.
Paolo – How come Wooga needed something as Locker? What are the main benefits one may experience using it?
Knut - Just like with Elli, eredis and statman, locker started out as a frustration with what was currently available to us. In the stateful application servers we were building, we needed a way of ensuring that each online user has only one process. This can be solved in many different ways and we found ourselves constantly thinking about it. Our first version was just a central point which serialized all updates, but we wanted it to be available in case of partitions or more common to us, node failure, software crash and operator error.
We thought about this challenge for more than a year before we decided to tackle it head on. A recurring question was “what if we could do the perfect solution to our problem, rather than trying to shoe-horn existing solutions with complex hacks for scaling?” closely followed by “how hard could it be?” Eventually, I decided to just try it. I’m very happy with the result.
Paolo – How long did it take to have a working version of Locker? I find interesting the fact that such a piece of software if available for free on GitHub, don’t you agree?
Knut - From I decided to try building it to having a working version took around three weeks, out of which one afternoon was spent coding. The rest of the time I spent thinking, drawing and reading. Writing tests, async replication, various optimizations, etc, maybe a month of total effort has been spent over the last year, with contributions from others at Wooga. It’s just 330 lines of code, but coming up with those exact lines was very hard.
From developers to CTO and CEO, Wooga is pushing to open-source what can be open-sourced to hopefully contribute something back to the community. It took some time to convince everybody that locker was something Wooga should give away for free, even to competitors. I was the one holding back, but was eventually convinced by Jesper Richter-Reichhelm, head of engineering. It’s always scary to put something out there for everybody to see.
Paolo – Who should follow your talk and why? What are the basic skills a wannabe attendant to your talk should have?
Knut - If you’re interested in distributed systems, it might be interesting to see how locker was built and why it was built that way. If you’re an expert, you could come and help me improve locker by providing feedback on why it’s good or why you think it’s total crap. If you’re interested in the cultural parts of a successful game studio and how we build amazing things in a very short time, you could get a glimpse inside the inner workings. Knowing Erlang isn’t really necessary. The challenging parts of understanding locker is not in the code.
