How an open-source contributor made real-time messaging a much easier development project
Bas van Essen,
Monday, January 14, 2019
Sebastiaan [5:47 PM]
Alexandr, you started building Centrifugo 6 years ago.
Alexandr Emelin [5:48 PM]
Sebastiaan [5:48 PM]
It’s a real-time messaging server.
Why did you decide to build it back then?
Alexandr Emelin [5:57 PM]:
Alexandr Emelin [6:12 PM]
At that time I was very interested in asynchronous programming, I liked things event loops can give – this is still like magic for me, working with many connections and network operations concurrently – this is just great. It’s not straightforward in Python though – even after hours of work with Twisted and Tornado it seemed not very obvious how things work inside especially in terms of error propagation and execution context. So when I first tried Go I quickly realized that this is an ideal option to develop concurrent software fast. So I just decided to migrate – I think this was a good decision because Centrifuge got many cool things fo free – like multicore, binary, cross-compilation, static typing and of course performance. And it got a new name – Centrifugo.
Maybe from performance perspective Go is not the greatest choice but I think it just has the right balance between performance and developer friendliness. (edited)
Sebastiaan [6:15 PM]
Which Go repos have you used to be able to build Centrifugo as it is now?
Alexandr Emelin [6:15 PM]
Interesting fact… Even before I met Go I tend to return errors in Tornado as last return value:) This is actually not very convenient in Python world:)
I just want to show my respect to mr Gary Burd – he made two beautiful libraries Centrifugo uses inside – http://github.com/gorilla/websocket and github.com/gomodule/redigo – both packages are very stable with great API.
Also github.com/igm/sockjs-go to work with SockJS Websocket polyfill connections
Github.com/gogo/protobuf to speed up Protobuf as it generates code instead of using refrection
And github.com/spf13/viper for simple configuration (actually Centrifugo uses its fork with lots of dependencies removed)
Sebastiaan [6:23 PM]
Recently you introduced version 2. What has fundamentally changed if you look back to the latest version compared to say 4/5 years ago?
Alexandr Emelin [6:25 PM]
Yep, Centrifugo v2 was released a couple of months ago – here are full release notes btw: https://github.com/centrifugal/centrifugo/releases/tag/v2.0.0
Alexandr Emelin [6:32 PM]
I think the biggest advantage is that it now uses strict Protobuf schema for protocol, having that schema it works with both JSON and Protobuf formats. I refactored and cleaned up protocol a bit, removed some parts that were not widely used. This allowed to keep Centrifugo v2 simple and compact and specialized on one main task – deliver real-time messages to clients. And it’s now based on separate real-time library for Go language https://github.com/centrifugal/centrifuge – this was rather challenging to separate monolithic server to library and server parts, but I am pretty happy with result.
Alexandr Emelin [6:37 PM]
With that library one can build sth similar to Centrifugo but with custom business logic in Go, native authentication etc. Still don’t know whether or not it will be in demand as it’s pretty heavy in terms of features (Go community tends to avoid huge framework-like libraries) but I think in terms of design keeping Centrifugo core as library could be beneficial… Time will show:)
Sebastiaan [6:42 PM]
In your commits you can see you are busy with the 2.1.1 version (edited)
what will come next?
Alexandr Emelin [6:49 PM]
Currently I am working on client libraries – as Centrifugo has custom protocol and many protocol features we need custom client libs to connect – and this is the hardest part of maintaining project. Because clients must be written in languages I don’t know myself so trying to do this with community help.
During New Year holidays I experimented with Swift to build iOS client – hope with a help of my colleagues it will be released very soon.
Alexandr Emelin [7:02 PM]
Heh, I still have wow effect when I see messages appearing in two browser tabs at one moment:) Also I work in Avito (biggest Russian classified) on Avito messenger – recently we integrated part of Centrifugo into our architecture stack (sharded Redis engine) and it works very well, allowed us to save great amount of resources – so I am using it myself. But you are right – maintaining interest to open source project is hard. I think many have read a recent post from vim-go author Fatih Arslan (https://arslan.io/2018/10/09/taking-an-indefinite-sabbatical-from-my-projects/). Sometimes I feel exactly the same. But from time to time I receive positive feedback from users and it just motivates me to continue working on project.
Sebastiaan [7:09 PM]
Do you also contribute to other open source projects in the meantime?
Alexandr Emelin [7:18 PM]
Not much – I have several contributions into libraries used inside Centrifugo: Gorilla Websocket, SockJS-Go. Also I’ve created several small libraries – like https://github.com/FZambia/sentinel to work with Redis Sentinel on top of Redigo, https://github.com/FZambia/eagle to instrument your Go program with Prometheus library primitives and still export them into Graphite for example. Also I contributed a lot into Python course lectures available on Github – https://github.com/park-python/course – it’s a collection of Python Jupyter notebooks for Russian students. (edited)
Alexandr Emelin [7:35 PM]
Both apply to Go and many more not mentioned, as far as I know there is no open source user-facing real-time messaging library in Go that allows to scale connections around many machines out of the box (Centrifuge library I mentioned above now can with sharded Redis) and there are many challenges when working with connections like proper shutdown, implementing per-connection queue to not block program flow on massive broadcasts, recovering missed messages upon reconnect.
You also need to invalidate sessions as Websocket connections can live forever. And of course you will have to deal with corporate proxies blocking your users’ websocket traffic and introduce some sort of fallback.
Btw one unpleasant problem with Go is very slow compression speeds, so permessage-deflate is very expensive. There are libraries that improve standard lib compression performance – for example https://github.com/klauspost/compress – but still performance is not great comparing to solutions in C. Hope in future the situation will be better. (edited)
Here is a thing that amazed me one day: https://play.golang.org/p/vflz1xO7UBp
Sebastiaan [7:44 PM]
You say Centrifugo is popular in PHP and Python communities. With regards to being built in the first place in Python, that seems to be obvious. Why is PHP another community?
Alexandr Emelin [7:51 PM]
I think because there are less alternatives that other languages provide. And PHP is rather popular in Russia, I wrote several articles here so maybe many users came after reading them. Actually even at work we have 3 main languages programmers write code in: Python, PHP and… Go of course:)
Sebastiaan [8:03 PM]
As an end of our interview: what are in your eyes the coolest applications, you have heard of, in which the use of Centrifugo is included?
Alexandr Emelin [8:11 PM]
The coolest one I’ve heard about is an extremely beautiful dashboard in Badoo office – it shows people matches on the world map in real-time, it’s just looks great! Unfortunately I’ve lost video showing it in action. Also Centrifugo is widely used in Mail.Ru (one of the biggest IT companies in Russia) – for example in it’s internal intranet system for lots of stuff like real-time comments, counters, meeting dashboard, corporate games and events. Also https://youla.ru/ is a great example where Centrifugo used for messenger app. (edited)
Ah and my favorite sport site I visit every day sports.ru – also uses it:)
Alexandr Emelin [8:24 PM]
It’s a good questions on a hard topic :) I’ve made non scientific bench while developing v2 using adopted benchmark suite from Nats – here are some results https://github.com/centrifugal/centrifuge-go/tree/master/examples/benchmark – it shows for example that Protobuf gives a great speed up in general, I also compared Protobuf Websocket with GRPC bidirectional streaming. I can tell you more about that benchmark and conclusions (you can find them in my article though). With JSON protocol Centrifugo v2 should perform the same as v1 but I don’t have exact numbers.
Also published on Medium.
Categorised in: Develop smarter, not harder
This post was written by Bas van Essen