Evolution of WebSockets Rails App
The Problem
The original problem was that we wanted a real-time ‘dashboard’ to keep track of system stats on a wall in the office without having to load the system page all the time. There was a previous effort to try polling the server for updates, but there’s always a problem with polling:
- the client is polling too much when nothing is happening on the server and…
- the client is adding too much load through update requests when the server is under load
This solution was to use a persistent connection between the dashboard and the server: WebSockets!
HTTP vs WebSockets
HTTP is a connection which creates a connection from the client to the server. The server never can never initiate a connection to the client. Websockets are designed as a persistent connection between the client and server. So, not only can a client make a request to the server any time (like HTTP), the server can also send data to the client any time. Having a persistent connection between a server and client solves the real-time dashboard problem very well. The server just sends the data as the system is updated. No client polling is required. The server sends messages at the right time. The client is always updated with the most current system stats. win-win!
Solutions available
When choosing which WebSockets implementation to use, there are a few:
WebSocket-Rails variant was chosen mainly for:
- it’s a solution developed specifically for rails
- the IRC support was fantastic. nothing better than getting real-time support
I got a lot of great hand holding and some code review. Other solutions seemed like they would work, but I liked WebSocket-Rails integration better. Playing around with WebSocket-Rails, I was able to get demo code working very quickly. It’s a simple echo app, but there are lots of logging messages so one can see how different parts of the system works (there’s a lot of moving parts when adding WebSockets).
Some things I discovered in implementing the solution:
- Heroku REALLY does not like open connections! Even if there’s traffic over a connection, like a ping, it will cut off the connection and the client will have to actively reconnect. The documentation says all connections with no traffic will be cut after 55 seconds… In practice, connections were being cut off all the time with ping traffic.
- Creating a WebSocket-Rails client wrapper with self-executing anonymous functions is really worth it down the road.
- Real-Time means: real-time! Less than 30ms transfers. I can’t believe the dashboard is up to date faster than the server’s main status page can be reloaded. It’s fantastic.
With magic, there’s always a price…
The problem came when we wanted to preload_app! with workers so Puma can restart faster. Our app restart times were on the order of minutes (our app is pretty big) but preloading the app would stop WebSocket-Rails from working. an error came up:
! WARNING: Detected 1 Thread(s) started in app boot:
! #<Thread:0x0000000475eed8 sleep> - /home/.../.rvm/gems/ruby-2.1.5/gems/redis-3.2.1/lib/redis/connection/ruby.rb:52:in `select'
When I asked the websocket-rails maintainer about this preload problem, he said it was a known issue and well, didn’t have a good idea to solve it. I looked at the source a lot but I definitely felt out of my element to fix it. There was only one solution:
Back to basics
It is a bit hard to go back to the drawing board when there’s something that works 90%. So, what do you do? I went back to basics and thought about what did we really need. WebSocket-rails is fantastic and provides a lot of features, but we just needed a persistent connection. (Some may say for our use case that WebSockets are overkill, especially since there are server-sent events, but WebSockets code samples & tutorials are more prevalent, but I digress.)
So, going back to basics, I found this article on websockets and ruby on Heroku:
Using WebSockets on Heroku with Ruby
It wasn’t Rails and it was my first attempt at working with a sinatra and rack app. it was short enough to grok, but seeing how it would fit in rails was trickier. The article author gave a few hints on how to make the sinatra app into a rails app by loading the app as a Rails middleware. I learned a lot about rack from the sample. I took that sample and put it into rails. did a bit of hacking around and got this:
Heroku WebSocket Ruby article ported into Rails code
It is essentially the same code as the original article. Very nifty. The author was right, mostly. I had to hack a few things to get it working.
Now the problem came when I tried to integrate into our main code base. I shoehorned it all in, and got the app working… until I turned on puma’s preload_app! switch with workers, and saw this again:
! WARNING: Detected 1 Thread(s) started in app boot:
! #<Thread:0x0000000475eed8 sleep> - /home/.../.rvm/gems/ruby-2.1.5/gems/redis-3.2.1/lib/redis/connection/ruby.rb:52:in `select'
Augh! That’s exactly what I was doing to avoid this. Now I spent time working to get a variable out from the rack app… it drove me nuts. Class variables, cattr_accessors, etc. no matter what, I couldn’t get access to @clients from rails. I just wanted to get the list of open WebSocket connections and manage it in the app level. That’s when I discovered there’s a barrier between rack apps. The program was running another instance of rack apps and it would just filter items down. Going from one rack app (Rails) into another was impossible, well, getting that specific instance of it.
Aha! moment!
I talked with my tech lead on this issue and he realized the New Relic gem did something very similar: parse through rack calls and pass certain ones to the rails app. The exact same situation I found myself in before was because of my approach. I was trying to get at the rack app instance variables from inside another rack app. The simpler solution would be to have one rack app call another, notably: have the WebSocket middleaware rack app call a model in rails.
Once this ‘aha’ moment was discovered, rack and rails made a lot of sense. Rails is just another rack app, well a very special one. When I look at where ‘rails’ boots, it’s actually being booted by rack, the request.env in every Rails controller is exactly the same as Rack::Request object.
Now, the WebSocket connection aspect is clear. WebSocket calls come in, passes through all the rack apps until the one written and is processed through.
This is a bit of a war story with code, but I hope others can learn from my experience (and code samples.)
Lessons learned:
- All Rails apps start with rack.
- Rack is the lowest common denominator for ruby web apps (sinatra, rails, etc.)
- sometimes it’s easier to go one way than another (i.e. from a rack app into rails)
- rolling your own websocket system with faye is easier than I thought
- puma preloading is important.
the final code we used is here: a simple rails system with websocket that has puma preloading.