Continuous Integration for Twilio (part 3)

In this post we continue to discuss CI for Twillo

The story so far In the previous post, we explained how to start your first step (Hop!) towards the test automation. The actual changes are summarised as follows: Configure spec_helper.rb to have #integration tag and only allow HTTP access from the tag.


The story so far

In the previous post, we explained how to start your first step (Hop!) towards the test automation. The actual changes are summarised as follows:

  • Configure spec_helper.rb to have #integration tag and only allow HTTP access from the tag.
  • Use selenium web driver to run tests in real browser.
  • Assert frontend behavior both via DOM id selector and directly evaluating Twilio JS function.
  • Setup ngrok to expose public callback URL on your local machine.

What the pain points are

Thanks to these changes, our #integration features launch browser and do real end to end testing with Twilio.

The downside is that you still have to be part of the test suite to run certain tasks such as the following:

  1. Click “Allow” to use microphone.
  2. Pick up phones every time a new Task is assigned.
  3. Change the task router callback url every time ngrok reallocates a new URL (which happens infrequently).



Our second step (Step!!) was focused on eliminating these nuisances: (1) use-fake-ui-for-media-stream and (2) use-fake-device-for-media-stream

The first one, use-fake-ui-for-media-stream, is relatively easy to get rid of. Let’s go back to our spec_helper.rb and change our driver setting slightly.

Capybara.register_driver :selenium_chrome do |app|  
  switches = %w(use-fake-ui-for-media-stream use-fake-device-for-media-stream), browser: :chrome, switches: switches)

These extra settings will fake connecting to browser microphone — you no longer receive ‘allow this app to access microphone’ popups (see here for a more detailed explanation). As an added bonus, this option enables us to run the test on a machine which does not have sound card and microphone. This is one step towards moving these tests into an external CI environment.

Automatic answer by TwiML

While (1) only happens once per test suite, (2) happened multiple times across one test run, and the number kept growing as our integration scenarios increased.

Scenario: Worker hangs up  
Scenario: User hangs up  
Scenario: Call timeouts  
Scenario: Call fails due to busy  
Scenario: Call fails due to immediate hangup  

The problem is not just taking many calls, but you also have to pay attention to test runner logs to be aware which scenarios you are in hence you have to act differently (eg: you hang up real phone for User hang up scenario while you wait tests to hangup for Worker hang up scenario).

To make matters worse, we have a QA engineer who worked abroad and it was too costly for him to pick up his test call. Hence every time he ran a test, it was calling one of our London engineers’ numbers. It’s not difficult to imagine how disruptive this could become.

This is where Twilio’s TwiML comes handy.

<?xml version="1.0" encoding="UTF-8"?>  
  <Pause length="10" />
  <Say>We have normality yo. Anything you still can't cope with is therefore your own problem</Say>

As you know, Twilio provides its own markup language for Interactive Voice Response (IVR). The above example will wait 10 seconds, say something, then hangup.

You can create multiple TwiML endpoints for each scenario, serve the XML somewhere (or use third party TwiML paste bin called TwiMLbin, then register each url to Twilio incoming numbers.


When we tried this approach, we were initially worried about the phone lines becoming busy if two tests called the same number at exactly the same time. This was not the case. It was great news because we have 21 subaccounts and did not want to set multiple incoming numbers per every subaccount. You can set these once on your master account and that’s it!

Dynamically detect ngrok url.

Solving (1) and (2) problems allowed us to run automated test almost “Hands free”, except for the times when ngrok URL changes (eg: it will change the URL if you restart ngrok).

The good news is that ngrok has API endpoints (default at localhost:4040) and you can programmatically obtain up to date ngrok url there.

So we created the following ruby module that starts up ngrok (either in port 3000 or 3001), enquire its url via API endpoint (4040 or 4041), then injects the url as a TaskRouter endpoint. This is invoked only when we start our application using foreman.

require 'timeout'  
require 'uri'

module NgrokRunner  

  def start_for(env)
    unless already_running?(env)
      pid = fork do
        `ngrok http --config #{config_file_path(env)} #{ngrok_client_api_port(env)}`
      at_exit { stop_for(env) }
      write_pid_to_file(env, pid)

  def stop_for(env)
    `pkill -TERM -P #{current_pid(env)}` if already_running?(env)

  def url(env)
    ngrok_api_uri = URI("http://localhost:#{ngrok_tunnelling_port(env)}/api/tunnels")

  # methods below are private methods

  def ngrok_client_api_port(env)
    case env
    when 'development'
    when 'test_integration'
      raise "#{env} is not supported by NgrokRunner"

  def ngrok_tunnelling_port(env)
    case env
    when 'development'
    when 'test_integration'
      raise "#{env} is not supported by NgrokRunner"

  def config_file_path(env)
    File.expand_path("../../../config/ngrok.#{env}.yml", __FILE__)

  def pid_file_path(env)
    File.expand_path("../../../tmp/pids/ngrok.#{env}", __FILE__)

  def current_pid(env) if File.exist?(pid_file_path(env))

  def already_running?(env)
    pid = current_pid(env)
    !(pid.nil? || Process.getpgid(pid).nil?)
  rescue Errno::ESRCH

  def write_pid_to_file(env, pid)
    FileUtils.mkdir_p(File.dirname(pid_file_path(env))), 'w') { |file| file.write(pid) }

  def ensure_finished_loading(env)
    Timeout.timeout(5) do
      loop do
        break unless url(env).nil?
        sleep 0.5

You can start ngrok per environment with NgrokRunner.start_for(@env). Once started, you can access to the ngrok url with NgrokRunner.url(@env)

You may wonder what is test_integration environment in the source code. That’s a special environment we inject only when you run test with #integration tag. This is how you configure the setting at your spec_helper.rb

RSpec.configure do |config|  
  config.before(:suite) do
    if config.inclusion_filter[:integration]
      ENV['RACK_ENV'] = 'test_integration'


Thanks to our script to dynamically detect ngrok URL, we unblocked the biggest hurdle of running callback tests on third party CI environment such as Semaphore CI where we have limited capability to customise their settings.

What else is left? Actually, not much. We initially thought that installing the Chrome executable into Semaphore would be challenging, but it turned out that it is already installed according to their website.

So the only things you still have left to install are ngrok and chrome driver. They can be installed using npm. We created a file called script/ci and configured Semaphore to run the file as part of the test.


#!/usr/bin/env bash

bundle install --deployment --path vendor/bundle --without development deployment

npm install --no-progress  
npm rebuild --no-progress

if [ "$BRANCH_NAME" == "master" ]; then  
  npm install chromedriver ngrok --no-progress

  bundle exec rake test:all
  bundle exec rake test:unit

/script/ci script installs drivers and runs full test if the branch is master.

We could invoke this test suite for each branch, but each call to Twilio actually costs money so we decided to run it only against master branch.


Within this final blog post, we talked about the following:

  • Change chrome driver to fake microphone settings.
  • Setup TwiML to pickup test phone calls.
  • Dynamically inject ngrok callback URL.
  • Run the integration test suits on Sempahore.

These test suites not only save time manually performing tests, but also allow us to refactor our internal code with confidence. In fact, we had one big internal refactoring where we changed the calling order (initially we were calling users before worker pick up the phone, which left some blank silence when users picked up phones) and it would have been scary to do so without end to end test.

As we initially said, we started small and kept iterating and there might be still room for improvement. If you have any opinions, comments, and suggestions, please feel free to share with us.