Continuous Integration for Twilio (part 3)

In this post we continue to discuss CI for Twillo

The story so far In the previous post, we explained how to start your first step (Hop!) towards the test automation. The actual changes are summarised as follows: Configure spec_helper.rb to have #integration tag and only allow HTTP access from the tag.

The story so far

In the previous post, we explained how to start your first step (Hop!) towards the test automation. The actual changes are summarised as follows:

• Configure spec_helper.rb to have #integration tag and only allow HTTP access from the tag.
• Use selenium web driver to run tests in real browser.
• Assert frontend behavior both via DOM id selector and directly evaluating Twilio JS function.
• Setup ngrok to expose public callback URL on your local machine.

What the pain points are

Thanks to these changes, our #integration features launch browser and do real end to end testing with Twilio.

The downside is that you still have to be part of the test suite to run certain tasks such as the following:

1. Click "Allow" to use microphone.
2. Pick up phones every time a new Task is assigned.
3. Change the task router callback url every time ngrok reallocates a new URL (which happens infrequently).

Step!!

Our second step (Step!!) was focused on eliminating these nuisances: (1) use-fake-ui-for-media-stream and (2) use-fake-device-for-media-stream

The first one, use-fake-ui-for-media-stream, is relatively easy to get rid of. Let's go back to our spec_helper.rb and change our driver setting slightly.

Capybara.register_driver :selenium_chrome do |app|
  switches = %w(use-fake-ui-for-media-stream use-fake-device-for-media-stream)
  Capybara::Selenium::Driver.new(app, browser: :chrome, switches: switches)
end

These extra settings will fake connecting to browser microphone — you no longer receive 'allow this app to access microphone' popups (see here for a more detailed explanation). As an added bonus, this option enables us to run the test on a machine which does not have sound card and microphone. This is one step towards moving these tests into an external CI environment.

Automatic answer by TwiML

While (1) only happens once per test suite, (2) happened multiple times across one test run, and the number kept growing as our integration scenarios increased.

Scenario: Worker hangs up
Scenario: User hangs up
Scenario: Call timeouts
Scenario: Call fails due to busy
Scenario: Call fails due to immediate hangup

The problem is not just taking many calls, but you also have to pay attention to test runner logs to be aware which scenarios you are in hence you have to act differently (eg: you hang up real phone for User hang up scenario while you wait tests to hangup for Worker hang up scenario).

To make matters worse, we have a QA engineer who worked abroad and it was too costly for him to pick up his test call. Hence every time he ran a test, it was calling one of our London engineers' numbers. It's not difficult to imagine how disruptive this could become.

This is where Twilio's TwiML comes handy.

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Pause length="10" />
  <Say>We have normality yo. Anything you still can't cope with is therefore your own problem</Say>
  <Hangup></Hangup>
</Response>

As you know, Twilio provides its own markup language for Interactive Voice Response (IVR). The above example will wait 10 seconds, say something, then hangup.

You can create multiple TwiML endpoints for each scenario, serve the XML somewhere (or use third party TwiML paste bin called TwiMLbin, then register each url to Twilio incoming numbers.

When we tried this approach, we were initially worried about the phone lines becoming busy if two tests called the same number at exactly the same time. This was not the case. It was great news because we have 21 subaccounts and did not want to set multiple incoming numbers per every subaccount. You can set these once on your master account and that's it!

Dynamically detect ngrok url.

Solving (1) and (2) problems allowed us to run automated test almost "Hands free", except for the times when ngrok URL changes (eg: it will change the URL if you restart ngrok).

The good news is that ngrok has API endpoints (default at localhost:4040) and you can programmatically obtain up to date ngrok url there.

So we created the following ruby module that starts up ngrok (either in port 3000 or 3001), enquire its url via API endpoint (4040 or 4041), then injects the url as a TaskRouter endpoint. This is invoked only when we start our application using foreman.

require 'timeout'
require 'uri'

module NgrokRunner
  module_function

  def start_for(env)
    unless already_running?(env)
      pid = fork do
        `ngrok http --config #{config_file_path(env)} #{ngrok_client_api_port(env)}`
      end
      at_exit { stop_for(env) }
      write_pid_to_file(env, pid)
    end
    ensure_finished_loading(env)
  end

  def stop_for(env)
    `pkill -TERM -P #{current_pid(env)}` if already_running?(env)
  end

  def url(env)
    ngrok_api_uri = URI("http://localhost:#{ngrok_tunnelling_port(env)}/api/tunnels")
    JSON.load(Net::HTTP.get(ngrok_api_uri))['tunnels'][0]['public_url']
  rescue
    nil
  end

  # methods below are private methods

  def ngrok_client_api_port(env)
    case env
    when 'development'
      3000
    when 'test_integration'
      3001
    else
      raise "#{env} is not supported by NgrokRunner"
    end
  end

  def ngrok_tunnelling_port(env)
    case env
    when 'development'
      4040
    when 'test_integration'
      4041
    else
      raise "#{env} is not supported by NgrokRunner"
    end
  end

  def config_file_path(env)
    File.expand_path("../../../config/ngrok.#{env}.yml", __FILE__)
  end

  def pid_file_path(env)
    File.expand_path("../../../tmp/pids/ngrok.#{env}", __FILE__)
  end

  def current_pid(env)
    File.read(pid_file_path(env)).to_i if File.exist?(pid_file_path(env))
  end

  def already_running?(env)
    pid = current_pid(env)
    !(pid.nil? || Process.getpgid(pid).nil?)
  rescue Errno::ESRCH
    false
  end

  def write_pid_to_file(env, pid)
    FileUtils.mkdir_p(File.dirname(pid_file_path(env)))
    File.open(pid_file_path(env), 'w') { |file| file.write(pid) }
  end

  def ensure_finished_loading(env)
    Timeout.timeout(5) do
      loop do
        break unless url(env).nil?
        sleep 0.5
      end
    end
  end
end

You can start ngrok per environment with NgrokRunner.start_for(@env). Once started, you can access to the ngrok url with NgrokRunner.url(@env)

You may wonder what is test_integration environment in the source code. That's a special environment we inject only when you run test with #integration tag. This is how you configure the setting at your spec_helper.rb

RSpec.configure do |config|
  config.before(:suite) do
    if config.inclusion_filter[:integration]
      ENV['RACK_ENV'] = 'test_integration'
    end
  end
end

Jump!!!

Thanks to our script to dynamically detect ngrok URL, we unblocked the biggest hurdle of running callback tests on third party CI environment such as Semaphore CI where we have limited capability to customise their settings.

What else is left? Actually, not much. We initially thought that installing the Chrome executable into Semaphore would be challenging, but it turned out that it is already installed according to their website.

So the only things you still have left to install are ngrok and chrome driver. They can be installed using npm. We created a file called script/ci and configured Semaphore to run the file as part of the test.

#!/usr/bin/env bash

bundle install --deployment --path vendor/bundle --without development deployment

npm install --no-progress npm rebuild --no-progress

if [ "$BRANCH_NAME" == "main" ]; then npm install chromedriver ngrok --no-progress PATH=$PATH:pwd/node_modules/ngrok/bin:pwd/node_modules/chromedriver/bin/

bundle exec rake test:all else bundle exec rake test:unit fi

/script/ci script installs drivers and runs full test if the branch is main.

We could invoke this test suite for each branch, but each call to Twilio actually costs money so we decided to run it only against main branch.

Summary

Within this final blog post, we talked about the following:

• Change chrome driver to fake microphone settings.
• Setup TwiML to pickup test phone calls.
• Dynamically inject ngrok callback URL.
• Run the integration test suits on Sempahore.

These test suites not only save time manually performing tests, but also allow us to refactor our internal code with confidence. In fact, we had one big internal refactoring where we changed the calling order (initially we were calling users before worker pick up the phone, which left some blank silence when users picked up phones) and it would have been scary to do so without end to end test.

As we initially said, we started small and kept iterating and there might be still room for improvement. If you have any opinions, comments, and suggestions, please feel free to share with us.