Writing asynchronous acceptance tests in Hound for Elixir and Phoenix can be difficult, especially if you are using a JavaScript framework like React, Vue.js, or Angular.
If you have ever used end-to-end testing in your web application’s test suite, you have undoubtedly come across the issue of “flapping” tests.
In many of the Ruby on Rails projects that I get asked to work on, I come across this problem frequently. So much so, that I wrote a presentation about Flapping JS Tests in Rails for the West Michigan Ruby Users Group back in 2015.
What are “Flapping” Tests?
Flapping tests are tests that fail inconsistently. Flapping tests are usually an indication of a race condition that is happening between the test suite and the browser under test. Many times, they occur because some asynchronous code is running and does not return before the test suite makes an assertion.
One common case is that the browser makes some asynchronous request via JavaScript and before the server can return the result and the JavaScript can display it on screen, the assertion is made.
End-to-End or acceptance tests in Elixir and Phoenix are not immune to these kinds of race conditions either.
What is Hound?
Hound is an end-to-end testing framework written for Elixir applications. Hound is different from Elixir’s own ExUnit integration tests. Hound can test the entire stack end-to-user (JS framework and all), closer to what people associate with “acceptance” tests.
Why Use End-to-End Tests?
There is a lot of controversy around end-to-end tests. Generally, it is because acceptance tests can be very slow to run. I feel that, used sparingly, they can provide a lot of value to a developer who may not have testers on their team to test their application end-to-end.
End-to-End tests can be run against critical application behavior and notify you of when you have broken an important part of the application. But, I tend to agree, in most cases, they are not necessary and shouldn’t be used.
What’s Wrong with Testing Asynchronous Text Changes in Hound?
That said, I have written a couple of integration tests for Calculate using Hound and came across an interesting race condition.
Let’s use something that I think is a fairly popular use case:
Problem:
A user performs some action which fires an asynchronous request to the server. Upon response from the server, a “counter” on the page updates based on the request.
Enough already, let’s see some code.
Here’s our template (could be a React/Vue/whatever-new-JS-hotness component):
<ul class="stats">
<li id="messages-count">10</li>
</ul>
And here’s our end-to-end test:
test "increment counter" do
visit(‘/‘)
assert visible_in_element?({:css, “#messages-count”}, “10”)
click(:css, “#send-message”)
assert visible_in_element?({:css, “#messages-count”}, “11”)
end
OK, that’s nice Adam. But what is the actual problem?
Well, the problem lies with a loophole in the WebDriver spec and the way that Hound’s matchers and finder functions work.
Matchers and Finder Functions in Hound
Hound provides two convenient “matchers” for testing that text is visible on a page. visible_in_page?/1 and visible_in_element?/2. The problem, however, is that both of these functions rely on Hound’s internal finder functions.
The way Hound’s finder functions work is that you have to provide a strategy and a selector to the finder functions. The WebDriver spec actually provides the types of valid strategies that are available, however, the WebDriver spec doesn’t actually specify a way to select an element by text…except for links (and partial links…what??)
So, if you are trying to query an element which is not a link, say, our list item above, then you have to provide another strategy and selector. In this case, we use css and an ID.
OK, great. So, that should work then right? Ah, you know I’m setting you up. Unfortunately, Hound is soooo fast at querying the browser, that a race condition occurs.
When our user clicks the #send-message button, the browser makes an asynchronous request and waits for a response before updating the #message-count.
However, Hound does not wait. After performing the click, Hound runs the next line which is to find the element again.
This next part is important, so we’re going to take a closer look.
How Matchers and Finder Functions Work
Under the hood, find_element/2 uses the make_req/2 function which actually will try 5 times (default) to find an element (after a 250ms default wait time), before failing if it can’t find the element.
The problem in our case is not that Hound can’t find the element. The element is still there, but the text may not have changed yet. So, if Hound is too fast, this line will likely fail:
assert visible_in_element?({:css, “#messages-count”}, “11”)
How To Test Asynchronous JavaScript Changes in Hound
First, I’m going to present to you how not to solve this problem. A popular way to solve this is to simply throw in a sleep function for an arbitrary amount of time. Please, please do NOT do this:
assert visible_in_element?({:css, “#messages-count”}, “10”)
click(:css, “#send-message”)
:timer.sleep(1000)
assert visible_in_element?({:css, “#messages-count”}, “11”)
What we’ve done is told the test suite to pause for an entire second before running the next line. The idea here is since we need the test suite to wait for the asynchronous request to return to the browser, we tell the suite to wait for a second, which should allow enough time for the request to be fulfilled.
What’s the big deal? It’s a harmless second right? Well, sure, it starts out as just one second.
As your test suite grows, you’ll likely have more end-to-end tests with more race conditions and more sleep functions.
This is not an effective way to test asynchronous JavaScript changes, but is an effective way to slow down your test suite. And slowing down your test suite is sure-fire way to ensure that no one wants to run your test suite. Believe me, I’ve seen it a lot in Rails projects.
So, what’s a better way?
Well, there are two options.
First, we can write our own test helper function which retries the visible_in_element?/2 matcher if it fails to find the new text. Here’s an example of one I wrote:
# Will automatically retry looking for asynchronous text change
defp text_visible?(element, pattern, retries \\ 5)
defp text_visible?(element, pattern, 0) do
visible_in_element?(element, pattern)
end
defp text_visible?(element, pattern, retries) do
case visible_in_element?(element, pattern) do
true -> true
false ->
:timer.sleep(10)
text_visible?(element, pattern, retries - 1)
end
end
assert text_visible?({:css, ".counter"}, ~r/11/)
This solution uses Elixir’s pattern matching and tail recursion to retry to find the text within the element. If it can’t find it, it will wait 10ms before trying again. It will do this a default of five times.
What makes the method more effective? If at any point in the retries, the visible_in_element?/2 function does return true, then the function will stop retrying and return true back to the assertion. It does not continue on. So, this function could take 20ms or 100ms versus the other method which will ALWAYS wait a full second before continuing.
The second option is less elegant, but perhaps, more straightforward. We can use the :xpath strategy along with a specific selector.
find_element(:xpath, "//ul/li[text()=\"11\"]")
The benefit of this is that it is one-line and can be written without any other code. This method works because it leverages Hound’s own make_req/2 function. The XPATH selector we provided will be retried by make_req/2 until it finds the element on the page.
If you prefer to have it not raise an error, but rather fail an assertion, you can use the element?/2 matcher function with an assertion. Tt would be:
assert element?(:xpath, "//ul/li[text()=\"11\"]")
However, I find this method less readable if you are not used to XPATH’s syntax. You could also make the argument that this method will be less DRY as every time you want to query a text change, you’ll have to write a new XPATH query for that specific element.
Either of these options is better than simply adding a :timer.sleep(1000) to you test suite, because both will retry to find the matching element and will return early if a match is ever found.
You may be wondering, why doesn’t Hound do this automatically with their matchers? Well, that’s up for debate, but I have opened an issue to try and address this going forward. It may not make sense for all cases, but might be worth exploring. Feel free to voice your opinion over there.
This post is part of my series of Elixir Tutorials
Hey, I’m Adam. I’m guessing you just read this post from somewhere on the interwebs. Hope you enjoyed it.
You can also follow me on the Twitters at: @DeLongShot