Performing Bulk Updates with Ecto And Elixir

Updating one database row in Ecto is straightforward. But figuring out how to bulk update rows using Ecto without updating one record at a time?

Now that is a pain.

What if there were a way to bulk update rows of data in your database using only ONE update call?

Well, there is, and in this post, I’ll show you how you can do it using Ecto without having to write a single line of SQL.

bulk update ecto

Ecto update/2 And Changesets

You can, of course, simply create changesets or structs for each row of data, group them together in something like a list, and then pass them all one-by-one using Enum.each/2 to Ecto.Repo.update/2.

defmodule Example.User do
  use Ecto.Schema
  import Ecto.Changeset

  def changeset(user, params) do
    cast(user, params, [:name, :deactivated])
  end

  schema "users" do
    field :name, :string
    field :deactivated, :boolean
    field :deactivated_on, :date

    timestamps()
  end
end

iex(1)> Enum.each(users, fn(user) ->
...(1)> changeset = Example.User.changeset(user, %{deactivated: true})
...(1)> Example.Repo.update(changeset)
...(1)> end)

...[debug] QUERY OK db=33.5ms
UPDATE "users" SET "deactivated" = $1, \
"updated_at" = $2 WHERE "id" = $3 [true, \
{{2018, 7, 10}, {1, 20, 19, 922659}}, 10]

...[debug] QUERY OK db=4.9ms
UPDATE "users" SET "deactivated" = $1, \
"updated_at" = $2 WHERE "id" = $3 [true, \
{{2018, 7, 10}, {1, 20, 19, 957723}}, 11]

...[debug] QUERY OK db=1.8ms
UPDATE "users" SET "deactivated" = $1, \
"updated_at" = $2 WHERE "id" = $3 [true, \
{{2018, 7, 10}, {1, 20, 19, 963006}}, 12]

...[debug] QUERY OK db=3.8ms
UPDATE "users" SET "deactivated" = $1, \
"updated_at" = $2 WHERE "id" = $3 [true, \
{{2018, 7, 10}, {1, 20, 19, 965138}}, 13]

...[debug] QUERY OK db=8.6ms queue=0.1ms
UPDATE "users" SET "deactivated" = $1, \
"updated_at" = $2 WHERE "id" = $3 [true, \
{{2018, 7, 10}, {1, 20, 19, 969666}}, 14]
:ok

Ecto and update_all

Obviously, performing individual updates can be time-prohibitive if you are performing a lot of updates. Ecto knows this, which is why it also provides the Ecto.Repo.update_all/3 function.

update_all/3 will actually do a one SQL UPDATE call on each of the rows and columns that you provide to it.

Here’s the catch:

The update_all/3 function works a bit differently than update/2.

“update_all” requires you to pass in a “queryable”. That is, something that implements the Queryable protocol.

Update_all and Schemas

Adding use Ecto.Schema and the schema/2 function to a module, automatically converts the current module into a queryable. Like the User example we saw above:

defmodule Example.User do
  use Ecto.Schema 
  import Ecto.Changeset

  def changeset(user, params) do
    cast(user, params, [:name, :deactivated])
  end

  # Queryable
  schema "users" do
    field :name, :string
    field :deactivated, :boolean
    field :deactivated_on, :date

    timestamps()
  end
end

Because Example.User implements uses the schema/2 macro, we can now pass it to update_all/3 as the first argument.

Ecto.Repo.update_all(Example.User,
  set: [deactivated_on: Date.utc_today()]
)

...[debug] QUERY OK source="users" db=21.0ms queue=0.1ms
UPDATE "users" AS u0 SET "deactivated_on" = $1 [{2018, 7, 10}]
{10, nil}

As you can see, Ecto does the update as one single UPDATE call. However, this example updates every single one of our Users.

What if we want to bulk update a sub-section of our Users?

Ecto and Queries

The other option to use with update_all/3 is to build a query directly using Ecto.Query and the from/2 function. Elixir calls these keyword-based queries.

import Ecto.Query

query = from(u in User, where: u.deactivated == false)

Ecto.Repo.update_all(query,
  set: [
    deactivated: true,
    deactivated_on: Date.utc_today()
  ]
)

However, there are a couple of issues with this approach.

First, update_all does not allow all options in queries the way that other query functions do, such as, Repo.all/1.

For example, you cannot use joins in your query or other modifiers like order_by. Ecto will throw an error if you try.

Second, there’s the issue of delay between queries.

Let’s take our example above. Let’s say you wanted to query a bunch of Users in your database, take some action on them (like send an email), and then update them in the database. Now, let’s say this could be 100s or 1000s of users.

You can build a query to grab the users from the database as structs which you can then use to do whatever-you-need-to-do. You could then use that same query you built before and pass that into the update_all/2 function (provided that query doesn’t have any exceptions noted above).

Here’s an example:

import Ecto.Query

query = from u in User, where: u.deactivated == false
users = Ecto.Repo.all(query)

do_something(users)

Ecto.Repo.update_all(query, 
  set: [
    deactivated: true, 
    deactivated_on: Date.utc_today()
  ]
)

The problem with this approach is, in the time it took us to query the database, take some actions, and update the database, what if another record was inserted which qualifies for our query? It will now get updated with the update_all/2 function without any actions being taken on it.

Use IDs for updating

If you wanted to use a complicated query for your updates, you can try this approach. Build a complicated query and retrieve the records you want using Ecto.

import Ecto.Query

query =
  from(u in User,
    join:
      p(
        from(Post,
          where: p.author_id = u.id,
          where: u.deactivated == false
        )
      )
   )

users = Ecto.Repo.all(query)

do_something(users)

Now, instead of passing the same query we used before (and we can’t since it contains a join clause), we create a list of the user ids that we retrieved.

import Ecto.Query

query =
  from(u in User,
    join:
      p(
        from(Post,
          where: p.author_id = u.id,
          where: u.deactivated == false,
          select: u.id
        )
      )
  )

users = Ecto.Repo.all(query)

do_something(users)

users_id = Enum.map(users, &(&1.id))

new_query = from u in User, where: id in ^users_id

Ecto.Repo.update_all(new_query, 
  set: [
    deactivated: true, 
    deactivated_on: Date.utc_today()
  ]
)

So, instead of update_all updating every user, it will only update the user with the ids that we passed into the query.

Problem with update_all

As Elixir points out in the documentation, using update_all/2 with a queryable means that certain autogenerated columns, such as inserted_at and updated_at, will not be updated when using update_all/2 as they would be if you used update.

There is one more way that you can do a bulk update using Ecto.

Upserts with insert_all

The insert_all/3 callback of Ecto has an interesting option called on_conflict. If you specify the on_conflict AND you provide a list of structs with ids of database rows that already exist in your database, a single UPDATE call will be made passing the ids provided your list.

Let’s take our use example before. We do the same thing as before: build a complicated query, pass the structs to a function, and then map over the structs taking their ids.

Finally, we pass that list into insert_all/3 with the on_conflict option. We also pass in [set: [deactivated: true, deactived_on: Date.utc_today()]].

import Ecto.Query

query =
  from(u in User,
    join:
      p(
        from(Post,
          where: p.author_id = u.id,
          where: u.deactivated == false,
          select: u.id
        )
      )
  )

users = Ecto.Repo.all(query)

do_something(users)

new_users = Enum.map(users, &(&1.id))

Example.Repo.insert_all(
  Example.User,
  ids,
  on_conflict: [
    set: [
      deactivated: true,
      deactivated_on: DateTime.utc_today()
    ]
  ],
  conflict_target: [:id]
)

The database will see the “on conflict” option and perform an UPDATE call with the set option we passed in. It will update the database rows in one call instead of one for each row.

The Problem with insert_all/3

Unfortunately, insert_all/3 is not without its flaws either. It also will not update any autogenerated columns, such as timestamps. Not only that, but if the row has a column that is not provided by your structs, in our case the list ids, then insert_all will replace those values with NULL in those columns.

A solution to this problem would be to provide these columns with values in your struct before you pass them to insert_all.

now = DateTime.utc_now()

new_users =
  users
  |> Enum.map(& &1.id)
  |> Enum.map(fn id -> struct(id, %{inserted_at: now, updated_at: now}) end)

Example.Repo.insert_all(
  Example.User,
  new_users,
  on_conflict: [
    set: [
      deactivated: true,
      deactivated_on: DateTime.utc_today()
    ]
  ],
  conflict_target: [:id]
)

As stated in the documentation, insert_all/3 can return some cryptic values depending on the database/persistence layer you choose to use. Some will return the amount of rows updated (postgresql), others will return the amount of rows attempted to be updated or inserted (mysql), etc.

Hope this information was helpful. If you have any questions or have another tip, feel free to leave a comment below.

This post is part of my series of Elixir Tutorials


Hey, I’m Adam. I’m guessing you just read this post from somewhere on the interwebs. Hope you enjoyed it. When I’m not writing these blog posts, I’m a freelance Elixir and Ruby developer and working on Calculate, a product which makes it easier for you to create software estimates for your projects. Feel free to leave a comment if you have any questions.

You can also follow me on the Twitters at: @DeLongShot

How to Use IEx.pry in Elixir Tests

Elixir’s IEx.pry is a great debugging tool.

It allows you to stop time (“pry”) in the middle of your application in order to inspect the conditions in which your app is currently running.

However, if you’ve ever had to try and use IEx.pry while running your Elixir tests using mix test, you’ve probably encountered a problem.

It won’t actually work.

You may have seen an error similar to this:

Cannot pry #PID<0.474.0> at Example.ProjectsTest ... 
Is an IEx shell running?

I’ll show you the way to use IEx.pry/0 in your Elixir tests and a couple of quick tips to make using IEx.pry in your tests even easier.

Solution

The solution is straightforward. You have to run your mix tests in an interactive elixir session.

How do you do that?

Simply prepend your mix test command with iex -S.

For example, iex -S mix test would run all of your available tests and anywhere you’ve put an IEx.pry the shell will ask you:

Request to pry #PID<0.464.0> at Example.ExampleTest...

....

Allow? [Yn]

Typing Y will drop you into the pry prompt.

Avoiding Timeouts Using IEx.pry in ExUnit

If you are going to be debugging for a while in your pry shell, you should consider adding the the --trace to the test task, i.e. iex -S mix test --trace to avoid timeouts while you are in IEx.pry.

Otherwise, your test process may timeout and crash while you are still debugging using pry. It may raise an ExUnit.TimeoutError after 60 seconds:

 ** (ExUnit.TimeoutError) test timed out after 60000ms. You can change the timeout:
...

Running One Test File or Line Numbers

You can, of course, do the same thing when running a single test file or even a single test.

# Run single test file
iex -S mix test --trace path/to/simple_test.exs

# Run single test
iex -S mix test --trace path/to/simple_test.exs:12

But here’s the deal:

If you are like me, you rarely run your Elixir tests in an interactive shell.

Why?

Because I want to write a test, run it quickly, watch it fail, and then make it pass. Over and over. Iterating quickly. I don’t have time to stop and run an interactive session every time.

Occasionally, however, I run into bugs when making my Elixir tests pass. When this occurs, my debugging workflow works like this:

I’ll try a couple of different things and if none of those reveals the bug, I’ll drop in a IEx.pry somewhere in my test (don’t forget to require IEx).

I do this enough that remembering the correct sequence of commands and typing them out becomes time-consuming and tedious. So, I’ve come up with a few tricks to speed this process up.

Here are two tips that will allow you to do this quickly.

VIM map for IEx Pry

Dropping in a IEx.pry in your code requires that you add both require IEx and IEx.pry to your code.

That is too much typing.

So, to save my hands from carpal tunnel, I have leveraged mappings in Vim. I added this mapping to my .vimrc file:

nmap <leader>r orequire IEx; IEx.pry<esc>

Now, all I have to do is hit “[spacebar] + r” to insert a require IEx; IEx.pry into my test. But, that is only have the problem. I’ll need to run my test again with an IEx shell.

Shell alias for IEx shell tests

Let’s say you’ve added IEx.pry to your test, but now you want to run the same test you just ran again, but this time using pry.

You’ll probably have to go back through your history, find the test you ran, then move your cursor to the beginning of the line, then add iex -S to your mix test command.

Or if you are like me, I’ll often forget to prepend the iex -S in front of my mix test.

Too. Much. Typing.

I came up with simple Bash/Zsh alias that I use all the time and now, you can too.

# Zsh users
alias repry='fc -e - mix\ test=iex\ -S\ mix\ test\ --trace mix\ test'

# Bash users
alias repry='fc -s mix\ test=iex\ -S\ mix\ test\ --trace mix\ test'

What does this alias do? Well, it uses the ‘fc’ command in *nix that searches for the last mix test in your command history and then replaces the mix test with iex -S mix test --trace. This alias works regardless of whether I ran my entire test suite using mix test or ran one test using a specific line number.

Here’s a graphic to pull it all together:

IEx Pry Test

Hope this information was helpful. If you have any questions or have another tip, feel free to leave a comment below.

This post is part of my series of Elixir Tutorials


Hey, I’m Adam. I’m guessing you just read this post from somewhere on the interwebs. Hope you enjoyed it. When I’m not writing these blog posts, I’m a freelance Elixir and Ruby developer and working on Calculate, a product which makes it easier for you to create software estimates for your projects. Feel free to leave a comment if you have any questions.

You can also follow me on the Twitters at: @DeLongShot

How to Solve Elixir’s “Module Is Not Available” Error

Elixir’s “module is not available” error can drive you nuts.

But fear not:

After writing a lot of Elixir and seeing this error often, I’ll developed a few quick tricks that you can use to solve why your module is not available in seconds instead of minutes.

Here’s 4 quick things you can check.

Elixir Module Not Available Tips

Check Your Module Definition

This almost goes without saying, but every time I don’t check the module definition first, it turns out I’ve misspelled one of my module definitions.

Check the top (usually top) of your file for your defmodule function. Some things to look for are that everything is spelled correctly, i.e. App.User and not App.Users, that you have included your application name if needed, i.e. App.User not User, that you are using the proper punctuation, i.e. App.User.Note not App.UserNote.

Did You Forget to include alias?

Sometimes, in a bout of “in the zone” programming, you may have written a call to a module that you believed you had aliased, but forgot to include the alias function call.

For example, trying to call User.full_name() when the full function call is App.User.full_name() would result in the module is not available error. App.User, in this case, would need to be aliased first.

Inspect Loaded Modules

Still can’t find the module? Well, one debugging trick is to check a list of all the currently loaded, user-defined modules in the application.

There’s two ways you can do this:

First, you can check the my_app.app compiled file in your /_build directory.

Specifically, this file is usually located in:

"_build/#{MIX_ENV}/lib/#{YOUR_APP_NAME_HERE}/ebin"

While this file may appear a bit cryptic, you can probably ascertain what the modules entry shows. It’s a list of user-defined modules available to your application from compilation.

The second trick is:

Use :application.get_key(:my_app, :modules) wherever the module is not available error is occurring. Combine this with the IO.inspect/1 function and it will show you all of the available user-defined Elixir modules at that point in time.

IO.inspect :application.get_key(:my_app, :modules)

It’s a quick way to see all the user-defined Elixir modules available to the :my_app application at that point in your Elixir application.

Using this Elixir debugging trick may allow you to catch if you have badly misspelled a module or perhaps forgot to include a module for compilation.

Which brings us to my last tip:

Make Sure Your Module is Available to Your Application

My last tip is the most complex in terms of implementation and understanding.

But here goes:

You need to check that your Elixir module is available to your application by telling the elixirc compiler to generate a .beam file for your module.

Why would the Elixir compiler not compile a .beam file? Well, 2 common reasons:

  1. You placed the module in a directory outside of lib and did not
    tell the compiler.

  2. You used the .exs extension instead of .ex

The first reason can be puzzling if you do not realize the problem. By default, the Elixir compiler only looks to compile files and load modules located in the lib directory of an Elixir application.

As your Elixir application complexity increases, you may decide to include Elixir modules that need to be compiled outside of the lib directory. A good example of this are modules that may be used for supporting tests such as mocks.

The solution to this problem is straightforward, but a bit hidden in the Elixir documentation.

You need to tell the Elixir compiler the directory where your modules are located.

To do this, you need to use the Elixir elixirc_paths option in your project configuration. All you need to do is set the elixirc_paths key to a list of paths of the directories that you want the Elixir compiler to compile.

Here’s the catch:

In many cases, you probably do not want to compile the same files across different environments.

For example, in a production environment, you may not want to compile the modules that you use for your test environment.

To get around this, we can use a function that we define in mix.exs that returns a different lists of paths depending on the environment.

In fact, this is exactly what Phoenix Framework does by default in new projects.

  def project do
    [
      ...
      elixirc_paths: elixirc_paths(Mix.env),
      ...
    ]
  end

  # Specifies which paths to compile per environment.
  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_),     do: ["lib"]

You can see in the code above the elixirc_paths/1 function includes the “test/support” directory in the Elixir compiler paths for the test environment, but not for other environments such as development.

Setting this directory in the Elixir compiler paths means that any modules defined within .ex extension files will automatically be compiled and their modules loaded for your application.

Which brings me to the second issue:

Another mistake that you may make is to use the .exs extension instead of the .ex extension. You may be defining Elixir modules that you need to compile for use in your application in files with a .exs extension.

The .exs extension is used for scripting. Like the .ex extension, .exs files are compiled, their modules loaded, and executed.

It is very well documented that .ex extension files get compiled down to .beam files. The modules in your compiled .beam files get loaded by your application automatically when the application starts.

However, the compiled BEAM bytecode from .exs files does not get output into a .beam file the way that .ex files do. They are, in essence, ephemeral. They are compiled, loaded, and executed, and then disappear. Hence, if you are trying to access a module in an .exs file from your application, it will fail with the “module not available” error.

Hope this information was helpful. If you have any questions or have another tip, feel free to leave a comment below.

This post is part of my series of Elixir Tutorials


Hey, I’m Adam. I’m guessing you just read this post from somewhere on the interwebs. Hope you enjoyed it. When I’m not writing these blog posts, I’m a freelance Elixir and Ruby developer and working on Calculate, a product which makes it easier for you to create software estimates for your projects. Feel free to leave a comment if you have any questions.

You can also follow me on the Twitters at: @DeLongShot

How To Test Asynchronous Text Changes with Hound and Phoenix

Writing asynchronous acceptance tests in Hound for Elixir and Phoenix can be difficult, especially if you are using a JavaScript framework like React, Vue.js, or Angular.

If you have ever used end-to-end testing in your web application’s test suite, you have undoubtedly come across the issue of “flapping” tests.

In many of the Ruby on Rails projects that I get asked to work on, I come across this problem frequently. So much so, that I wrote a presentation about Flapping JS Tests in Rails for the West Michigan Ruby Users Group back in 2015.

What are “Flapping” Tests?

Flapping tests are tests that fail inconsistently. Flapping tests are usually an indication of a race condition that is happening between the test suite and the browser under test. Many times, they occur because some asynchronous code is running and does not return before the test suite makes an assertion.

One common case is that the browser makes some asynchronous request via JavaScript and before the server can return the result and the JavaScript can display it on screen, the assertion is made.

End-to-End or acceptance tests in Elixir and Phoenix are not immune to these kinds of race conditions either.

What is Hound?

Hound is an end-to-end testing framework written for Elixir applications. Hound is different from Elixir’s own ExUnit integration tests. Hound can test the entire stack end-to-user (JS framework and all), closer to what people associate with “acceptance” tests.

Why Use End-to-End Tests?

There is a lot of controversy around end-to-end tests. Generally, it is because acceptance tests can be very slow to run. I feel that, used sparingly, they can provide a lot of value to a developer who may not have testers on their team to test their application end-to-end.

End-to-End tests can be run against critical application behavior and notify you of when you have broken an important part of the application. But, I tend to agree, in most cases, they are not necessary and shouldn’t be used.

What’s Wrong with Testing Asynchronous Text Changes in Hound?

That said, I have written a couple of integration tests for Calculate using Hound and came across an interesting race condition.

Let’s use something that I think is a fairly popular use case:

Problem:

A user performs some action which fires an asynchronous request to the server. Upon response from the server, a “counter” on the page updates based on the request.

Enough already, let’s see some code.

Here’s our template (could be a React/Vue/whatever-new-JS-hotness component):

<ul class="stats">

  <li id="messages-count">10</li>
</ul>

And here’s our end-to-end test:

test "increment counter" do

  visit(‘/‘)

  assert visible_in_element?({:css, “#messages-count”}, “10”)

  click(:css, “#send-message”)
  assert visible_in_element?({:css, “#messages-count”}, “11”)

end


OK, that’s nice Adam. But what is the actual problem?

Well, the problem lies with a loophole in the WebDriver spec and the way that Hound’s matchers and finder functions work.

Matchers and Finder Functions in Hound

Hound provides two convenient “matchers” for testing that text is visible on a page. visible_in_page?/1 and visible_in_element?/2. The problem, however, is that both of these functions rely on Hound’s internal finder functions.

The way Hound’s finder functions work is that you have to provide a strategy and a selector to the finder functions. The WebDriver spec actually provides the types of valid strategies that are available, however, the WebDriver spec doesn’t actually specify a way to select an element by text…except for links (and partial links…what??)

So, if you are trying to query an element which is not a link, say, our list item above, then you have to provide another strategy and selector. In this case, we use css and an ID.

OK, great. So, that should work then right? Ah, you know I’m setting you up. Unfortunately, Hound is soooo fast at querying the browser, that a race condition occurs.

When our user clicks the #send-message button, the browser makes an asynchronous request and waits for a response before updating the #message-count.

However, Hound does not wait. After performing the click, Hound runs the next line which is to find the element again.

This next part is important, so we’re going to take a closer look.

How Matchers and Finder Functions Work


Under the hood, find_element/2 uses the make_req/2 function which actually will try 5 times (default) to find an element (after a 250ms default wait time), before failing if it can’t find the element.

The problem in our case is not that Hound can’t find the element. The element is still there, but the text may not have changed yet. So, if Hound is too fast, this line will likely fail:


assert visible_in_element?({:css, “#messages-count”}, “11”)

How To Test Asynchronous JavaScript Changes in Hound

First, I’m going to present to you how not to solve this problem. A popular way to solve this is to simply throw in a sleep function for an arbitrary amount of time. Please, please do NOT do this:

assert visible_in_element?({:css, “#messages-count”}, “10”)

click(:css, “#send-message”)
:timer.sleep(1000)
assert visible_in_element?({:css, “#messages-count”}, “11”)

What we’ve done is told the test suite to pause for an entire second before running the next line. The idea here is since we need the test suite to wait for the asynchronous request to return to the browser, we tell the suite to wait for a second, which should allow enough time for the request to be fulfilled.

What’s the big deal? It’s a harmless second right? Well, sure, it starts out as just one second.

As your test suite grows, you’ll likely have more end-to-end tests with more race conditions and more sleep functions.

This is not an effective way to test asynchronous JavaScript changes, but is an effective way to slow down your test suite. And slowing down your test suite is sure-fire way to ensure that no one wants to run your test suite. Believe me, I’ve seen it a lot in Rails projects.

So, what’s a better way?

Well, there are two options.

First, we can write our own test helper function which retries the visible_in_element?/2 matcher if it fails to find the new text. Here’s an example of one I wrote:

# Will automatically retry looking for asynchronous text change
defp text_visible?(element, pattern, retries \\ 5)

defp text_visible?(element, pattern, 0) do
  visible_in_element?(element, pattern)
end

defp text_visible?(element, pattern, retries) do
  case visible_in_element?(element, pattern) do
    true -> true

    false ->
      :timer.sleep(10)
      text_visible?(element, pattern, retries - 1)
  end
end

assert text_visible?({:css, ".counter"}, ~r/11/)

This solution uses Elixir’s pattern matching and tail recursion to retry to find the text within the element. If it can’t find it, it will wait 10ms before trying again. It will do this a default of five times.

What makes the method more effective? If at any point in the retries, the visible_in_element?/2 function does return true, then the function will stop retrying and return true back to the assertion. It does not continue on. So, this function could take 20ms or 100ms versus the other method which will ALWAYS wait a full second before continuing.

The second option is less elegant, but perhaps, more straightforward. We can use the :xpath strategy along with a specific selector.

find_element(:xpath, "//ul/li[text()=\"11\"]")

The benefit of this is that it is one-line and can be written without any other code. This method works because it leverages Hound’s own make_req/2 function. The XPATH selector we provided will be retried by make_req/2 until it finds the element on the page.

If you prefer to have it not raise an error, but rather fail an assertion, you can use the element?/2 matcher function with an assertion. Tt would be:


assert element?(:xpath, "//ul/li[text()=\"11\"]")

However, I find this method less readable if you are not used to XPATH’s syntax. You could also make the argument that this method will be less DRY as every time you want to query a text change, you’ll have to write a new XPATH query for that specific element.

Either of these options is better than simply adding a :timer.sleep(1000) to you test suite, because both will retry to find the matching element and will return early if a match is ever found.

You may be wondering, why doesn’t Hound do this automatically with their matchers? Well, that’s up for debate, but I have opened an issue to try and address this going forward. It may not make sense for all cases, but might be worth exploring. Feel free to voice your opinion over there.

This post is part of my series of Elixir Tutorials


Hey, I’m Adam. I’m guessing you just read this post from somewhere on the interwebs. Hope you enjoyed it. When I’m not writing these blog posts, I’m a freelance Elixir and Ruby developer and working on Calculate, a product which makes it easier for you to create software estimates for your projects. Feel free to leave a comment if you have any questions.

You can also follow me on the Twitters at: @DeLongShot

How to Count Specific Items in a Collection with Elixir

So, I was working on this exercise over at Exercism.io, when I stumbled up an interesting function.

To make a long story short, I needed a count of a specific character (codepoint) in a character list i.e. 'hello' contains two ls. However, this solution could apply to any collection which implements the Enumerable protocol in Elixir.

I have been reading Programming in Elixir. The author, Dave Thomas, emphasizes the use of recursion and pattern matching throughout Elixir. So, of course, I had implemented the solution using recursion and pattern matching. You can see that solution here.

You Can Count (Ha!) on Elixir

Of course, I missed the obvious. Elixir already had a very convenient function for doing exactly this: Enum.count/2.

I had anticipated that Elixir would have a function to do a straight count of all the items in a collection (which it does: Enum.count/1 doc). However, I had not considered one for looking for a specific kind of item.

Well, Enum.count/2 provides a convenient API for this exact problem. You simply pass the enumerable (in my case, the character list) and a function which takes one argument (the current item) and will return true for whatever items you want to count.

Here’s a convenient graphic to demonstrate:

Elixir Enum Count

Elixir Enum.count/2 Examples

Of course, the expression used to evaluate the current item does not need to be a simple equality operation. Here are some examples using a variety of different operators:

Count all the binaries (strings) which equal “price”:

Enum.count(["Total", "retail", "price", "$14.95"], &(&1 == "price"))
1

Pipe syntax:

["Total", "retail", "price", "$14.95"]
  |> ... # some other functions
  |> Enum.count(&(&1 == "price"))

Count all the binaries which are members of a list:

roles = ["User", "Customer", "Partner", "Admin"]
Enum.count(roles, &Enum.member?(["Admin", "Editor", "Developer"], &1))
1

Count all the binaries (strings) which match a regular expression:

Enum.count(["Total", "price", "$14.95"], &(&1 =~ ~r/\$\d+\.\d+/))
1

Count all the binaries (strings) in a list:

Enum.count([1, "list", 3], &is_binary(&1))
1

Count all the binaries (strings) in a multi-type list:

Enum.count([1, "list", 3], &is_binary(&1))
1

Count all the integers in a multi-type list:

Enum.count([1, "list", 3], &is_integer(&1))
2

Count all the integers greater than 3:

Enum.count([1,2,4,8,16,32], &(&1 > 3))
4

Read the documentation on Enum.count/2

This post is part of my series of Elixir Tutorials


Hey, I’m Adam. I’m guessing you just read this post from somewhere on the interwebs. Hope you enjoyed it. When I’m not writing these blog posts, I’m a freelance Elixir and Ruby developer and working on Calculate, a product which makes it easier for you to create software estimates for your projects. Feel free to leave a comment if you have any questions.

You can also follow me on the Twitters at: @DeLongShot