Deduping arrays in javascript

10 Apr 2014 Tweet this post

Knowing how to deduplicate arrays is pretty useful, and it can be handled in a variety of ways. This is my favorite. It uses two loops, many random “Archer” references, and a cool characteristic of objects to get the job done.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
function dedupe(array) {
  //declares dedupe object
  var dedupeObj = {}
  for (var i = 0; i < array.length; i++) {
    //stores each array element
    dedupeObj[array[i]] = true
  }
  //empties array
  array = [];
  //refills array with the object keys
  for (var key in dedupeObj) {
    array.push(key);
  }
  return array;
}

First, the function maps each item in the array to an object key, and sets the key’s value equal to true. We could just as easily set that value to false or 5 or ‘blue’ or whatever we wanted, all that matters is we’re storing the array as keys. Object keys are always unique.

1
2
3
4
5
6
7
8
var array = ['archer','lana','pam','lana','cyril']

for (var i = 0; i < array.length; i++) {
  dedupeObj[array[i]] = true
}

//Result: {archer: true, lana: true, pam: true, cyril: true}
//Notice only one LAAAANAAAAAAAAAAAA

So when we loop through the array setting elements to object keys, if two elements in the array are both ‘lana’, the object will get a key dedupeObj.lana set to true, and then the loop will set dedupeObj.lana to true again. Uniqueness!

To finish up, we empty the array, and use a simple for-in loop to reload the array with the object’s keys, yielding an array with only one of each element.

###what about an array of objects?

If you want to deduplicate an array of objects, we’ll need to do something slightly different. If we used the above method, objects in the array ALL get stored in the deduplicator as [object Object]. So the deduplication doesn’t work, and the data or archer references stored in that object will be lost.

We need to add in conditionals to account for this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
var example = [{"name":"barry","email":"barry@kgb.ru"},
              {"name":"barry","email":"otherbarry@kgb.ru"},
              {"name":"nikolai","email":"nikolai@kgb.ru"},
              {"name":"nikolai","email":"nikolai@kgb.ru"},
              7,
              'asdf',
              'asdf',
              8];

function dedupe(array) {
  var dedupeObj = {}
  var itemKey;
  for (var i = 0; i < array.length; i++) {
      //make a key of the object's own keys and values
      if(typeof array[i] === 'object'){
          itemKey = array[i].name + "|" + array[i].email;
          dedupeObj[itemKey] = array[i];
      } else {
          dedupeObj[array[i]] = true
      }
  }
  array = [];
  for (var key in dedupeObj) {
      //rebuild the object in the returned array
      if(typeof dedupeObj[key] === 'object'){
          array.push(dedupeObj[key]);
      }
      else {
          array.push(key);
      }
  }
    return array;
}

dedupe(example);

First we use typeof to test if the array element is an object. If so, the object’s key in dedupeObj is a concatenation of that objects own keys and values, to make sure it’s unique. We need to have a conditional in the second loop as well, because this time we need to push a value to the rebuilt array, not a key.

One thing to keep in mind when deduping an array that contains objects, is you need a sort of schema. You need to know what to expect in each object in order to build the deduplicating key itemKey = array[i].name + "|" + array[i].email.

If you have a better or different way of doing things (perhaps with the array.filter() method?) leave them in the comments below!

Tweet this post