How to sort and remove duplicates from Python list?

Given a list of strings, I want to sort it alphabetically and remove duplicates. I know I can do this:

from sets import Set
[...]
myHash = Set(myList)

but I don’t know how to retrieve the list members from the hash in alphabetical order.

I’m not married to the hash, so any way to accomplish this will work. Also, performance is not an issue, so I’d prefer a solution that is expressed in code clearly to a fast but more opaque one.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

A list can be sorted and deduplicated using built-in functions:

myList = sorted(set(myList))

• set is a built-in function for Python >= 2.3
• sorted is a built-in function for Python >= 2.4

Method 2

If your input is already sorted, then there may be a simpler way to do it:

from operator import itemgetter
from itertools import groupby
unique_list = list(map(itemgetter(0), groupby(yourList)))

Method 3

If you want to keep order of the original list, just use OrderedDict with None as values.

In Python2:

    from collections import OrderedDict
    from itertools import izip, repeat

    unique_list = list(OrderedDict(izip(my_list, repeat(None))))

In Python3 it’s even simpler:

    from collections import OrderedDict
    from itertools import repeat

    unique_list = list(OrderedDict(zip(my_list, repeat(None))))

If you don’t like iterators (zip and repeat) you can use a generator (works both in 2 & 3):

    from collections import OrderedDict
    unique_list = list(OrderedDict((element, None) for element in my_list))

Method 4

If it’s clarity you’re after, rather than speed, I think this is very clear:

def sortAndUniq(input):
  output = []
  for x in input:
    if x not in output:
      output.append(x)
  output.sort()
  return output

It’s O(n^2) though, with the repeated use of not in for each element of the input list.

Method 5

> but I don’t know how to retrieve the list members from the hash in alphabetical order.

Not really your main question, but for future reference Rod’s answer using sorted can be used for traversing a dict‘s keys in sorted order:

for key in sorted(my_dict.keys()):
   print key, my_dict[key]
   ...

and also because tuple‘s are ordered by the first member of the tuple, you can do the same with items:

for key, val in sorted(my_dict.items()):
    print key, val
    ...

Method 6

For the string data

 output = []

     def uniq(input):
         if input not in output:
            output.append(input)
 print output

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0