Combining two lists and removing duplicates, without removing duplicates in original list

I have two lists that i need to combine where the second list has any duplicates of the first list ignored. .. A bit hard to explain, so let me show an example of what the code looks like, and what i want as a result.

first_list = [1, 2, 2, 5]

second_list = [2, 5, 7, 9]

# The result of combining the two lists should result in this list:
resulting_list = [1, 2, 2, 5, 7, 9]

You’ll notice that the result has the first list, including its two “2” values, but the fact that second_list also has an additional 2 and 5 value is not added to the first list.

Normally for something like this i would use sets, but a set on first_list would purge the duplicate values it already has. So i’m simply wondering what the best/fastest way to achieve this desired combination.

Thanks.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

You need to append to the first list those elements of the second list that aren’t in the first – sets are the easiest way of determining which elements they are, like this:

first_list = [1, 2, 2, 5]
second_list = [2, 5, 7, 9]

in_first = set(first_list)
in_second = set(second_list)

in_second_but_not_in_first = in_second - in_first

result = first_list + list(in_second_but_not_in_first)
print(result)  # Prints [1, 2, 2, 5, 9, 7]

Or if you prefer one-liners 😎

print(first_list + list(set(second_list) - set(first_list)))

Method 2

resulting_list = list(first_list)
resulting_list.extend(x for x in second_list if x not in resulting_list)

Method 3

You can use sets:

first_list = [1, 2, 2, 5]
second_list = [2, 5, 7, 9]

resultList= list(set(first_list) | set(second_list))

print(resultList)
# Results in : resultList = [1,2,5,7,9]

Method 4

first_list = [1, 2, 2, 5]
second_list = [2, 5, 7, 9]

print( set( first_list + second_list ) )

Method 5

You can bring this down to one single line of code if you use numpy:

a = [1,2,3,4,5,6,7]
b = [2,4,7,8,9,10,11,12]

sorted(np.unique(a+b))

>>> [1,2,3,4,5,6,7,8,9,10,11,12]

Method 6

resulting_list = first_list + [i for i in second_list if i not in first_list]

Method 7

Simplest to me is:

first_list = [1, 2, 2, 5]
second_list = [2, 5, 7, 9]

merged_list = list(set(first_list+second_list))
print(merged_list)

#prints [1, 2, 5, 7, 9]

Method 8

You can also combine RichieHindle’s and Ned Batchelder’s responses for an average-case O(m+n) algorithm that preserves order:

first_list = [1, 2, 2, 5]
second_list = [2, 5, 7, 9]

fs = set(first_list)
resulting_list = first_list + [x for x in second_list if x not in fs]

assert(resulting_list == [1, 2, 2, 5, 7, 9])

Note that x in s has a worst-case complexity of O(m), so the worst-case complexity of this code is still O(m*n).

Method 9

Based on the recipe :

resulting_list = list(set().union(first_list, second_list))

Method 10

This might help

def union(a,b):
    for e in b:
        if e not in a:
            a.append(e)

The union function merges the second list into first, with out duplicating an element of a, if it’s already in a. Similar to set union operator. This function does not change b. If a=[1,2,3] b=[2,3,4]. After union(a,b) makes a=[1,2,3,4] and b=[2,3,4]

Method 11

you can use dict.fromkeys to return a list with no duplicates:

def mergeTwoListNoDuplicates(list1, list2):
    """
    Merges two lists together without duplicates
    :param list1:
    :param list2:
    :return:
    """
    merged_list = list1 + list2
    merged_list = list(dict.fromkeys(merged_list))
    return merged_list

Method 12

    first_list = [1, 2, 2, 5]
    second_list = [2, 5, 7, 9]

    newList=[]
    for i in first_list:
        newList.append(i)
    for z in second_list:
        if z not in newList:
            newList.append(z)
    newList.sort()
    print newList

[1, 2, 2, 5, 7, 9]

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0