json.loads allows duplicate keys in a dictionary, overwriting the first value

>>> raw_post_data = request.raw_post_data
>>> print raw_post_data
{"group":{"groupId":"2", "groupName":"GroupName"}, "members":{"1":{"firstName":"fName","lastName":"LName","address":"address"},"1": {"firstName":"f_Name","lastName":"L_Name","address":"_address"}}}
>>> create_request = json.loads(raw_post_data)
>>> print create_request
{u'group': {u'groupName': u'GroupName', u'groupId': u'2'}, u'members': {u'1': {u'lastName': u'L_Name', u'firstName': u'f_Name', u'address': u'_address'}}}

As you can see members with key ‘1’ is overwritten when I use json.dumps()

Is there any way to catch it as exception in python, saying found duplicate keys in request from client ?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

The rfc 4627 for application/json media type recommends unique keys but it doesn’t forbid them explicitly:

The names within an object SHOULD be unique.

From rfc 2119:

SHOULD This word, or the adjective “RECOMMENDED”, mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.

import json

def dict_raise_on_duplicates(ordered_pairs):
    """Reject duplicate keys."""
    d = {}
    for k, v in ordered_pairs:
        if k in d:
           raise ValueError("duplicate key: %r" % (k,))
        else:
           d[k] = v
    return d

json.loads(raw_post_data, object_pairs_hook=dict_raise_on_duplicates)
# -> ValueError: duplicate key: u'1'

Method 2

One alternative I wrote based on the solution posted by other users of this question is to convert those duplicates into an array:

def array_on_duplicate_keys(ordered_pairs):
    """Convert duplicate keys to arrays."""
    d = {}
    for k, v in ordered_pairs:
        if k in d:
            if type(d[k]) is list:
                d[k].append(v)
            else:
                d[k] = [d[k],v]
        else:
           d[k] = v
    return d

And then:

dict = json.loads('{"x": 1, "x": 2}', object_pairs_hook=array_on_duplicate_keys)

gives you the output:

{'x': [1, 2]}

Later one, one can check easily how meany duplicates an entry has by using:

if type(dict['x']) is list:
    print('Non-unique entry in dict at x, found', len(dict['x']),'repetitions.')

Method 3

Alternatively if you want to catch all the duplicate keys (per level) you can use a collections.Counter

from collections import Counter

class KeyWatcher(dict):

    def __init__(self, *args):
        duplicates = [d for d,i in Counter([pair[0] for pair in args[0]]).items() if i > 0]
        if duplicates:
            raise KeyError("Can't add duplicate keys {} to a json message".format(duplicates))
        self.update(*args[0])

json.loads(raw_post_data, object_pairs_hook=KeyWatcher)

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0