python eval vs ast.literal_eval vs JSON decode

I am converting 2 MB of data as a string into a dict.
The input is serialized in JSON.

Anyways I am currently using ast.literal_eval and I get the dictionary I want, but then when I tried just running eval it seems to run faster, and also returns the same result.

Is there any reason to use the ast module or the json module when eval works just fine?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

I don’t really like this attitude on stackoverflow (and elsewhere) telling people without any context that what they are doing is insecure and they shouldn’t do it. Maybe it’s just a throwaway script to import some data, in that case why not choose the fastest or most convenient way?

In this case, however, json.loads is not only more secure, but also more than 4x faster (depending on your data).

In [1]: %timeit json.loads(data)
10000 loops, best of 3: 41.6 µs per loop

In [2]: %timeit eval(data)
10000 loops, best of 3: 194 µs per loop

In [3]: %timeit ast.literal_eval(data)
1000 loops, best of 3: 269 µs per loop

If you think about it makes sense json is a such more constrained language/format than python, so it must be faster to parse with an optimized parser.

Method 2

Yes, there’s definitely a reason: eval() is evil. Your code might read untrusted data one day, an this would allow an attacker to run arbitrary code on your machine.

You shouldn’t use ast.literal_eval() to decode JSON either. It cannot decode every valid JSON string and is not meant to be used for this purpose. Simply use json.loads(), it’s reasonably fast.

Method 3

No. Unless you hit one of two scenarios:

1. That’s not JSON!

   Someone puts __import__('os').system('rm -rf /') in the file instead. You are boned.

2. It’s JSON, but not the Python-like part!

   Someone puts true, false, null, or a Unicode escape somewhere in it. Happy birthday.

Method 4

Not an answer exactly, but it should be noted that eval and literal_eval are not the same thing. The ast.literal_eval won’t run arbitrary code.

That said, I agree with using JSON; I just wanted to point out that eval != literal_eval

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0