Perl and some other current regex engines support Unicode properties, such as the category, in a regex. E.g. in Perl you can use p{Ll} to match an arbitrary lower-case letter, or p{Zs} for any space separator. I don’t see support for this in either the 2.x nor 3.x lines of Python (with due regrets). Is anybody aware of a good strategy to get a similar effect? Homegrown solutions are welcome.
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
The regex module (an alternative to the standard re module) supports Unicode codepoint properties with the p{} syntax.
Method 2
Have you tried Ponyguruma, a Python binding to the Oniguruma regular expression engine? In that engine you can simply say p{Armenian} to match Armenian characters. p{Ll} or p{Zs} work too.
Method 3
You can painstakingly use unicodedata on each character:
import unicodedata
def strip_accents(x):
return u''.join(c for c in unicodedata.normalize('NFD', x) if unicodedata.category(c) != 'Mn')
Method 4
Speaking of homegrown solutions, some time ago I wrote a small program to do just that – convert a unicode category written as p{...} into a range of values, extracted from the unicode specification (v.5.0.0). Only categories are supported (ex.: L, Zs), and is restricted to the BMP. I’m posting it here in case someone find it useful (although that Oniguruma really seems a better option).
Example usage:
>>> from unicode_hack import regex
>>> pattern = regex(r'^\p{Lu}(\p{L}|\p{N}|_)*')
>>> print pattern.match(u'疂_1+2').group(0)
疂_1
>>>
Here’s the source. There is also a JavaScript version, using the same data.
Method 5
You’re right that Unicode property classes are not supported by the Python regex parser.
If you wanted to do a nice hack, that would be generally useful, you could create a preprocessor that scans a string for such class tokens (p{M} or whatever) and replaces them with the corresponding character sets, so that, for example, p{M} would become [u0300–u036Fu1DC0–u1DFFu20D0–u20FFuFE20–uFE2F], and P{M} would become [^u0300–u036Fu1DC0–u1DFFu20D0–u20FFuFE20–uFE2F].
People would thank you. 🙂
Method 6
Note that while p{Ll} has no equivalent in Python regular expressions, p{Zs} should be covered by '(?u)s'.
The (?u), as the docs say, “Make w, W, b, B, d, D, s and S dependent on the Unicode character properties database.” and s means any spacing character.
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0