Normalization

The normalization of Urdu text is necessary to make it useful for the machine learning tasks. In the normalize module, the very basic problems faced when working with Urdu data are handled with ease and efficiency. All the problems and how normalize module handles them are listed below.

This modules fixes the problem of correct encodings for the Urdu characters as well as replace Arabic characters with correct Urdu characters. This module brings all the characters in the specified unicode range (0600-06FF) for Urdu language.

It also fixes the problem of joining of different Urdu words. By joining we mean that when space between two Urdu words is removed, they must not make a new word. Their rendering must not change and even after the removal of space they should look the same.

You can use the library to normalize the Urdu text for correct unicode characters. By normalization we mean to end the confusion between Urdu and Arabic characters, to replace two words with one word keeping in mind the context they are used in. Like the character ‘ﺁ’ and ‘ﺂ’ are to be replaced by ‘آ’. All this is done using regular expressions.

The normalization of Urdu text is necessary to make it useful for the machine learning tasks. This module provides the following functionality:

• Normalizing Single Characters
• Normalizing Combine Characters
• Put Spaces Before & After Digits
• Put Spaces After Urdu Punctuations
• Put Spaces Before & After English Words
• Removal of Diacritics from Urdu Text

urduhack.normalization.normalize_characters(text: str) → str

The most important module in the UrduHack is the character module, defined in the module with the same name. You can use this module separately to normalize a piece of text to a proper specified Urdu range (0600-06FF). To get an understanding of how this module works, one needs to understand unicode. Every character has a unicode. You can search for any character unicode from any language you will find it. No two characters can have the same unicode. This module works with reference to the unicodes. Now as urdu language has its roots in Arabic, Parsian and Turkish. So we have to deal with all those characters and convert them to a normal urdu character. To get a bit more of what the above explanation means is.:

>>> all_fes = ['ﻑ', 'ﻒ', 'ﻓ', 'ﻔ', ]
>>> urdu_fe = 'ف'

All the characters in all_fes are same but they come from different languages and they all have different unicodes. Now as computers deal with numbers, same character appearing in more than one place in a different language will have a different unicode and that will create confusion which will create problems in understanding the context of the data. character module will eliminate this problem by replacing all the characters in all_fes by urdu_fe.

This provides the functionality to replace wrong arabic characters with correct urdu characters and fixed the combine|join characters issue.

Replace urdu text characters with correct unicode characters.

Parameters:text (str) – Urdu text Returns:Returns a str object containing normalized text. Return type:str

Examples

>>> from urduhack.normalization import normalize_characters
>>> # Text containing characters from Arabic Unicode block
>>> text = "مجھ کو جو توڑا ﮔیا تھا"
>>> normalized_text = normalize_characters(text)
>>> # Normalized text - Arabic characters are now replaced with Urdu characters
>>> normalized_text
مجھ کو جو توڑا گیا تھا

urduhack.normalization.normalize_combine_characters(text: str) → str

To normalize combine characters with single character unicode text, use the normalize_combine_characters() function in the character module.

Replace combine|join urdu characters with single unicode character

Parameters:text (str) – Urdu text Returns:Returns a str object containing normalized text. Return type:str

Examples

>>> from urduhack.normalization import normalize_combine_characters
>>> # In the following string, Alif ('ا') and Hamza ('ٔ ') are separate characters
>>> text = "جرأت"
>>> normalized_text = normalize_combine_characters(text)
>>> # Now Alif and Hamza are replaced by a Single Urdu Unicode Character!
>>> normalized_text
جرأت

urduhack.normalization.english_characters_space(text: str) → str

Functionality to add spaces before and after English words in the given Urdu text. It is an important step in normalization of the Urdu data.

this function returns a String object which contains the original text with spaces before & after English words.

Parameters:text (str) – Urdu text Returns:Returns a str object containing normalized text. Return type:str

Examples

>>> from urduhack.normalization import english_characters_space
>>> text = "خاتون Aliyaنے بچوںUzma and Aliyaکے قتل کا اعترافConfession کیا ہے۔"
>>> normalized_text = english_characters_space(text)
>>> normalized_text
خاتون Aliya نے بچوں Uzma and Aliya کے قتل کا اعتراف Confession کیا ہے۔

urduhack.normalization.punctuations_space(text: str) → str

Add spaces after punctuations used in urdu writing

Parameters:text (str) – Urdu text Returns:Returns a str object containing normalized text. Return type:str

Examples

>>> from urduhack.normalization import punctuations_space
>>> text = "ہوتا ہے   ۔  ٹائپ"
>>> normalized_text = punctuations_space(text)
>>> normalized_text
ہوتا ہے۔ ٹائپ

urduhack.normalization.digits_space(text: str) → str

Add spaces before|after numeric and urdu digits

Parameters:text (str) – Urdu text Returns:Returns a str object containing normalized text. Return type:str

Examples

>>> from urduhack.normalization import digits_space
>>> text = "20فیصد"
>>> normalized_text = digits_space(text)
>>> normalized_text
20 فیصد

urduhack.normalization.remove_diacritics(text: str) → str

Remove urdu diacritics from text. It is an important step in pre-processing of the Urdu data. This function returns a String object which contains the original text minus Urdu diacritics.

Parameters:text (str) – Urdu text Returns:Returns a str object containing normalized text. Return type:str

Examples

>>> from urduhack.normalization import remove_diacritics
>>> text = "شیرِ پنجاب"
>>> normalized_text = remove_diacritics(text)
>>> normalized_text
شیر پنجاب

urduhack.normalization.normalize(text: str) → str

To normalize some text, all you need to do pass unicode text. It will return a str with normalized characters both single and combined, proper spaces after digits and punctuations and diacritics removed.

Parameters:text (str) – Urdu text Returns:Normalized urdu text Return type:str Raises:TypeError – If text param is not not str Type.

Examples

>>> from urduhack import normalize
>>> text = "اَباُوگل پاکستان ﻤﯿﮟ20سال ﺳﮯ ، وسائل کی کوئی کمی نہیں ﮨﮯ۔"
>>> normalized_text = normalize(text)
>>> # The text now contains proper spaces after digits and punctuations,
>>> # normalized characters and no diacritics!
>>> normalized_text
اباوگل پاکستان ﻤﯿﮟ 20 سال ﺳﮯ ، وسائل کی کوئی کمی نہیں ﮨﮯ ۔