Machine Learning for Natural Language Processing (NLP)

Natural Language Processing, or NLP for short, is broadly defined as the automatic manipulation of natural language, such as speech and text, by software.

What Is NLP Used For?

NLP is used to understand the structure and meaning of human language by analyzing different aspects such as syntax, semantics, pragmatics and morphology. Then, computer science turns this linguistic knowledge into rule-based machine learning algorithms that can solve specific problems and perform desired tasks.

The most popular supervised NLP machine learning algorithms are:

• Support Vector Machines

• Bayesian Networks

• Maximum Entropy

• Conditional Random Field

• Neural Networks/Deep Learning

Lexalytics uses supervised machine learning to build and improve core text analytics functions and our NLP capabilities.

Supervised machine learning for natural language processing and text analysis

Tokenisation

Tokenization involves breaking a text document into pieces that a machine can understand, such as words.

Part of voice tagging

Part of voice tagging (PoS tagging) involves identifying each token's part of speech (noun, adverb, adjective, etc.) and then tagging it as such. PoS markup forms the basis for a number of important natural language processing tasks.

Recognition of named entities

In their simplest form, named entities are people, places, and things (products) mentioned in a text document. Unfortunately, entities can also be hashtags, email addresses, street addresses, phone numbers, and Twitter handles. In fact, just about anything can be an entity if you look at it the right way. And don't make us enunciate on tangential references.

Sentiment analysis

Sentiment analysis involves determining whether a piece of writing is positive, negative, or neutral, and then assigning a weighted sentiment score to each entity, theme, topic, and category in the document. This is an incredibly complex task that varies enormously depending on the context. For example, take the phrase "sick burn." In the context of video games, that might actually be a positive statement.

Unsupervised Machine Learning for Natural Language Processing and Text Analytics

Unsupervised machine learning involves training a model without pre-tagging or annotation. Some of these techniques are surprisingly easy to understand.

Matrix factorization is another unsupervised NLP machine learning technique. This uses "latent factors" to decompose a large matrix into the combination of two smaller matrices. Latent factors are similarities between items.

Conclusion

Natural language processing generally refers to the study and development of computer systems capable of interpreting speech and text as humans speak and type it naturally. Human communication is sometimes frustratingly inaccurate; we all use colloquialisms, abbreviations and often don't bother to correct spelling mistakes.