​+1 (917) 512-9523
99 Wall Street Suite 1672, New York, NY 10005​
info@investmentscy.com
 
 
Schedule a Meeting
Investment Science | NYC Consulting Services
  • What We Offer
  • Who We Serve
  • About
  • Podcast
  • Insights
  • Case Studies
  • Testimonials
  • Consumers
  • Contact Us
  • What We Offer
  • Who We Serve
  • About
  • Podcast
  • Insights
  • Case Studies
  • Testimonials
  • Consumers
  • Contact Us

Insights

Understanding The Bert Model For NLP

8/8/2020

0 Comments

 
Picture
Picture
The purpose of this post is to give a firm a high-level strategy as to how one could conduct a language processing project. According to wikipedia, natural language processing is 'A subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human languages, in particular how to program computers to process and analyze large amounts of natural language data.' For now, we will first have to create a strategy around language processing. Subsequent posts will show some tutorials on how to conduct language processing in python on some real live data. For now, we are just giving you a pre-cursor on how to get started.

Steps:
1) Select which model to use - For the purpose of this post, you could use the language processing model called 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.' The BERT model was created by Google in 2018, as a neural network (A neural network is a network or circuit of neurons, or in a modern sense, an artificial neural network, composed of artificial neurons or nodes) and is used for pre-training data. One needs to pre-train data for the model to have sufficient information to analyze subsequent data points to make decisions.

What we want to point out is that when our firm conducts any type of data science projects, we choose the best model for you, based off of every single model available. Our findings have been that most individuals will just pick a random model to implement on a client side, but have not looked at other alternatives. Today, you may not be obtaining the best model for your business problem. 
​
How does the BERT model work?

1) All text that is entered into the model must be lowercase (as you can see in the figure below [if you are on mobile it is above] in red)
2) Tokenization must occur (Tokenization is a way of separating a piece of text into smaller units called
​tokens. Here, tokens can be either words, characters, or subwords) - please look at the yellow box below [if you are on mobile it is above]
​3) Next, text segmentation must occur  (segmentation is the process of dividing written text into meaningful units, such as words, sentences, or topics. The term applies both to mental processes used by humans when reading text, and to artificial processes implemented in computers, which are the subject of natural language processing. 
4) The position embedding's must be applied (positional embedding's are used to encode order dependencies into the input representation. However, this input representation only involves static order dependencies based on discrete numerical information, that is, are independent of word content.)
5) In part II of this tutorial, we will select a programming language to conduct natural language processing with. We prefer python, which is a general purpose interpreted programming language.
6) We need to define what we are looking for. For the purpose of this tutorial, we will focus on negative news, so that we can determine whether or not a new piece of news is negative
7) Lastly, each tokenized  text source will bounce off the corpus (https://www.english-corpora.org/wiki/), and generally will be given a computational output of the model as to how close the text is to the corpus to categorize the text. Typically, the Wikipedia corpus is used because the Wikipedia corpus contains the full text of Wikipedia, and it contains 1.9 billion words in more than 4.4 million articles. The corpus can also be a sub-set of data such as all negative news. We need to classify the corpus in order to determine whether or not a piece of news is negative.

​









Stay tuned for the tutorial!
​

Hire Us For Language Processing
0 Comments

    Author

    Michael Kelly has been working within banking technology for over a decade, and his experience spans across algorithmic trading, project management, product management, alternative finance, hedge funds, private equity, and machine learning. This page is intended to educate others across interesting topics, inclusive of finance.

    Archives

    February 2023
    January 2023
    December 2022
    November 2022
    July 2022
    June 2022
    May 2022
    April 2022
    March 2022
    January 2022
    December 2021
    November 2021
    October 2021
    September 2021
    July 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    December 2020
    November 2020
    September 2020
    August 2020

    Categories

    All
    Agile Project Management
    Change Management
    Compliance
    Data Science
    Disruptive Innovation
    Economic Analysis
    Education Training
    Financial Education
    Marketing
    Natural Language Processing
    Portfolio Management
    Technology Strategy
    Trading Research
    Venture Capital
    XBRL

    RSS Feed

Picture
WHAT WE OFFER  /  WHO WE SERVE  /  ABOUT /  ​PODCAST  /  INSIGHTS  /   CASE STUDIES  /  TESTIMONIALS  /  CONSUMERS   /  CONTACT US  
OUR OFFICE
SAY HELLO
If you are interested in working with us or ​just want to say hello simply drop us a line!

Email: info@investmentscy.com
Phone: ​+1 (917) 512-9523
​
RESERVE TIME
OUR INSIGHTS
Stay up to date with our latest content from our insights page.
Subscribe To Our Insights
  © 2023 Investment Science, LLC  |  All Rights Reserved  |  Legal Statement