natural language processing (almost) from scratch.: Difference between revisions
Line 58: | Line 58: | ||
:<math> f_\theta^1 = \langle LT_W([w]_1^T) \rangle _t ^{d_{win}} = \left( \begin{array}{c} \langle W\rangle_{[w]_{t-d_{win}/2}}^1 \\ \vdots \\ \langle W\rangle_{[w]_{t}}^1 \\ \vdots \\ \langle W\rangle_{[w]_{t+d_{win}/2}}^1 \end{array} \right) </math> | :<math> f_\theta^1 = \langle LT_W([w]_1^T) \rangle _t ^{d_{win}} = \left( \begin{array}{c} \langle W\rangle_{[w]_{t-d_{win}/2}}^1 \\ \vdots \\ \langle W\rangle_{[w]_{t}}^1 \\ \vdots \\ \langle W\rangle_{[w]_{t+d_{win}/2}}^1 \end{array} \right) </math> | ||
The output from the first layer is fed to one or more linear layers: | |||
:<math> f_\theta^l = \langle W \rangle^l f_\theta^{l-1} + b^l</math> | |||
Where <math>\,W^l \in R^{n_{hu}^l × n_{hu}^l-1}</math> and <math>b^l \in R^{n_{hu}^l}</math> are parameters to be trained. | |||
Linear layers may be interleaved with layers which apply a hard hyperbolic tangent function over their inputs: | |||
:<math>\left[f_\theta^l\right]_i = HardTanh(\left[f_\theta^{l-1} \right]_i)</math> | |||
Where HardTanh is defined as: | |||
:<math>HardTanh(x) = \left\{ \begin{array}{l} -1\ if\ x < -1 \\ x\ if\ -1 \leq x \leq 1 \\ 1\ if\ x > 1 \end{array} \right. </math> | |||
Finally, the output size of the last layer is the number of tags for a particular task, and each output entry can be interpreted as a score for that tag. | |||
== Sentence-Level Approach == | == Sentence-Level Approach == |
Revision as of 11:59, 12 November 2015
[Please don't edit yet - updating section-by-section]
Overview
Benchmark Tasks
Part of Speech Labelling (POS)
Chunking (CHUNK)
Named Entity Recognition (NER)
Semantic Role Labelling (SLR)
Network Design
Words are preprocessed by lower-casing and encoding capitalization as an additional feature.
Notation and Hyper-Parameters
[math]\displaystyle{ \,f_\theta }[/math](.) denotes any neural network with parameters [math]\displaystyle{ \,\theta }[/math]. Any feed-forward neural network with L layers can be written as a composition of functions as follows: [math]\displaystyle{ \,f_\theta = f_\theta^L(f_\theta^{L-1}(...f_\theta^1(.)...)) }[/math].
[math]\displaystyle{ \,[A]_{i,j} }[/math] denotes the element of a matrix A at row i and column j.
[math]\displaystyle{ \,\langle A\rangle_i^{d_{win}} }[/math] denotes the matrix obtained by concatenating the dwin columns vectors around the ith column vector of a matrix A ∈ Rd1xd2. i.e., [math]\displaystyle{ \,[\langle A\rangle_i^{d_{win}}]^T = ([A]_{1,i-d_{win}/2} ... [A]_{d_1,i-d_{win}/2},...., [A]_{1,i+d_{win}/2} ... [A]_{d_1,i+d_{win}/2}) }[/math].
[math]\displaystyle{ \,\langle A\rangle_i^1 }[/math] is a special case which denotes the ith column vector of A.
[math]\displaystyle{ \,[x]_1^T }[/math] denotes the sequences of elements [math]\displaystyle{ \left\{ x_1, x_2, ... , x_T \right\} }[/math] and [math]\displaystyle{ [x]_i }[/math] denotes the ith element of the sequence.
[math]\displaystyle{ \,d_{wrd} }[/math] is the word vector size, given by the user.
[math]\displaystyle{ \,d_{wrd}^k }[/math] is the vector size for a feature k, given by the user.
[math]\displaystyle{ \,k_{sz} }[/math] is the window size, given by the user.
[math]\displaystyle{ \,n_{hu}^l }[/math] is the number of hidden units for the lth layer, given by the user.
Feature Vectors
Words are fed into the network as indices in a finite dictionary D. The first layer of the network maps each of the word indices to a feature vector.
For each word w ∈ D the lookup table layer LTw(.) gives a dwrd-dimensional feature vector representation:
- [math]\displaystyle{ LT_W(.) = \langle W\rangle_w^1 }[/math]
Where W ∈ Rdwrdx|D|. is a matrix of parameters to be learnt.
For a sequence of words [math]\displaystyle{ \,[x]_1^T }[/math], the lookup table layer produces the following output matrix:
- [math]\displaystyle{ LT_W([w]_1^T) = (\langle W\rangle_{[w]_1}^1 \langle W\rangle_{[w]_2}^1) ... \langle W\rangle_{[w]_T}^1) }[/math]
To add additional features, we can represent the word as K discrete features w ∈ D1×...×Dk, where Dk is the dictionary for the kth feature. Each feature is associated with a lookup table LTwk(.) with parameters Wk ∈ Rdwrdx|D|, and the outputs are concatenated to form the final feature vector.
- [math]\displaystyle{ LT_{w^1,...,w^K}(w)= \left( \begin{array}{c} LT_{w^1}(w_1) \\ \vdots \\ LT_{w^2}(w_2)\end{array} \right) = = \left( \begin{array}{c} \langle W\rangle_{[w]_1}^1 \\ \vdots \\ \langle W\rangle_{[w]_K}^1 \end{array} \right) }[/math]
Windowed Approach
This approach assumes the tag for a word depends mostly on its surrounding words. Given a word to be tagged, we feed a window of ksz words around the word of interest to the lookup table layer, which outputs a matrix of size dwrd×ksz. Finally, the columns of this output matrix are concatenated to use as input to the next layer.
- [math]\displaystyle{ f_\theta^1 = \langle LT_W([w]_1^T) \rangle _t ^{d_{win}} = \left( \begin{array}{c} \langle W\rangle_{[w]_{t-d_{win}/2}}^1 \\ \vdots \\ \langle W\rangle_{[w]_{t}}^1 \\ \vdots \\ \langle W\rangle_{[w]_{t+d_{win}/2}}^1 \end{array} \right) }[/math]
The output from the first layer is fed to one or more linear layers:
- [math]\displaystyle{ f_\theta^l = \langle W \rangle^l f_\theta^{l-1} + b^l }[/math]
Where [math]\displaystyle{ \,W^l \in R^{n_{hu}^l × n_{hu}^l-1} }[/math] and [math]\displaystyle{ b^l \in R^{n_{hu}^l} }[/math] are parameters to be trained.
Linear layers may be interleaved with layers which apply a hard hyperbolic tangent function over their inputs:
- [math]\displaystyle{ \left[f_\theta^l\right]_i = HardTanh(\left[f_\theta^{l-1} \right]_i) }[/math]
Where HardTanh is defined as:
- [math]\displaystyle{ HardTanh(x) = \left\{ \begin{array}{l} -1\ if\ x \lt -1 \\ x\ if\ -1 \leq x \leq 1 \\ 1\ if\ x \gt 1 \end{array} \right. }[/math]
Finally, the output size of the last layer is the number of tags for a particular task, and each output entry can be interpreted as a score for that tag.
Sentence-Level Approach
Results
Unlabelled Data
Discussion
Task-Specific Engineering
References
R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493-2537, 2011.