A classification training system for binary and multi-class classification comprises a neural network operable to perform classification of input data, a training dataset including pre-segmented, labeled training samples, and a classification training module operable to train the neural network using the training dataset. The classification training module includes a forward pass processing module, and a backward pass processing module. The backward pass processing module is operable to determine whether a current frame is in a region of target (ROT), determine ROT information such as beginning and length of the ROT and update weights and biases using a cross-entropy cost function and connectionist temporal classification cost function. The backward pass module further computes a soft target value using ROT information and computes a signal output error using the soft target value and network output value.
|
1. A computer-implemented method for training a recurrent neural network for speech recognition comprising:
receiving, at a computing device, a stream of training data including a plurality of input samples having segmented labeled data;
computing, by the computing device, a network output for each input sample in a forward pass through the training data; and
updating, by the computing device, weights and biases through a backward pass through the training data, including determining whether an input frame is in a Region of Target (ROT), estimating the update of the weights and the biases of the recurrent neural network, and adaptively learning to improve a convergence rate of the recurrent neural network.
12. A classification training system comprising:
a neural network for use in a classification of input data;
a training dataset providing segmented labeled training examples; and
a classification training module for training the neural network using the segmented labeled training data, the classification training module comprising a forward pass processing module, and a backward pass processing module;
wherein the forward pass processing module is operable to train the neural network by generating neural network outputs for the training data using current value of weights and bias for the neural network; and
wherein the backward pass processing module is configured to train the neural network by updating the weights and biases by passing backward through generated neural network outputs, determining whether an input frame is in a Region of Target (ROT), and adaptively learning to improve a convergence rate of the neural network.
2. The computer-implemented method of
3. The computer-implemented method of
4. The computer-implemented method of
5. The computer-implemented method of
6. The computer-implemented method of
7. The computer-implemented method of
8. The computer-implemented method of
9. The computer-implemented method of
calculating a signal output error for each network output based on the determination of whether the input frame is in the ROT; and
updating the weights and biases during a backward pass based on the calculated signal output error.
10. The computer-implemented method of
11. The computer-implemented method of
obtaining ROT information for the input frame, including a length and beginning of the ROT;
computing a soft target value using the ROT information; and
computing the signal output error using the computed soft target value and network output value.
13. The training system of
14. The training system of
15. The training system of
16. The training system of
17. The training system of
18. The training system of
19. The training system of
20. The training system of
|
This application claims the benefit of and priority to both U.S. Provisional Patent Application No. 62/457,663 filed Feb. 10, 2017 and entitled “Connectionist Temporal Classification (CTC) Network Using Cross-Entropy Cost Function for Binary Classification” and U.S. Provisional Patent Application No. 62/465,723 filed Mar. 1, 2017 and entitled “Recurrent Network For Connectionist Temporal Classification (CTC) Using Segmented Labeled Sequence Data,” both of which are incorporated herein by reference in their entirety.
The present application relates generally to information classification, and more particularly, for example, to binary and multi-class classification systems and methods for use in speech recognition systems.
Recurrent neural networks are commonly used in information classification systems, such as binary classification speech recognition applications, such as keyword spotting, and multi-class classification speech recognition applications, such as phoneme recognition. In one approach, an Artificial Neural Network (ANN) classifier is combined with a Hidden Markov Model (HMM) to transform network classifications into labeled sequences. The HMM is typically used to model the long range sequential structure of the data while the ANN is used to provide localized classifications. The use of an HMM model, however, requires unnecessary assumptions about the data. A Recurrent Neural Network (RNN) may also be combined with HMMs to label input sequences, but traditional approaches fail to exploit the full potential of RNN for modeling sequential data.
Further, many approaches are often highly complex and may not be practical for use in applications with memory, power and processing limitations, such as mobile telephones and other low power devices. Efforts to reduce complexity often come at the cost of less flexibility, memory inefficiencies, and other undesirable performance measures. In view of the foregoing, there is a need in the art for solutions to optimize information classification systems for training neural networks that are both fast and resource efficient.
The present disclosure provides systems and methods that address a need in the art for improved binary and multi-class classification. In various embodiments, a computer-implemented method for training a recurrent neural network for speech recognition includes receiving, at a computing device, a stream of training data including a plurality of input samples having segmented labeled data, computing, by the computing device, a network output for each input sample in a forward pass through the training data, and updating, by the computing device, weights and biases through a backward pass through the training data, including determining whether an input frame is in a Region of Target (ROT), estimating the update of the weights and the biases of the recurrent neural network based, and adaptively learning to improve a convergence rate of the recurrent neural network. The method may further include finding ROT information, including a beginning and length of the ROT. Adaptively learning to improve a convergence rate of the recurrent neural network may include determining a momentum for the weights and biases of an output layer, and computing the root mean square of gradients for all of the weights and biases not in the output layer.
The recurrent neural network may be trained for phenome recognition using multi-class classification, and the method may further include updating, by the computing device, weights and biases through a backward pass further comprises computing signal error for all out nodes using a cross-entropy cost function for non-ROT regions and computing a soft target for all the frames of the ROT. In various embodiments, two cost functions are used to compute the signal error, the two cost functions including a connectionist temporal classification cost function used for a target node of the ROT and a blank node, and a cross entropy cost function used for all remaining nodes of the recurrent neural network.
The recurrent neural network may be trained for keyword detection using binary classification, including generating a spike when the keyword is detected. The method may further include calculating a signal output error for each network output based on the determination of whether the input frame is in the ROT, and updating the weights and biases during a backward pass based on the calculated signal output error. Updating the weights and biases during the backward pass may further include applying a cross-entropy cost function if the input frame is not in the ROT. Calculating the signal output error for each output may include obtaining ROT information for the input frame, including a length and beginning of the ROT, computing a soft target value using the ROT information, and computing the signal output error using the computed soft target value and network output value.
In various embodiments, a classification training system includes a neural network for use in a classification of input data, a training dataset providing segmented labeled training examples, and a classification training module for training the neural network using the segmented labeled training data, the classification training module comprising a forward pass processing module, and a backward pass processing module. The forward pass processing module may be operable to train the neural network by generating neural network outputs for the training data using current value of weights and bias for the neural network. The backward pass processing module may be configured to train the neural network by updating the weights and biases by passing backward through generated neural network outputs, determining whether an input frame is in a Region of Target (ROT), and adaptively learning to improve a convergence rate of the neural network. In some embodiments, adaptively learning to improve a convergence rate of the neural network includes determining a momentum for the weights and biases of an output layer, and computing a root mean square of gradients for all of the weights and biases other than in the output layer.
In various embodiments, the weights and biases are updated by computing signal error for all out nodes using a cross-entropy cost function for non-ROT regions. The weights and biases may be updated by improving the convergence rate by using an adaptive learning rate algorithm. The training system may further include a memory storing an ROT table, and the backward pass module may include an ROT information module operable to find a beginning and a length of the ROT using the ROT table. In one embodiment, the backward pass module further includes a soft target module operable for computing a soft target for all the frames of the ROT. In some embodiments, two cost functions are used to compute the signal error, and the two cost functions include a CTC cost function used for target node of the ROT and a blank node and a cross entropy cost function used for all remaining nodes.
In various embodiments, the classification training module may be operable to train for phoneme recognition using multi-class classification or train for keyword spotting using binary classification.
The scope of the disclosure is defined by the claims, which are incorporated into this section by reference. A more complete understanding of embodiments of the present disclosure will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.
Aspects of the disclosure and their advantages can be better understood with reference to the following drawings and the detailed description that follows. The included drawings are for illustrative purposes and serve only to provide examples of possible systems and methods for the disclosed methods and systems. These drawings in no way limit any changes in form and detail that may be made to that which is disclosed by one skilled in the art without departing from the spirit and scope of this disclosure.
In accordance with various embodiments of the present disclosure, systems and methods for training a recurrent neural network are disclosed, including systems and methods for binary classification, such as keyword spotting, and multi-class classification, such as phoneme recognition.
Binary Classification
In various embodiments disclosed herein, a novel Connectionist Temporal Classification (CTC) algorithm is proposed to train a neural network, such as a Long Short Term Memory (LSTM) network, for binary classification, such as keyword spotting, when pre-segmented labeled data is available. One goal of this binary classification system is to generate a single spike when an event is detected, without requiring generation of multiple spikes in a period that the event is happening. For example, in keyword spotting task, a goal may be to generate one spike when the keyword is recognized. The proposed method disclosed herein provides relative effectiveness and computational efficiency over conventional CTC network training solutions for this type of binary classification task. In various embodiments, the methods disclosed herein provide faster convergence than conventional solutions.
For many binary classifications such as keyword spotting, connectionist temporal classification (CTC) training may be more desirable than the frame-wise training. For example, a trained system may be expected to have only one spike anywhere during the keyword duration indicating the detection of the keyword. In that case, the system does not need to produce multiple spikes at all the frames of the keyword. This leads to a system training that provides better performance. See, e.g., Santiago Fernandez, Alex Graves and Jurgen Schmidhuber, “An application of recurrent neural networks to discriminative keyword spotting,” Proceedings of ICANN. Porto. pp 220-229 (2007), which is hereby incorporated by reference in its entirety.
A CTC network may be designed to maximize the probability of having correct labelling over all possible labelling. On the other hand, frame-wise training is designed to force the network to produce spikes at all the frames of the keyword. The frame-wise training may be sensitive to the alignment of training labeled data, so the training data should be correctly labeled consistently across all data. However, this requirement is not essential for CTC training since it removes the need for pre-segmented training data. Finally, a network using a frame-wise algorithm may be trained to make a series of independent label classifications. This implies that the network output will be post processed to give the final label sequence.
Although CTC training has several advantages over frame-wise training for binary classification, the conventional training algorithm of CTC training is not computationally efficient and it has some problems and limitations such as an underflow problem for the computation and it can also have undesirable latency. In the present disclosure, a solution for training a neural network for binary classification is proposed that overcomes many of the aforementioned limitations and provides better performance. One difference between the proposed method and conventional solutions for training a CTC network is that pre-segmented labeled data is used in various embodiments of the solution disclosed herein. This information is used to realize a more computational efficient algorithm for training and it can also lead to better performance.
Among conventional approaches, a common solution for the task of binary classification such as keyword spotting is to use Hidden Markov Models (HMM). One drawback of HMM based systems is that they model the garbage (i.e. non-keyword) parts of the signal as well as the keywords themselves. A structure flexible enough to model all possible garbage words is likely to be able to model the keywords as well. However, HMM modeling solutions are characterized by suboptimal convergence of the Expectation Maximization (EM) algorithm to local maxima, the assumption of conditional independence of the observations, and the fact that HMMs do not directly maximize the keyword detection rate. It is recognized that a recurrent neural network can achieve better performance for this task. Among the recurrent networks, LSTM or bidirectional LSTM (BLSTM) networks may be used to improve context modeling to build a better keyword spotting system.
Conventional methods of using an LSTM network can be divided in two categories. The first category uses the LSTM network to provide the higher level features for a classifier like Dynamic Bayesian Network (DBN) to perform the keyword spotting. Thus, the LSTM network combines the flexibility of the classifier like DBN architecture with the ability of a BLSTM network to capture long-range time dependencies. This category of methods has the flexibility of defining new keywords during operation, with no need to retrain the network and so it does not require excessive amounts of training data for large vocabularies. However, such systems typically have high complexity due to having a big network, such as a large BLSTM network, to provide higher level features such as phoneme recognitions for a keyword spotting task. Therefore, these methods are not practical solutions for many applications, such as mobile phones and other low power devices.
The second category exploits LSTM in combination with Connectionist Temporal Classification (CTC) which is a discriminative system based on recurrent neural networks that use information from long time spans to estimate keyword probabilities. In a keyword spotting task, for example, this method attempts to model full keywords in the sequence data stream, while the first approach is based on sub-word units (phonemes). Unlike the methods in the first category, these methods lack flexibility to new keyword selection in a keyword spotting task and may require more keyword samples in the training data. One advantage of these methods is their efficiency and low complexity as compared to the first category of approaches. These methods are more practical for low power devices where computational complexity and memory consumption are a major bottleneck.
The conventional solutions using LSTM based on CTC have some limitations. First, a dynamic programming algorithm, similar to the forward backward algorithm for HMMs is used in the CTC network which increases the computational complexity. Second, unsegmented input data is used to train the CTC network and so there is insufficient control on the latency of the system. This can be a problem for the applications where latency is a concern. Third, the training convergence rate is slower compared to frame-wise training. Finally, for the CTC training of both unidirectional and bidirectional LSTM, conventional approaches unroll the network by the length of the input sequence, which can consume a large amount of memory when the sequence is very long. Also, this “full unrolling” makes a parallel training with multiple sequences inefficient on shared memory models such as graphics processing units (GPUs), since the length of training sequences is usually not uniform, and thus a load imbalance problem may occur. In addition, the CTC algorithm may require full unrolling for the backward variable propagation, which starts from the end of the sequence.
The embodiments described herein have several advantages over conventional methods utilizing LSTM for binary classification. For example, a conventional CTC network (such as proposed in Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J., “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets,” Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh-Pa. (U.S.A.) (2006), which is hereby incorporated by reference) computes two variables, namely the forward and backward variables, recursively for the entire input sequence, which increases the computational complexity of the method because of its high number of multiplications especially with small valued numbers ranging between zero and one. The algorithms disclosured herein have low computational complexity and are well designed for the application of binary classification to train a recurrent neural network, such as an LSTM network, with high computational efficiently.
Another advantage is that in a binary classification such as a keyword spotting task it is desired to build a system to predict the occurrence of the keyword towards the ending of the occurrence of the keyword. Otherwise, the system can have high false-detections for the case of non-keywords containing part of the keyword (e.g. the beginning of a non-keyword is the same as the beginning of the keyword). Conventional CTC networks, such as proposed in the preceding reference, cannot guarantee that the trained system will predict the keyword towards the ending of its occurrence since the network is forced to maximize the correct output labeling regardless of the alignment. In contrast, the methods proposed herein do not have this limitation; a trained network using the disclosed methods can be forced to predict the true keyword towards the ending of the keyword.
A third difference is that existing methods utilizing a CTC network have relatively high memory requirements during training since they require obtaining the network's output for the entire input sequence as well as recursively computing the forward and backward variables of the same length of input sequence. For the task of keyword spotting, for example, when the input sequence is very long the aforementioned conventional method is not memory efficient. In contrast, the methods disclosed herein are memory efficient since some embodiments update the network frame-wise for non-keyword parts of the input sequence and concentrate on a particular region which is called Region Of Target (ROT) that can include the tail of the keyword. Another advantage over conventional approaches is that some embodiments disclosed herein use a most probable path to predict the keyword similarly to traditional frame-wise training.
Referring to
An embodiment of the backward pass process 200 is illustrated in
In various embodiments, a binary classification network will have two output nodes, and the ground truth label would be either zero or one. One goal is to train a recurrent neural network to generate outputs which are one in the ROT and zero anywhere else. But in some application such as keyword spotting, it may be sufficient to produce only one spike somewhere in the ROT and be zero anywhere else. For example, in a keyword spotting application, it is often desired to have a system that can detect the keyword by producing one spike close to the end of the keyword. In such a system, the ROT can be defined to cover the entire keyword (or at least the end of the keyword in some embodiments). In various embodiments, a goal is to build a system that can produce a spike (output close to one) anywhere in the ROT and generate zero anywhere else.
For example,
In some embodiments, one goal is to force the network to produce output close to one somewhere during the ROT from nr to nr+N−1 and to be zero from n0 to nr as well as from nr+N−1 to nL−1. To do so, an LSTM network may be used with two outputs having softmax non-linear function for the two output nodes. If we assume that the two outputs of the network before applying the non-linear function are zn(1) and zn(2) (where n is the time index), then the final outputs of the network (yn(1) and yn(2)) using the softmax non-linear function can be obtained as follows:
To train the network, a cross-entropy cost function is used. Assume that tn is the ground truth (target value) at frame n, then the cross entropy cost function for the two outputs may be defined as follows:
C(yn(i),tn)=−(tn log(yn(1))+(1−tn)log(yn(2))) i=1,2
yn(2)=1−yn(1) (2)
As the output of the network (yn(1)) has value between zero and one, it can be treated as the probability of being in ROT at frame n. In other words, this can be translated as follows:
yn(1)→probability of being in ROT at frame n
yn(2)=1−yn(1)→probability of being in non-ROT at frame n
In one embodiment, the cross entropy cost function maximizes the logarithm of the probability at frame n.
In order to train the network, which includes updating the weights and biases, a signal error (δn(i), i=1, 2) received at each node of the output layer (i.e., zn(1) and zn(2)) is computed. The signal error is propagated backwards through the unfolded net, using the standard BPTT as described above. The cost function derivative may be calculated as follows, which shows how to compute the derivative for the first node, zn(1). The derivative for the second one, zn(2), can be calculated similarly:
The signal error using (3) can be obtained as follows:
δn(1)=tn−yn(1)
δn(2)=−δn(1) (4)
As it is mentioned above, in various embodiments a system to train a recurrent neural network, such as an LSTM network, for each sequence of training data has two parts: 1) a forward pass in which the network outputs for all the frames of the training sequence are generated using the current values of weights and biases of the network; and 2) backward pass in which updated weights and biases are computed and the network is updated backward in time. To compute the update of the weights and biases, the signal error as discussed above is computed and then the signal error is propagated backwards through the unfolded net from the output layer to the input layer using the standard BPTT. As illustrated in the embodiment of
For non-ROT frames (step 208), the cross-entropy cost function in (2) is equal to maximizing the logarithm of the probability of being in non-ROT at each frame. This is aligned with a goal of having zero outputs for all the frames of non-ROT. The error signal in this region can be computed using (4) as
δn(1)=−yn(1)
δn(2)=−δn(1) (5)
From now on, we drop n to show the frame index and so the network output at the first node for ROT of
The cost function in (2) is revised as
C(yn(i),tn)=−log(PROT) (7)
Similar to (4), the signal error can be computed as
Instead of having tn which is the hard target value and equal to one in the ROT, tnsoft which is the soft target value is used in (8).
In various embodiments, calculating the PROT or pn includes multiplication of small valued numbers ranging between zero and one. To avoid the problem of underflow, the multiplication may be carried in logarithmic domain in which multiplication is replaced with addition. An exemplary embodiment of a procedure to compute tnsoft is set forth below:
Now the soft target at n-th frame can be computed using the following step:
It has been observed that the network training may converge more slowly than desired for certain implementations. To improve the performance and increase training convergence rate, an adaptive learning rate algorithm may be utilized (see step 212 in
where m=0.9 is the momentum and μ=1e−3 is the learning rate. θlowweight and θupweight are the lower bound and upper bound for the update of the weights (or the biases). δweight(t) is the error signal which is received at the corresponding weights (or biases) using the standard BPTT. For example, for bias of the first node of the output layer, δweight(t) equals to δn(1) for n-th frame. As shown in equation (10), the weights (or the biases) at each epoch are updated using update value. The following rule may be used to update other weights and biases of the network: initialization for the first epoch→E(0)=0
where ρ=0.95 is a smoothing factor and ε=1e−6 is a fixed constant to avoid infinity. The other parameters may be similar to what is discussed above.
In some embodiments, the binary classification system 600 can include an analog-to-digital converter 615 that converts the analog audio signals received from the audio inputting devices 605 into digital audio signals and sends the digital audio signals to processor 620, which can be a micro-controller, a digital signal processor (DSP), or other processing components, for controlling and facilitating the operations of the binary classification system 600, including controlling communications with internal and external devices. The binary classification system 600 may also include a digital-to-analog converter 650 that converts digital audio signals generated by the different modules and components of the binary classification system 600 to analog signals before transmitting the analog signals to the one or more audio outputting devices 610.
The binary classification system 600 includes one or more communication channels such as a bus for facilitating communication of data between various components of the binary classification system 600. Components may include device modules 625 for providing device operation and functionality, which may include input/output components 626 such as a touch screen, keyboard, mouse, etc., a display 627 such as a monitor, a memory 628 such as RAM, solid state drive, disk drive, database, etc., and a communications interface 629. In some embodiments, the communications interface 629 may include a network interface (e.g., Ethernet interface) or a wireless transceiver for enabling the binary classification system to communicate with remote devices over a network 632. Remote devices may include user devices 630 (e.g., household appliances, other user electronics, etc.), or a web server 635 that is communicatively coupled with a media storage 637.
The binary classification system 600 also includes a binary classification module 645 programmed to use the neural network that was trained and updated by the binary classification training system 500. The binary classification module includes the forward pass processing module 644. In one embodiment, Forward Pass Processing Module 644 is programmed to perform binary classification and prediction on audio input data received from audio inputting devices 605. In various embodiments, binary classification module 645 may include an automatic speech recognition (ASR) module providing voice command processing, an image search and classification system, an object detection system, medical diagnostic module, or other application configured for use with the neural network as described herein. As mentioned, for a given set of audio input data, the forward pass processing module 644 can produce either one of only two possible responses (e.g., a positive response or a negative response). Preferably, the forward pass processing module 644 can generate a response based on a neural network in real-time or close to real-time of the audio input. In some embodiments, the binary classification module 645 can be programmed to output the response as an audio sound via the digital-to-analog converter 650 and the audio outputting devices 610. Instead of or in addition to producing and audio signal, the binary classification module 645 can be programmed to send a signal to an external device (e.g., to initiate an action or a transaction through the external device) based on the generated response. For example, the binary classification system 600 can be part of or communicatively coupled with a smart home system, and send a signal to a user device (e.g., a household appliance) via a network 632 (e.g., a local area network within the user's residence) based on the generated response (e.g., sending a signal to turn on the television based on the response generated by an audio input from the user). In another example, the binary classification system 600 can initiate a transaction with a web server 635 over the Internet based on the generated response (e.g., sending a signal to the web server 635 to purchase a movie stored in media storage 637 based on the response generated by an audio input from the user). The binary classification system disclosed herein is not limited to processing audio signals, but can be used to train a neural network to process different input (e.g., image data, video data, etc.) as well.
Multi-Class Classification
In various embodiments, a novel Connectionist Temporal Classification (CTC) algorithm is proposed to train a recurrent neural network, such as an LSTM network, for multi-class classification, such as phoneme recognition.
Conventional recurrent neural networks (RNNs) can use internal memory to process arbitrary sequences of inputs. This makes RNNs suitable for tasks such as speech recognition since RNNs are able to model both the acoustic and temporal pattern of the input sequences. One approach for temporal classification with RNNs uses a Connectionist Temporal Classification (CTC) network, such as described in Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J., “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets,” Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh-Pa. (U.S.A.) (2006) (hereinafter, “Graves, et al.”), which is incorporated herein by reference in its entirety. One idea behind CTC is that instead of generating a label as output from the neural network, a probability distribution is generated at every time step. This probability distribution may then be decoded into a maximum likelihood label. The network is then trained by creating an objective function that coerces the maximum likelihood decoding for a given sequence to correspond to a desired label.
Unlike the approach described above using RNN combined with HMM, CTC network models all aspects of the sequence with a single RNN, and does not require the addition of an HMM to model the temporal pattern of the input sequence. The RNN may be trained directly for temporal classification tasks, which can be achieved by allowing the network to make label predictions at any point in the input sequence provided the overall sequence of labels is correct. Because CTC directly estimates the probabilities of the complete label sequences, external post-processing is not required to use the network as a temporal classifier.
Conventional systems, such as the system described in Graves et al., may include frame-wise and CTC networks classifying a speech signal. The system tracks probabilities of observing phonemes at particular times. The CTC network predicts the sequence of phonemes (typically as a series of spikes, separated by ‘blanks’, or null predictions), while the frame-wise network attempts to align the sequence of phonemes with temporal segmentation. The frame-wise network may receive an error or may misalign the segment boundaries, even if it predicts the correct phoneme. When one phoneme always occurs beside another, CTC tends to predict them together in a double spike. The choice of labeling can be read directly from the CTC outputs (follow the spikes), whereas the predictions of the frame-wise network must be post-processed before use.
Conventional CTC techniques for use in end-to-end Automatic Speech Recognition (ASR) face some obstacles such as the challenge of incorporating the lexicons and language models into decoding. It has been shown that combining the CTC network with another classifier can improve the performance of the ASR. For example, RNNs for Large Vocabulary Conversational Speech Recognition (LVCSR) trained with CTC can be improved with level minimum Bayes risk (sMBR) sequence training criterion and approaches the state-of-the-art performance. Despite the promising results obtained using the CTC technique, conventional approaches have several limitations including high computational complexity, unknown latency, and high amount of memory usage which is required to perform the forward-backward algorithm especially when the sequence of training is long.
Another approach is described in A. Graves, “Sequence transduction with recurrent neural networks,” in ICML Representation Learning Workshop, 2012, which is incorporated herein by reference in its entirety. As disclosed, the RNN Transducer approach is an extension of the CTC algorithm. Unlike the CTC, which can be seen as an acoustic-only model, the RNN Transducer has another RNN that acts as a language model. Similar to the CTC, the probability of observing an output sequence for a given input is computed using the forward-backward algorithm and has similar limitations as other conventional approaches.
Conventional systems, such as the systems described herein, have several limitations and disadvantages for practical usage, which are overcome by the systems and methods disclosed herein. First, the CTC network of Graves does not use pre-segmented labeled training data. The embodiments described herein make use of all the information available from the training data to improve the performance of training. Thus, the present disclosure is suitable for speech recognition applications since pre-segmented labeled training data, such as the TIMIT dataset, is widely available. The embodiments disclosed here include solutions which utilize the information of pre-segmented labeled training data to improve the performance. While some conventional algorithms that use labeled training data may be sensitive to alignment errors, the embodiments disclosed herein are not sensitive to the exact alignment of the labels of the input data sequence. As a result, the present disclosure provides a good solution to take the advantage of segmented data even where the labeling alignments are not exactly correct.
Another disadvantage of conventional approaches discussed herein is that the recursion algorithm for computing the forward and backward variables will eventually lead to underflows on a digital computer. Two well-known methods to avoid underflow are to either work in the log scale or to rescale the forward and backward variables. However, these two methods are less robust and can fail for very long sequences. The present disclosure includes embodiments that do not have this limitation, limiting computation on a Region of Target (ROT), which is slightly longer than the duration of a phoneme for the speech recognition task. The recursive computations of forward and backward variables further adds computational complexity to the CTC network. The system disclosed herein includes a more computationally efficient approach which is more suitable for training a large data sizes.
In many approaches, the memory requirements for training the CTC network is directly related to the input sequence length. This is because for each input sequence the forward pass will be performed to obtain the network's outputs for the entire sequence based on the network's outputs. Also forward and backward variables of the same length as the input sequence are computed. However, in the present disclosure, after computing the forward pass for the entire sequence, it computes the soft targets for the ROT, which has much shorter length than the entire sequence. Therefore, the method proposed herein is more suitable for practical applications especially when parallel training is conducted.
The CTC network proposed in Graves interprets the network outputs as a probability distribution over all possible label sequences, conditioned on a given input sequence. Given this distribution, an objective function is derived to directly maximize the probabilities of the correct labeling. In contrast, the methods disclosed herein are designed so that the most probable path will correspond to the most probable labeling. The decoder of the method is relatively simple and highly computationally efficient compared to conventional approaches.
For tasks where segmentation is required (e.g. protein secondary structure prediction), it would be problematic to use a conventional CTC network such as described herein since there is no guarantee that the network prediction would be aligned with the corresponding part of the sequence. On the other hand, the method disclosed herein can solve this problem since it would preserve the alignment of the network prediction.
Finally, there is no control on the latency of the conventional CTC algorithms discussed above, which may result in a trained system having unreasonably high latency. In contrast, the methods disclosed herein define a Region of Target (ROT) in a way that avoids this latency problem.
Referring to
In the Backward Pass 800 (
In various embodiments, the backward pass includes the following steps. In step 810, at each frame, the Region Of Target (ROT) is determined, using information retrieved from the ROT table 820. The beginning and the end of this region is used to compute the signal error. The soft target is computed in step 812 using the ROT information from step 810 and the network outputs. In step 814, the signal error is computed based on the network outputs and the estimated soft target. Finally, an adaptive learning rate algorithm is used to update the weights and biases of the network in step 816.
In the present embodiment, it is assumed that the total number of target labels (total number of phonemes for speech recognition task) is M and the m-th unnormalized output of the network before applying the non-linear function (here it is softmax function) is zn(m) (n is the frame index), then the final outputs of the network can be obtained as follows:
For a phoneme recognition application, each frame belongs to one of M classes of phonemes. Assume tn(m) is the target value of the network at frame n for m-th output node of the network, then this value is one when frame n belongs to m-th class of phoneme and it would be zero for all other nodes. For frame-wise training, the cross-entropy cost function may be used for training the network. The cross entropy cost function is defined as follows:
In order to train the network and update the weights and biases, the signal error (δn(m)) is computed which is the error that is received at each node of the output layer before applying the nonlinear function. The cost function derivative is calculated as follows (the derivative is only computed for the first node, zn(1) and the derivative for others can be done similarly):
By using equations (12)-(14), the signal error at the output nodes can be obtained as follows:
In various embodiments, frame-wise training using (15) may force the network to predict the phoneme in the entire duration of the phoneme for all frames. It is desirable to train the network to predict a phoneme after it sees the entire duration of the phoneme including the transitions, because the phonemes may have similar temporal-spectral characteristic at the initial parts but unique characteristic when considering the entire duration of the phoneme. As a result, it is often desirable to force the network to predict a phoneme almost at the end of it. The frame-wise training using (15) may also force the network to have the outputs aligned to the ground-truth labeling. However, this alignment may not be correct due to the mistakes in the ground-truth labeling and also the exact correct labeling is not always available.
Next, changing the cost function in (13) and the signal error in (15) to obtain the proposed CTC algorithm that can solve the problems described above will be described. As previously explained, the present embodiment trains the network (which may be an LSTM network) for each sequence of training data with two main steps: 1) a Forward Pass through the training sequence in which network outputs for each of the frames of the training sequence are generated using the current values of weights and biases of the network; and 2) a Backward Pass in which the update of the weights and biases are computed and the network is updated backward in time. To compute the update of the weights and biases, the signal error as discussed above is computed and then the signal error is propagated backwards through the unfolded network from the output layer to the input layer using the standard BPTT. As illustrated in
Find the ROT Information for n-th Frame (Step 810,
Assume that the number of classes or phonemes is M, then the proposed CTC algorithm has M+1 output nodes for the output layer of the LSTM network. The one additional node is called a blank node. In the present embodiment, the algorithm to compute the signal error of each frame is based on a region that is called Region Of Target (ROT). This region is defined as a region that includes all the frames corresponding to the target label of that frame plus some additional frames near the two boundaries (on the left and right) sufficient to include the transitions.
An example of ROT labeling is illustrated in
Compute the Soft Target (Step 812,
As mentioned above with respect to step 810, the number of output nodes is M+1. Thus, the softmax outputs in (12) are modified as:
In one embodiment, a goal is to force the network to predict the correct label (here “sh”) one time at any frame during the ROT (ROT frames include nr−b<frame index<nr+N−1+f) and set the other frames of the ROT as the blank label. In this example, “sh” is the target node which is assumed to be the k-th node of the output layer. From now on, we drop n to show the frame index and so the network output at k-th node for ROT of
The soft target of ROT at n-th frame (tnROT) is defined as follows:
In the present embodiment, calculating PROT and pn requires some multiplications of small valued numbers ranging between zero and one, which may lead to underflow. To address this problem, in one embodiment the multiplication is carried in logarithmic domain (in the logarithmic domain, multiplication is replaced by addition).
For any ROT compute PROT and pn−r+b (r−b≤n≤r+N−1+f) one time and keep the results k is assumed to be the target node in the ROT
Now the soft target at n-th frame can be computed using the following steps.
Compute Signal Error for all the Output Nodes (Step 814,
In this section, modifications to the cost function of (13) and the signal error of (15) are discussed in accordance with an embodiment of the invention are discussed. In one embodiment, the cost function includes two different cost functions for two different sets of output nodes of the network. Referring to the example in
Signal Error for all Other Nodes Except the Target Node and Blank Node
In the present embodiment, the cost function for all other nodes except the target node and the blank node is cross entropy similar to equation (13) and so the signal error for these nodes can be computed using equation (15). As the target value for all these nodes is zero (tn(m)=0 if 1≤m≤M and m≠k), the signal error for these nodes can be computed as:
δn(m)=−yn(m) if 1≤m≤M and m≠k (18)
CTC Cost Function and Signal Error for Target Node and Blank Node
The proposed CTC cost function is defined as follows:
C(yn(m),tn(m))=−log(PROT) (19)
Similar to (15), the derivative for the k-th node with respect to zn(k) can be obtained as:
where δn(k) is the signal error for k-th node at n-th frame for the ROT as it depicted in
δn(M+1)=(1−tnsoft−yn(M+1)) (21)
Exemplary CTC Cost Function if the ROT has Overlap with Adjacent ROT
In this section, an exemplary method in accordance with one embodiment is described in case there is overlap between adjacent ROTs. If the n-th frame falls in the non-overlap region, then the above method can be used to estimate the signal error for all the nodes. However, if the n-th frame falls in the overlap region of two ROTs, then the probability of having one-time prediction during all the two ROTs is considered. It is assumed that k′-th node of the network belongs to “ix” in
Similar to (17), the soft target of ROT1 (tnsoft1) and the soft target of ROT2 (tnsoft2) are defined as follows:
Again calculating PROT2, PROT1 and pi, pl includes some multiplications of small valued numbers ranging between zero and one, which may lead to underflow. To address this problem the multiplication is carried in logarithmic domain as it is explained before.
Signal Error for all Nodes
As it is explained before, the signal error for all the nodes except the target nodes (here “sh” and “ix”) are computed according to (18). By minimizing the CTC cost function (−log(PROT
δn(k)=tnsoft1−yn(k)(1+tnsoft2)
δn(k′)=tnsoft2−yn(k′)(1+tnsoft1) (24)
The signal error for blank node can be computed as follows.
In (22), there is no condition to force the network to spike “ix” after “sh” in the overlap region of
Proposed Adaptive Learning Rate (Step 816,
The training the network of the present embodiment may converge too slowly for many applications. In order to improve the performance and increase the training convergence rate, an adaptive learning rate algorithm is utilized. In one embodiment, two different methods are used to update the weights and biases of the network. The weights and biases connected to the output layer are updated according to the following rule for t-th epoch:
where m=0.9 is the momentum and μ=1e−3 is the learning rate. θlowweight and θupweight are the lower bound and upper bound for the update of the weights (or the biases). δweight(t) is the error signal which is received at the corresponding weights (or biases) using the standard BPTT. For example, for bias of the first node of the output layer, δweight(t) equals to δn(1) for n-th frame. As shown in (26), the weights (or the biases) at each epoch are updated using update value. For other weights and biases of the LSTM network, the following rule may be used to update them in accordance with one embodiment:
initialization for the first epoch→E(0)=0
where ρ=0.95 is a smoothing factor and ε=1e−6 is a fixed constant to avoid infinity. The other parameters are similar to what is discussed above.
Components may include device modules 1125 for providing device operation and functionality, which may include input/output components 1126 such as a touch screen, keyboard, mouse, etc., a display 1127 such as a monitor, a memory 1128 such as RAM, solid state drive, disk drive, database, etc., and a communications interface 1129. In some embodiments, the communications interface 1129 may include a network interface (e.g., Ethernet interface) or a wireless transceiver for enabling the classification system to communicate with remote devices over a network. In operation, training of the neural network is performed by classification training system 1100 offline and the trained model including the weights and biases of the neural network will be stored in Classification system 1200.
In some embodiments, the classification system 1200 can include an analog-to-digital converter 1215 that converts the analog audio signals received from the audio inputting devices 1205 into digital audio signals and sends the digital audio signals to processor 1220, which can be a micro-controller, a digital signal processor (DSP), or other processing components, for controlling and facilitating the operations of the classification system 1200, including controlling communications with internal and external devices. The classification system 1200 may also include a digital-to-analog converter 1250 that converts digital audio signals generated by the different modules and components of the classification system 1200 to analog signals before transmitting the analog signals to the one or more audio outputting devices 1210.
The classification system 1200 includes one or more communication channels such as a bus for facilitating communication of data between various components of the classification system 1200. Components may include device modules 1225 for providing device operation and functionality, which may include input/output components 1226 such as a touch screen, keyboard, mouse, etc., a display 1227 such as a monitor, a memory 1228 such as RAM, solid state drive, disk drive, database, etc., and a communications interface 1229. In some embodiments, the communications interface 1229 may include a network interface (e.g., Ethernet interface) or a wireless transceiver for enabling the classification system to communicate with remote devices over a network 1232. Remote devices may include user devices 1230 (e.g., household appliances, other user electronics, etc.), or a web server 1235 that is communicatively coupled with a media storage 1237.
The classification system 1200 also includes a classification module 1245 programmed to use the neural network that was trained and updated by the classification training system 1100. The classification module includes the forward pass processing module 1244. In one embodiment, Forward Pass Processing Module 1244 is programmed to perform classification and prediction on audio input data received from audio inputting devices 1205. In various embodiments, classification module 1245 may include other parts of the automatic speech recognition (ASR) module such as language processing providing voice command processing, an image search and classification system, an object detection system, or other application configured for use with the neural network as described herein. The forward pass processing module 1244 can generate a response based on a recurrent neural network in real-time or close to real-time of the audio input. In some embodiments, the classification module 1245 can be programmed to output the response as an audio sound via the digital-to-analog converter 1250 and the audio outputting devices 1210. Instead of or in addition to producing and audio signal, the classification module 1245 can be programmed to send a signal to an external device (e.g., to initiate an action or a transaction through the external device) based on the generated response. For example, the classification system 1200 can be part of or communicatively coupled with a smart home system, and send a signal to a user device (e.g., a household appliance) via a network 1232 (e.g., a local area network within the user's residence) based on the generated response (e.g., sending a signal to turn on the television based on the response generated by an audio input from the user). In another example, the classification system 1200 can initiate a transaction with a web server 1235 over the Internet based on the generated response (e.g., sending a signal to the web server 1235 to purchase a movie stored in media storage 1237 based on the response generated by an audio input from the user). The classification system disclosed herein is not limited to processing audio signals, but can be used to train a neural network to process different input (e.g., image data, video data, etc.) as well.
It should be noted that any language directed to a computer should be read to include any suitable combination of computing devices, including servers, interfaces, systems, databases, agents, peers, engines, modules, controllers, or other types of computing devices operating individually or collectively. One should appreciate the computing devices comprise a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (e.g., hard drive, solid state drive, RAM, flash, ROM, etc.). The software instructions preferably program the computing device to provide the roles, responsibilities, or other functionality as discussed above with respect to the disclosed apparatus. In especially preferred embodiments, the various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on Hypertext Transfer Protocol (HTTP), Time-Triggered Protocol (TTP), Hypertext Transfer Protocol Secure (HTTPS), Advanced Encryption Standard (AES), public-private key exchanges, web service application program interfaces (APIs), known financial transaction protocols, or other electronic information exchanging methods. Data exchanges preferably are conducted over a packet-switched network, the Internet, local area network (LAN), wide area network (WAN), virtual private network (VPN), or other type of packet switched network.
Where applicable, various embodiments provided by the present disclosure may be implemented using hardware, software; or combinations of hardware and software. Also, where applicable, the various hardware components and/or software components set forth herein may be combined into composite components comprising software, hardware, and/or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and/or software components set forth herein may be separated into sub-components comprising software, hardware, or both without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components and vice-versa.
Software, in accordance with the present disclosure, such as program code and/or data, may be stored on one or more computer readable mediums. It is also contemplated that software identified herein may be implemented using one or more general purpose or specific purpose computers and/or computer systems, networked and/or otherwise. Where applicable, the ordering of various steps described herein may be changed, combined into composite steps, and/or separated into sub-steps to provide features described herein.
The foregoing disclosure is not intended to limit the present disclosure to the precise forms or particular fields of use disclosed. As such, it is contemplated that various alternate embodiments and/or modifications to the present disclosure, whether explicitly described or implied herein, are possible in light of the disclosure. Having thus described embodiments of the present disclosure, persons of ordinary skill in the art will recognize that changes may be made in form and detail without departing from the scope of the present disclosure. Thus, the present disclosure is limited only by the claims.
Nesta, Francesco, Thormundsson, Trausti, Kaskari, Saeed Mosayyebpour
Patent | Priority | Assignee | Title |
11586988, | Jun 13 2018 | Fujitsu Limited | Method of knowledge transferring, information processing apparatus and storage medium |
11853884, | Feb 10 2017 | Synaptics Incorporated | Many or one detection classification systems and methods |
Patent | Priority | Assignee | Title |
10013640, | Dec 21 2015 | GOOGLE LLC | Object recognition from videos using recurrent neural networks |
10114554, | Jan 20 2015 | Intellectual Property Systems, LLC | Arrangements for storing more data in faster memory when using a hierarchical memory structure |
5155763, | Dec 11 1990 | International Business Machines Corp. | Look ahead method and apparatus for predictive dialing using a neural network |
5630023, | Oct 14 1993 | Ricoh Company, Limited | Signal processor |
6041299, | Mar 11 1997 | ATR Interpreting Telecommunications Research Laboratories | Apparatus for calculating a posterior probability of phoneme symbol, and speech recognition apparatus |
6470261, | Jul 31 1998 | CET Technologies PTE LTD | Automatic freeway incident detection system and method using artificial neural network and genetic algorithms |
6615382, | Aug 27 1999 | SAMSUNG ELECTRONICS CO , LTD | Method for controlling errors in link layer in wideband wireless communication and computer readable media therefor |
6728404, | Sep 12 1991 | FUJIFILM Corporation | Method for recognizing object images and learning method for neural networks |
7110526, | Oct 14 1998 | Wilmington Trust, National Association, as Administrative Agent | Neural network for controlling calls in a telephone switch |
8675839, | Jun 27 2012 | Noble Systems Corporation | Reporting unavailable trunk-related resources for a predictive dialer using a dynamic pacing algorithm |
9191304, | Aug 12 2013 | The United States of America as represented by the Secretary of the Navy | Reinforcement learning-based distributed network routing method utilizing integrated tracking and selective sweeping |
9723144, | Sep 20 2016 | Noble Systems Corporation | Utilizing predictive models to improve predictive dialer pacing capabilities |
20030115041, | |||
20030225786, | |||
20050216426, | |||
20050261898, | |||
20100198054, | |||
20110224975, | |||
20150381809, | |||
20160048766, | |||
20170005644, | |||
20170083829, | |||
20170148431, | |||
20170249551, | |||
20170344829, | |||
20170372197, | |||
20180203848, | |||
20180233130, | |||
20180253648, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Feb 12 2018 | Synaptics Incorporated | (assignment on the face of the patent) | / | |||
Aug 14 2019 | THORMUNDSSON, TRAUSTI | Synaptics Incorporated | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 051968 | /0607 | |
Oct 14 2019 | NESTA, FRANCESCO | Synaptics Incorporated | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 051968 | /0607 | |
Feb 14 2020 | Synaptics Incorporated | Wells Fargo Bank, National Association | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 051936 | /0103 | |
Feb 25 2020 | KASKARI, SAEED MOSAYYEBPOUR | Synaptics Incorporated | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 051968 | /0607 |
Date | Maintenance Fee Events |
Feb 12 2018 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Feb 21 2024 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Sep 01 2023 | 4 years fee payment window open |
Mar 01 2024 | 6 months grace period start (w surcharge) |
Sep 01 2024 | patent expiry (for year 4) |
Sep 01 2026 | 2 years to revive unintentionally abandoned end. (for year 4) |
Sep 01 2027 | 8 years fee payment window open |
Mar 01 2028 | 6 months grace period start (w surcharge) |
Sep 01 2028 | patent expiry (for year 8) |
Sep 01 2030 | 2 years to revive unintentionally abandoned end. (for year 8) |
Sep 01 2031 | 12 years fee payment window open |
Mar 01 2032 | 6 months grace period start (w surcharge) |
Sep 01 2032 | patent expiry (for year 12) |
Sep 01 2034 | 2 years to revive unintentionally abandoned end. (for year 12) |