An object detecting device for detecting an object in a given gradation image. A scaling section generates scaled images by scaling down a gradation image input from an image output section. A scanning section sequentially manipulates the scaled images and cutting out window images from them and a discriminator judges if each window image is an object or not. The discriminator includes a plurality of weak discriminators that are learned in a group by boosting and an adder for making a weighted majority decision from the outputs of the weak discriminators. Each of the weak discriminators outputs an estimate of the likelihood of a window image to be an object or not by using the difference of the luminance values between two pixels. The discriminator suspends the operation of computing estimates for a window image that is judged to be a non-object, using a threshold value that is learned in advance.
|
0. 41. A detecting method for detecting objects in an image comprised of pixels comprising the steps of:
determining inter-pixel differences, each a difference between luminance values of two arbitrary pixels at two different positions of the image, including non-adjacent pixels, and sequentially outputting estimates that the image includes an object, based on said inter-pixel differences;
learning in advance estimates of objects included in plural training images by making plural determinations equal to the difference of the luminance values of two pixels at arbitrarily selected two different positions in respective training images; and
using estimates of objects learned from group learning for determining whether the image includes an object according to a comparison of the outputted estimate to the learned estimates and suspending the outputting of estimates depending upon the value of an outputted estimate compared to a threshold value learned in advance.
0. 38. A detecting device for detecting objects in an image comprised of pixels comprising:
a discriminating unit configured to determine inter-pixel differences, each a difference between luminance values of two arbitrary pixels at two different positions of the image, including non-adjacent pixels, and sequentially outputting estimates that the image includes an object, based on said inter-pixel differences;
a group learning machine configured to provide learned estimates of objects included in plural training images, learned in advance, by making plural determinations equal to the difference of the luminance values of two pixels at arbitrarily selected two different positions in respective training images; and
a judging unit using estimates of objects learned from group learning for determining whether the image includes an object according to a comparison of the estimate outputted by the discriminating unit to the learned estimates and for suspending the outputting of estimates by the discriminating unit depending upon the value of an outputted estimate compared to a threshold value learned in advance.
0. 1. An object detecting device for detecting if a given grayscale image is an object, the device comprising:
a plurality of weak discriminating means for computing an estimate indicating that the grayscale image is an object or not according to a characteristic quantity that is equal to the difference of the luminance values of two pixels at two different positions; and
a discriminating means for judging if the grayscale image is an object according to the estimate computed by one of or the estimates computed by more than one of the plurality of weak discriminating means.
0. 2. The device according to
wherein the discriminating means computes the value of the weighted majority decision by multiplying each of the estimates by the reliability of the corresponding weak discriminating means obtained as a result of the learning and adding the products of the multiplications and judges if the grayscale image is an object according to the majority decision value.
0. 3. The device according to
wherein the plurality of discriminating means compute estimates sequentially; and
wherein the discriminating means sequentially updates the value of weighted majority decision each time when an estimate is computed and controls the object detecting operation of the device so as to judge if the computation of estimates is suspended according to the updated value of weighted majority decision.
0. 4. The device according to
wherein the discriminating means is configured to suspend the operation of computing estimates depending on if the value of weighted majority decision is smaller than a suspension threshold value; and
wherein the weak discriminating means are sequentially generated by ensemble learning, using a leaning sample of a plurality of grayscale images provided with respective correct answers telling if each of the grayscale images is an object;
wherein the suspension threshold value being the minimum value in the values of weighted majority decision updated by adding the weighted reliabilities to the respective estimates of the learning samples of the objects, as computed each time a weak discriminating means is generated in the learning session by the generated weak discriminating means.
0. 5. The device according to
wherein, if the minimum value in the values of the weighted majority decision obtained in the learning session is positive, 0 is selected as the suspension threshold value.
0. 6. The device according to
wherein each of the weak discriminating means decisively outputs its estimate by computing the estimate as binary value indicating if the grayscale image is an object depending on if the characteristic quantity is smaller than a predetermined threshold value.
0. 7. The device according to
wherein each of the weak discriminating means outputs the probability that the grayscale image is an object as computed on the basis of the characteristic quantity so as to decisively output its estimate.
0. 8. The object detecting device according to
0. 9. The object detecting device according to
0. 10. The object detecting device according to
0. 11. The object detecting device according to
0. 12. An object detecting method for detecting if a given grayscale image is an object, the method comprising:
a weak discriminating step of computing an estimate indicating that the grayscale image is an object or not according to a characteristic quantity that is equal to the difference of the luminance values of two pixels at two different positions that is learned in advance for each of a plurality of weak discriminators; and
a discriminating step of judging if the grayscale image is an object according to the estimate computed by one of or the estimates computed by more than one of the plurality of weak discriminators.
0. 13. The method according to
wherein the value of the weighted majority decision is computed by multiplying each of the estimates by the reliability of the corresponding weak discriminator obtained as a result of the learning and adding the products of the multiplications and it is judged if the grayscale image is an object according to the majority decision value in the discriminating step.
0. 14. The method according to
wherein estimates are computed sequentially by the plurality of weak discriminators in the weak discriminating step; and
the value of weighted majority decision is updated each time when an estimate is computed in the discriminating step;
the method further comprising:
a control step of judging if the computation of estimates is suspended according to the value of weighted majority decision updated in the discriminating step.
0. 15. An ensemble learning device for ensemble learning using learning samples of a plurality of grayscale images provided with respective correct answers telling if each of the grayscale images is an object, the device comprising:
a learning means for learning a plurality of weak discriminators for outputting an estimate indicating that the grayscale image is an object or not in a group, using a characteristic quantity that is equal to the difference of the luminance values of two pixels at arbitrarily selected two different positions as input; and
a combining means for selectively combining more than one weak discriminator from the plurality of weak discriminators according to a predetermined learning algorithm.
0. 16. The device according to
wherein the learning means includes:
a weak discriminator generating means for computing the characteristic quantity of each of the learning samples and generating the weak discriminators according to the respective characteristic quantities;
an error ratio computing means for computing the error ratio of judging each of the learning samples according to the data weight defined for the learning sample for the weak discriminators generated by the weak discriminator generating means;
a reliability computing means for computing the reliability of the weak discriminators according to the error ratio; and
a data weight computing means for updating the data weight relatively increase the weight of each learning sample that is discriminated as error by the weak discriminators;
wherein the weak discriminator generating means is capable of generating a new weak discriminator when the data weight is updated.
0. 17. The device according to
wherein the weak discriminator generating means computes characteristic quantities of a plurality of different types by repeating the process of computing a characteristic quantity for a plurality of times, generate a weak discriminator candidate for each characteristic quantity, computes the error ratio of judging each learning sample according to the data weight defined for the learning sample and select the weak discriminator candidate showing the lowest error ratio as weak discriminator.
0. 18. The device according to
wherein the weak discriminator generating means generates a weak discriminator candidate configured to judge if the grayscale image is an object depending on if the characteristic quantity of the grayscale image is greater than a predetermined threshold value.
0. 19. The device according to
wherein the weak discriminator generating means generates a weak discriminator candidate configured to output the probability that the grayscale image is an object according to the characteristic quantity.
0. 20. The device according to
a suspension threshold value storing means for storing the minimum value in the values of weighted majority decision, each being obtained as a result of that, each time the weak discriminator generating means generates a weak discriminator, the weak discriminator generating means computes an estimate for each learning sample that is an object by means of the weak discriminator and also computes the value of the weighted majority decision obtained by weighting the estimate with the reliability.
0. 21. An ensemble learning method of using learning samples of a plurality of grayscale images provided with respective correct answers telling if each of the grayscale images is an object, the device comprising:
a learning step of learning a plurality of weak discriminators for outputting an estimate indicating that the grayscale image is an object or not in a group, using a characteristic quantity that is equal to the difference of the luminance values of two pixels at arbitrarily selected two different positions as input; and
a combining step of selectively combining more than one weak discriminator from the plurality of weak discriminators according to a predetermined learning algorithm.
0. 22. The method according to
wherein the learning step is configured to repeat a series of steps including:
a weak discriminator generating step of for computing the characteristic quantity of each of the learning samples and generating the weak discriminators according to the respective characteristic quantities;
an error ratio computing step of computing the error ratio of judging each of the learning samples according to the data weight defined for the learning sample for the weak discriminators generated by the weak discriminator generating means;
a reliability computing step of computing the reliability of the weak discriminators according to the error ratio; and
a data weight computing step of updating the data weight so as to relatively increase the weight of each learning sample that is discriminated as error by the weak discriminators.
0. 23. The method according to
wherein characteristic quantities of a plurality of different types are computed by repeating the process of computing a characteristic quantity for a plurality of times and a weak discriminator candidate is generated for each characteristic quantity, while the error ratio of judging each learning sample is computed according to the data weight defined for the learning sample and the weak discriminator candidate showing the lowest error ratio is selected as weak discriminator in the weak discriminator generating step.
0. 24. The method according to
wherein a weak discriminator candidate configured to judge if the grayscale image is an object depending on if the characteristic quantity of the grayscale image is greater than a predetermined threshold value is generated in the weak discriminating generating step.
0. 25. The method according to
wherein a weak discriminator candidate configured to output the probability that the grayscale image is an object according to the characteristic quantity is generated in the weak discriminator generating step.
0. 26. The method according to
a suspension threshold value storing step of storing the minimum value in the values of weighted majority decision, each being obtained as a result of that, each time weak discriminating is generated in the weak discriminator generating step, an estimate for each learning sample that is an object is computed by means of the weak discriminator and the value of the weighted majority decision obtained by weighting the estimate with the reliability is also computed in the weak discriminator generating step.
0. 27. An object detecting device for cutting out a window image of a fixed size from a grayscale image and detecting if the grayscale image is an object, the device comprising:
a scale converting means for generating a scaled image by scaling up or down the size of the input grayscale image;
a window image scanning means for scanning the window of the fixed size out of the scaled image and cutting out a window image; and
an object detecting means for detecting if the given window image is an object;
the object detecting means having:
a plurality of weak discriminating means for computing an estimate indicating that the window image is an object according to a characteristic quantity that is equal to the difference of the luminance values of two pixels at two different positions that is learned in advance; and
a discriminating means for judging if the window image is an object according to the estimate computed by one of or the estimates computed by more than one of the plurality of weak discriminating means.
0. 28. The device according to
wherein the discriminating means computes the value of the weighted majority decision by multiplying the estimate or each of the estimates by the reliability of each of the weak discriminating means obtained as a result of the learning and adding it and judges if the grayscale image is an object according to the value of the majority decision.
0. 29. The device according to
wherein the plurality of weak discriminating means sequentially compute the estimates; and
the discriminating means sequentially updates the value of the weighted majority decision each time an estimate is computed and controls the estimate computing operation so as to suspend it or not according to the updated value of the weighted majority decision.
0. 30. The object detecting device according to
0. 31. The object detecting device according to
0. 32. The object detecting device according to
0. 33. The object detecting device according to
0. 34. An object detecting method for cutting out a window image of a fixed size from a grayscale image and detecting if the grayscale image is an object, the method comprising:
a scale converting step of generating a scaled image by scaling up or down the size of the input grayscale image;
a window image scanning step of scanning the window of the fixed size out of the scaled image and cutting out a window image; and
an object detecting step of for detecting if the given window image is an object;
the object detecting step having:
a weak discriminating step of computing an estimate indicating that the window image is an object or not according to a characteristic quantity that is equal to the difference of the luminance values of two pixels at two different positions that is learned in advance by each of a plurality of weak discriminators; and
a discriminating step of judging if the window image is an object according to the estimate computed by one of or the estimates computed by more than one of the plurality of weak discriminating means.
0. 35. The method according to
wherein the value of the weighted majority decision is computed by multiplying the estimate or each of the estimates by the reliability of each of the weak discriminators obtained as a result of the learning and adding it and it is judged if the grayscale image is an object according to the value of the majority decision in the discriminating step.
0. 36. The method according to
wherein the plurality of weak discriminators sequentially compute the estimates in the weak discriminating step; and
wherein the value of the weighted majority decision is sequentially updated each time an estimate is computed and
wherein the estimate computing operation is so controlled as to suspend it or not according to the updated value of the weighted majority decision in the discriminating step.
0. 37. An object detecting device for detecting if a given grayscale image is an object, the device comprising:
a plurality of weak discriminating units configured to compute an estimate indicating that the grayscale image is an object or not according to a characteristic quantity that is equal to the difference of the luminance values of two pixels at two different positions; and
a discriminating unit configured to judge if the grayscale image is an object according to the estimate computed by one of or the estimates computed by more than one of the plurality of weak discriminating units.
0. 39. The detecting device of claim 38, wherein the discriminating unit outputs an estimate of probability that the image includes an object.
0. 40. The detecting device of claim 38, wherein the discriminating unit comprises plural discriminators, each sequentially outputting an estimate of probability that the image includes an object, and the outputted estimate is made by combining the estimates of the plural discriminators.
0. 42. The method of claim 41, wherein the outputted estimate is an estimate of probability that the image includes an object.
0. 43. The method of claim 41, wherein the outputted estimate is a combination of plural determinations of differences between pixel values of said two arbitrary pixels.
|
inter-pixel difference characteristic: d=I1−I2 (2)
The ability of a weak discriminator depends on if its inter-pixel difference characteristic is used for detecting a face or not. Therefore, it is necessary to select a combination of pixel positions (to be also referred to as filter or weak hypothesis) contained in a cut out image so as to be used for weak discriminators.
For example, AdaBoost requires each weak discriminator to decisively output +1 (a object) or −1 (a non-object). Thus, in AdaBoost, a weak discriminator is generated by bisecting the inter-pixel difference characteristic at a pixel position, using one or more than one threshold values (+1 or −1).
In the case of the boosting algorithm of Real-AdaBoost or Gentle Boost, in which not a binary value but a continuous value (real number) is output to indicate the probability distribution of a learning sample, each weak discriminator outputs the probability telling if the input image is an object or not. Thus, the output of a weak discriminator may be decisive or in the form of probability. Firstly, weak discriminators of these two types will be discussed.
(3-1) Weak Discriminator Adapted to Output a Binary Value
A weak discriminator adapted to produce a decisive output makes a two class judgment on the object according to the inter-pixel difference characteristic. If the luminance values of two pixels located in the area of an image are I1 and I2 and the threshold value for judging if the image is an object or not by means of the inter-pixel difference characteristic is Th, it is possible to determine the class to which the image belongs depending on if it satisfies the requirement of formula (3) below or not.
[Formula 3]
I1−I2>Th (3)
While each weak discriminator is required to select two pixel positions and a threshold value for them, the method for selecting them will be described hereinafter. The determination of the threshold value as indicated by the above formula (3) is the most simple case. For determining a threshold value, two threshold values expressed by formula (4) or formula (5) below may be used.
[Formula 4]
Th1>I1−I2>Th2 (4)
[Formula 5]
I1−I2>Th1 and Th2>I2−I2 (5)
When the histogram shows a normal distribution curve for the non-object data as indicated by a broken line and also another normal distribution curve for the object data as indicated by a solid line in
When, on the other hand, the peaks of the two curves are found substantially at a same position but the distribution curves show different widths, it is possible to judge a window image to be an object or not by means of the above formula (4) or (5), using a value close to the upper limit value and a value close to the lower limit value of the inter-pixel difference characteristic of the distribution curve showing the smaller width.
While a weak discriminator is formed by determining an inter-pixel difference characteristic and one or two threshold values for it, it is necessary to select an inter-pixel difference characteristic that minimizes the error ratio of the judgment of the weak discriminator or maximizes the right judgment ratio. For instance, the threshold value(s) may be determined by selecting two pixel positions, determining a histogram for learning samples provided with correct answers as shown in
(3-2) Weak Discriminator for Outputting a Continuous Value
Weak discriminators that produce an output in the form of probability include those used in Real-AdaBoost and Gentle Boost. Unlike a weak discriminator adapted to solve a discrimination problem by means of a predetermined constant value (threshold value) and output a binary value (f(x)=1 or −1) as described above, a weak discriminator of this type outputs the degree of likelihood of an object for the input image typically in the form of a probability density function.
The probability output indicating the degree of likelihood (probability) of an object is expressed by function f(x) of formula (6) below, where Pp(x) is the probability density function of being an object of the learning sample and Pn(x) is the probability density function of being a non-object of the learning sample.
[Formula 6]
probability output of weak discriminator: f(x)=Pp(x)−Pn(x) (6)
The discriminating ability may be further improved by combining the above described estimation methods (discrimination methods) for use in ensemble learning. On the other hand, the processing speed can be improved by using only one of the methods.
This embodiment provides an advantage of being able to discriminate an object from a non-object at very high speed because it employs weak discriminators that use a very simple characteristic quantity (inter-pixel difference characteristic). When detecting an object that is a face, an excellent result of judgment can be obtained by using a threshold value that is determined by the method using the simplest formula (3) out of the above described discriminating methods for the inter-pixel difference characteristic. However, the selection of a discriminating method for the purpose of effectively exploiting weak discriminators may depend on the problem to be solved and hence an appropriate method may be used for selecting the threshold value(s). Depending on the problem, a characteristic quantity may be obtained not as the difference of the luminance values of two pixels but as the difference of the luminance values of more than two pixels or a combination of such differences.
(4) Suspension Threshold Value
Now, a suspension threshold value will be discussed. In a group learning machine using boosting, a window image is judged to be an object or not by way of a weighted majority decision that is the output of all the weak discriminators constituting the discriminator 5. The weighted majority decision is determined by sequentially adding the results (estimates) of discrimination of the weak discriminators. For example, if the number of weak discriminators is t (=1, . . . , K) and the weight (reliability) of majority decision that corresponds to each weak discriminator is αt, while the output of each weak discriminator is ft(x), the value of weighted majority decision F(x) in AdaBoost can be obtained by using formula (7) below.
Here, a technique different from the ordinary boosting algorithm is introduced into this embodiment. With this technique, the process of sequentially adding the results of discrimination of weak discriminators is suspended for a window image that can be judged to be obviously a non-object before the time when all the results of discrimination are obtained from the weak discriminators. To do this, a threshold value to be used for determining a suspension of discrimination or not is learnt in advance in the learning step. The threshold value to be used for determining a suspension of discrimination or not is referred to as suspension threshold value hereinafter.
Due to the use of a suspension threshold value, it is possible to suspend the operation of the weak discriminators for computing their estimates f(x) for each window image if it can be reliably estimated to be a non-object without using the outputs of all the weak discriminators. As a result, the volume of computational operations can be remarkably reduced if compared with an occasion where all the weak discriminators are used to make a weighted majority decision.
The suspension threshold value may be the minimum value that the weighted majority decision can take for the learning sample that indicates the object of detection in the labeled learning samples. The results of the discriminating operations of the weak discriminators for the window image are sequentially weighted and output in the discriminating step. In other words, as the value of the weighted majority decision is sequentially updated and each time the suspension threshold value is updated and hence the result of discriminating operation of a weak discriminator is output, the updated value of the weighted majority decision and the updated suspension threshold value are compared and the window image is judged to be a non-object when the updated value of the weighted majority decision undergoes the suspension threshold value. Then, the computational process may be suspended to consequently eliminate wasteful computations and further raise the speed of the discriminating process.
More specifically, the minimum value of the weighted majority decision, that is obtained when the learning sample Xj, which is an object, is used out of the learning samples xi(=xi through XN) is selected for the suspension threshold value RK for the output fK(x) of the K-th weak discriminator, which is defined by formula (8) below.
As seen from the formula (8), when the minimum value of the weighted majority decision of the learning samples xi through XJ, which are objects, exceeds 0, 0 is selected for the suspension threshold value RK. The minimum value of the weighted majority decision is made so as not to exceed 0 in AdaBoost that selects 0 as threshold value for discrimination. Therefore, the process of defining the threshold value may differ depending on the selected group learning technique. In the case of AdaBoost, the minimum value that all the data D1 through D4 that are obtained when an object is input as input image can take is selected for the suspension threshold as indicated by the thick line in
In this embodiment, with the arrangement of learning the suspension threshold value Rt(Ri through RK) each time a weak discriminator is generated, the estimates of a plurality of weak discriminators are sequentially output and the value of the weighted majority decision is sequentially updated. Then, the discriminating operations of the subsequent weak discriminators are omitted when the value undergoes the suspension threshold value as indicated by data D5 in
(5) Learning Method
Now, the learning method of the group learning machine 6 will be described. Images (training data) that are used as labeled learning samples (learning samples provided with correct answers) are manually prepared in advance as prerequisite for a pattern recognition problem of 2-class discrimination such as a problem of discriminating a face from a non-face in the given data. The learning samples include a group of images obtained by cutting out areas of an object to be detected and a group of random images obtained by cutting out areas of an unrelated object, which may be a landscape view.
A learning algorithm is applied on the basis of the learning samples to generate learning data that are used at the time of discriminating process. In this embodiment, the learning data to be used for the discriminating process include the following four sets of learning data that include the above described learning data.
Now, the algorithm for learning the four types of learning data (A) through (D) as listed above from the large number of learning samples as described above will be described.
(Step S0) Labeling of Learning Samples
Learning samples (xi, yi) that are labeled so as to show an object or non-object in advance are prepared in a manner as described above.
In the following description, the following notations are used.
In other words, xi denotes a characteristic vector formed by all the luminance values of the learning sample images and yi=−1 indicates a case where a learning sample is labeled as non-object, while yi=1 indicates a case where a learning sample is labeled as object.
(Step S1) Initialization of Data Weight
For boosting, the weights of learning samples (data weights) are differentiated in such a way that the date weight of a learning sample that is hard to discriminate is made relatively large. While the result of discrimination of a weak discriminator is used to compute the error ratio for evaluating the weak discriminator, the evaluation of a weak discriminator that made an error in discriminating a relatively difficult learning sample will become lower than the proper evaluation for the achieved discrimination ratio when the result of discrimination is multiplied by a data weight. While the data weight is sequentially updated by the method as will be described hereinafter, the data weight of the learning sample is firstly initialized. The data weights of the learning samples are initialized so as to make the weights of all the learning samples equal to a predetermined value. The data weight is defined by formula (9) below.
[formula 9]
data weight: D1,i=1/N (9)
In the above formula, the data weight D1,i indicates that it is the data weight of learning sample xi(=x1, through XN) at the number of times of repetition t=1 and N denotes the number of learning samples.
(Step S2 through S7) Repetition of Processing Operation
Then, the processing operation of Step S2 through S7 is repeated to generate a discriminator 5. The number of times of repetition of the processing operation t is made equal to t=1, 2, . . . , K. Each time the processing operation is repeated, a weak discriminator is generated and hence a pair of pixels and the inter-pixel difference characteristic for the positions of the pixels are leant. Therefore, as many weak discriminators as the number of times (K) of repetition of the processing operation are generated and a discriminator 5 is generated from the K weak discriminators. While hundreds to thousands of weak discriminators are normally generated as a result of repetition of the processing operation for hundreds to thousands times, the number of times of the processing operation (the number of the weak discriminators) t may be appropriately selected depending on the required level of discriminating ability and the problems (objects) to be discriminated.
(Step S2) Leaning of Weak Discriminators
Learning (generation) of weak discriminators takes place in Step S2 but the learning method to be used for it will be described in greater detail hereinafter. In this embodiment, a weak discriminator is generated each time the processing operation is repeated by means of the method that will be described hereinafter.
(Step S3) Computation of Weighted Error Ratio et
Then, the weighted error ratio of the weak discriminators generated in Step S2 is computed by using formula (10) below.
As shown in the above formula (10), the weighted error ratio et is obtained by adding the data weights of only the learning samples of which the results of discrimination of the weak discriminators are wrong (ft(xi).≠yi) out of all the learning samples. As pointed out above, the weighted error ratio et is such that it is made to show a large value when weak discriminators make an error in discriminating a learning sample having a large data weight Dt,i (a learning sample difficult to discriminate). The weighted error ratio et is smaller than 0.5 but the reason for it will be described hereinafter.
(Step S4) Computation of Weight of Weighted Majority Decision (reliability of weak discriminator)
Then, the reliability αt of the weight of weighted majority decision (to be referred to simply as reliability hereinafter) is computed by using formula (11) below on the basis of the weighted error ratio et as computed by means of the above formula (10). The weight of weighted majority decision indicates the reliability αt of the weak discriminator that is generated at the t-th time of repetition.
As clear from the above formula (11), a weak discriminator whose weighted error ratio et is small can acquire a large reliability αt.
(Step S5) Updating of Data Weights of Learning Samples
Then, the data weights Dt, i of the learning samples are updated by means of formula (12) below, using the reliabilities αt obtained by using the above formula (11). The data weights Dt, i are normalized ordinarily in such a way that the sum of adding them all is equal to 1. Formula (13) below is used to normalize the data weights Dt, i.
[formula 12]
date weight: Di+1,i=Di,1exp(−αiyifi(xi)) (12)
(Step S6) Computation of Suspension Threshold Value Rt
Then, as described above, the threshold value Rti for suspending the discriminating operation of the discriminating step is computed. The smallest one of the values of the weighted majority decision of the learning samples (positive learning samples) x1 through xJ and 0 that are objects is selected for the suspension threshold value Rt according to the above described formula (8). Note that the smallest value or 0 is selected for the suspension threshold value in the case of AdaBoost that is adapted to discriminating operations using 0 as threshold value. Anyway, the largest value that allows at least all the positive learning samples to pass is selected for the suspension threshold value Rt.
Then, in Step S7, it is determined if boosting is made to take place for the predetermined number of times (=K) and, if the answer to this question is negative, the processing operation from Step S2 to Step S7 is repeated. When boosting is made to take place for the predetermined number of times, the learning session is made to end. The process of repetition is terminated when the number of learnt weak discriminators is sufficient for discriminating objects from the images as objects of detection such as learning samples.
(5-2) Generation of Weak Discriminators
Now, the leaning method (generating method) of weak discriminators of above described Step S2 will be discussed below. The method of generating weak discriminators differs between when the weak discriminators are adapted to output a binary value and when they are adapted to output a continuous value as function f(x) expressed by the formula (6) above. Additionally, when the weak discriminators are adapted to output a binary value, it slightly differs between when they discriminate an object and a non-object by means of a single threshold value and when they discriminate an object and a non-object by means of two threshold values as shown in the formula (2) above. The learning method (generating method) of weak discriminators adapted to output a binary value at a single threshold value Th will be described below.
(Step S11) Selection of Pixels
In this step, two pixels are arbitrarily selected from all the pixels of a learning sample. When, for example, a learning sample with 20×20 pixels is used, there are 400×399 different ways of selecting two pixels from that number of pixels and one of such ways will be selected. Assume here that the positions of the two pixels are S1 and S2 and the luminance values of the two pixels are I1 and I2.
(Step S12) Preparation of Frequency Distribution
Then, the inter-pixel difference characteristic d, which is the difference (I1-I2) of the luminance values of the two pixels selected in Step S11, is determined for all the learning samples and a histogram (frequency distribution) as shown in
(Step S13) Computation of Threshold Value Thmin
Thereafter, the threshold value Thmin that minimizes the weighted error ratio et (emin) as shown in the above formula (10) is determined from the frequency distribution obtained in Step S12.
(Step S14) Computation of Threshold Value Thmax
Then, the threshold value Thmax that maximizes the weighted error ratio et (emax) as shown in the above formula (10) is determined and inverts the threshold value by means of the method expressed by formula (14) below. In other words, each weak discriminator is adapted to output either of two values that respectively represent the right answer and the wrong answer depending on if the determined inter-pixel difference characteristic d is greater than the single threshold value or not. Therefore, when the weighted error ratio et is smaller than 0.5, it can be made not smaller than 0.5 by the inversion.
(Step S15) Determination of Parameters
Finally, the parameters of each weak discriminator including the positions S1 and S2 of the two pixels and the threshold value Th are determined from the above emin and emax′. More specifically,
Then, in Step S16, it is determined if the processing operation has been repeated for the predetermined number of times M or not. If the processing operation has been repeated for the predetermined number of times, the operation proceeds to Step S17 and the weak discriminator that shows the smallest error ratio et is selected out of the weak discriminators generated by the repetition of M times. Then the operation proceeds to Step S3 shown in
While the data weight Dt, i determined in Step S5 as a result of repeating the processing operation is used to learn the characteristic quantities of a plurality of weak discriminators and the weak discriminator showing the smallest error ratio as indicated by the above formula (10) is selected from the weak discriminators (weak discriminator candidates) in this embodiment, the weak discriminator may alternatively be generated by arbitrarily selecting pixel positions from a plurality of pixel positions that are prepared or learnt in advance. Still alternatively, the weak discriminator may be generated by using learning samples different from the learning samples employed for the operation of repeating Steps S2 through S7. The weak discriminators and the discriminator that are generated may be evaluated by bringing in samples other than the learning samples as in the case of using a cross-validation technique or a jack-knife technique. A cross-validation technique is a technique by which a learning sample is equally divided into I samples and a learning session is conducted by using them except one and the result of the learning session is evaluated by the remaining one. Then, the above operation is repeated for I times to finalize the evaluation of the result.
When, on the other hand, each weak discriminator uses two threshold values Th1 and Th2 as indicated by the above formula (4) or (5), the processing operation of Steps S13 through 15 shown in
When a weak discriminator outputs the result of discrimination by using two threshold values Th1 and Th2, the frequency distribution of inter-pixel difference characteristics is determined in Step S12 shown in
In the case of weak discriminators adapted to output a continuous value as indicated by the above formula (6), firstly two pixels are randomly selected as in Step S1 of
When a learning sample of 20×20 pixels is used to generate a weak discriminator, there are a total of 159,000 ways of selecting two pixels from that number of pixels. Therefore, the one that shows the smallest error ratio may be adopted for the weak discriminator after repeating the selecting process for M=159,000 times at most. While a highly performable weak discriminator can be generated when the selecting process is repeated for the largest possible number of times and a weak discriminator that shows the smallest error ratio is adopted as described above, a weak discriminator that shows the smallest error ratio may be adopted after repeating the selecting process for a number of times less than the largest possible number of times, e.g., hundreds times.
(6) Object Detecting Method
Now, the object detecting method of the object detecting device illustrated in
(Step S21) Generation of Scaled Image
The scaling section 3 as shown in
The scanning section 4 shown in
(Steps S23, S24) Computation of Evaluation Value s
Then, it is judged if the window image output from the scanning section 4 is an object or not. The discriminator 5 sequentially adds weights to the respective estimates f(x) of the above described plurality of weak discriminators to obtain the updated value of the weighted majority decision as evaluation value s. Then, it is judged if the window image is an object or not according to the evaluation value s and also if the discriminating operation is to be suspended or not.
Firstly, as a window image is input, its evaluation value s is initialized to s=0. The first stage weak discriminator 211 of the discriminator 5 computes the inter-pixel difference characteristic dt (Step S23). Then, the estimate value output from the weak discriminator 211 is reflected to the above evaluation value s (Step S24).
As described above by referring to the formulas (3) through (5), a weak discriminator that outputs a binary value as estimate value and a weak discriminator that outputs a function f(x) as estimate value differs from each other in terms of the way of reflecting the estimate to the evaluation value s.
Firstly, when the above formula (2) is used to a weak discriminator that outputs a binary value as evaluation value, the evaluation value s is expressed by formula (15) below.
When the above formula (3) is used to a weak discriminator that outputs a binary value as evaluation value, the evaluation value s is expressed by formula (16) below.
When the above formula (4) is used to a weak discriminator that outputs a binary value as evaluation value, the evaluation value s is expressed by formula (17) below.
Finally, when the above formula (5) is used to a weak discriminator that outputs a function f as evaluation value, the evaluation value s is expressed by formula (18) below.
[Formula 17]
evaluation value: s←s+f(d) (18)
(Steps S25, S26) Judgment of Suspension
Then, the discriminator 5 determines if the evaluation value s obtained (updated) by any of the above described four techniques is greater than the suspension threshold value Rt or not. If it is determined that the evaluation value s is the threshold value Rt, it is then determined if the processing operation has been repeated to the predetermined number of times (=K times) or not (Step S26). If it is determined that the processing operation has not been repeated for the predetermined number of times, the processing from Step S23 is repeated.
If, on the other hand, it is determined that the processing operation has been repeated for the predetermined number of times (=K times), the operation proceeds to Step S27 when the evaluation s is smaller than the suspension threshold value Rt, where it is determined if the window image is an object or not according to if the obtained evaluation value s is greater than 0 or not. If it is determined that the window image is an object, the current window position is stored and it is determined if there is the next search window or not (Step S27). If it is determined that there is the next search window, the processing operation from Step S22 is repeated. If, on the other hand, all the search windows have been scanned for all the next area, the processing operation proceeds to Step S28, where it is determined if there is the next scaled image or not. If it is determined that there is no next scaled image, the processing operation proceeds to Step S29, where the overlapping area is removed. If, on the other hand, it is determined that there is the next scaled image, the processing operation from Step S21 is repeated. The scaling operation of Step S21 is terminated when the scaled image becomes smaller than the window image.
(Steps S29 through S31) Removal of Overlapping Area
When all the scaled images are processed for a single input image, the processing operation moves to Step S29. In the processing operation from Step S29 on, one of the areas in an input image that are judged to be objects and overlapping with each other, if any, is removed. Firstly, it is determined if areas that are overlapping with each other or not and, if it is determined that there are a plurality of areas stored in Step S26 and any of them are overlapping, the processing operation proceeds to Step S30, where the two overlapping areas are taken out and one of the areas that shows a smaller evaluation value s is removed as it is regarded to show a low reliability and the area that shows a greater evaluation value is selected for use (Step S29). Then, the processing operation from Step S29 is repeated once again. As a result, of the areas that are extracted for a plurality of times to overlap with each other, a single area that shows the highest evaluation value is selected. When there are not two or more than two object areas that overlap with each other and when there is no object area, the processing operation on the input image is terminated and the processing operation on the next frame starts.
As described above in detail, with the object detecting method of this embodiment, it is possible to process each window image to detect a fact from the image at very high speed on a real time basis because the operation of computing the characteristic quantity of the object in the above described Step S23 is terminated simply by reading the luminance values of two corresponding pixels of the window image, using a discriminator that has learnt by group learning the weak discriminators that weakly discriminate an object and a non-object by way of the inter-pixel difference characteristic of the image. Additionally, each time the evaluation value s is updated by multiplying the result of discrimination (estimate) obtained from the characteristic quantity by the reliability of the weak discriminator used for the discrimination and adding the product of multiplication, the updated evaluation value s is compared with the suspension threshold value Rt to determine if the operation of computing the estimates of the weak discriminators is to be continued or not. When the evaluation value s falls below the suspension threshold value Rt, the computing operation of the weak discriminators is suspended to proceed to the operation of processing the next window image so that it is possible to dramatically reduce wasteful computing operations to further improve the speed of detecting a face. When all the areas of the input image and the scaled images obtained by scaling down the input image are scanned to cut out window images, the probability of being an object of each window image is very small and most of the window images are non-objects. As the operation of discriminating an object and a non-object in the window images, which are mostly non-objects, is suspended on the way, it is possible to dramatically improve the efficiency of the discriminating step. If, to the contrary, the window images include many objects to be detected, a threshold value similar to the above described suspension threshold value may be provided to suspend the computing operation using the window images that are apparently objects. Furthermore, it is possible to detect objects of any size by scaling the input image by means of the scaling section to define a search window of an arbitrarily selected size.
(7) Example
Now, the present invention will be described further by way of an example where a face was actually detected as object. However, it may be needless to say that the object is not limited to a face and it is possible to detect any object other than the face of a man that shows characteristic features on a two-dimensional plane such as a logotype or a pattern and can be discriminated to a certain extent by the inter-pixel difference characteristic thereof as described above (so that it can constitute a weak discriminator).
In this example, face discrimination problems were learnt from the learning samples according to the algorithm illustrated in
In this example, a correct answer ratio of 70% (performance relative to the learning samples) was achieved by the first weak discriminator f1. The correct answer ratio rose to 80% when all the weak discriminators f1 through f6 were used. The correct answer ratio further rose to 90% when 40 weak discriminators were combined and to 99% when 765 weak discriminators were combined.
The present invention is by no means limited to the above described embodiment, which may be modified and altered in various different ways without departing from the scope of the present invention.
Hidai, Kenichi, Sabe, Kohtaro, Kawamoto, Kenta
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
5874966, | Oct 30 1995 | International Business Machines Corporation | Customizable graphical user interface that automatically identifies major objects in a user-selected digitized color image and permits data to be associated with the major objects |
6272247, | May 18 1999 | SCOTT D ROTH, D B A HACIENDA TECHNOLOGY | Rotation and scale invariant image finder |
6487304, | Jun 16 1999 | Microsoft Technology Licensing, LLC | Multi-view approach to motion and stereo |
6711279, | Nov 17 2000 | COLUMBIA PEAK VENTURES, LLC | Object detection |
7050607, | Dec 08 2001 | Microsoft Technology Licensing, LLC | System and method for multi-view face detection |
7054489, | Sep 30 1999 | Matsushita Electric Industrial Co., Ltd. | Apparatus and method for image recognition |
7536044, | Nov 19 2003 | SIEMENS HEALTHINEERS AG | System and method for detecting and matching anatomical structures using appearance and shape |
7953253, | Dec 31 2005 | ARCSOFT, INC | Face detection on mobile devices |
8520955, | Dec 28 2007 | Tsinghua University; Omron Corporation | Object detection apparatus and method |
8660320, | Oct 23 2009 | Saturn Licensing LLC | Image processing apparatus and image processing method |
20020102024, | |||
20030110147, | |||
20040233299, | |||
20050013479, | |||
20060193520, | |||
20070076954, | |||
20070086660, | |||
20130159292, | |||
20140180977, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jun 18 2015 | Sony Corporation | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Sep 23 2020 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Sep 13 2021 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Jun 11 2022 | 4 years fee payment window open |
Dec 11 2022 | 6 months grace period start (w surcharge) |
Jun 11 2023 | patent expiry (for year 4) |
Jun 11 2025 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jun 11 2026 | 8 years fee payment window open |
Dec 11 2026 | 6 months grace period start (w surcharge) |
Jun 11 2027 | patent expiry (for year 8) |
Jun 11 2029 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jun 11 2030 | 12 years fee payment window open |
Dec 11 2030 | 6 months grace period start (w surcharge) |
Jun 11 2031 | patent expiry (for year 12) |
Jun 11 2033 | 2 years to revive unintentionally abandoned end. (for year 12) |