Systems and methods for evaluating perception systems for autonomous vehicles using quality temporal logic

Abstract

Various embodiments for systems and methods of evaluating perception systems for autonomous vehicles using a quality temporal logic are disclosed herein.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a non-provisional application that claims benefit to U.S. provisional application Ser. No. 62/791,412 filed on Jan. 11, 2019, which is herein incorporated by reference in its entirety.

FIELD

The present disclosure generally relates to evaluating perception systems for autonomous vehicles; and in particular, to systems and methods for evaluating perception system for autonomous vehicles using quality temporal logic.

BACKGROUND

The wide availability of high-performance GPU-based hardware has led to an explosion in the applications of Machine Learning (ML) techniques to real-time image recognition problems, especially using deep learning. Such techniques are being used in safety-critical applications such as self-driving vehicles. Testing of these systems is largely based on either (a) measuring the recognition error on a pre-recorded data-set, or (b) running actual driving tests on the road with a backup human driver and focusing on the disengagements. Disengagement is an event when the autonomous car returns control back to the human driver. There is, thus, an urgent need for techniques to formally reason about the correctness and performance of such driving applications that use perception systems based on Deep Neural Networks (DNN) and ML algorithms.

The key challenge in formal verification is that it is infeasible to specify functional correctness of components using learning-based models in an abstract fashion. However, the confidence in the vision-based system can be vastly improved by extensive, safety-driven virtual testing of the vision algorithms.

It is with these observations in mind, among others, that various aspects of the present disclosure were conceived and developed.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

FIG. 1 shows various image frames of a cyclist resulting from the application of SqueezeDet on a KITTI data stream;

FIG. 2 shows an implementation of one embodiment of the present system within an autonomous vehicle testing framework; and

FIG. 3 is a simplified block diagram showing an exemplary computing system for effectuating the system and method for evaluating perception systems for autonomous vehicles.

Corresponding reference characters indicate corresponding elements among the view of the drawings. The headings used in the figures do not limit the scope of the claims.

DETAILED DESCRIPTION

The present disclosure focuses on deep learning algorithms that analyze images or sequences of images in order to detect and classify objects for intention recognition and scenario classification. In order to evaluate the performance of the perception algorithms over time, quality requirements are required that capture temporal dependencies between detected objects. Most importantly, going beyond ad-hoc validation and testing, a formal framework is needed that facilitates temporal reasoning over the quality of the perception systems. Such a formal framework would enable the community to create a precise and real-life set of requirements that need to be met by any learning-based model.

In the present disclosure, temporal logic based quality requirements are considered for scoring or grading the results of perception algorithms. Then, a quality monitor considers the quality requirements to score the learning-based perception results. The present disclosure considers evaluating timed object data with respect to quality requirements presented in Timed Quality Temporal Logic (TQTL), which is based on Timed Propositional Temporal Logic (TPTL).

Problem Formulation

It is assumed that a data stream is provided by a perception algorithm, LIDAR, RADAR, or other devices. An atomic block in a data stream is a frame which is a set {data₁, . . . , data_m}, wherein each data_j is an element of a data domain or data object. A stream is a sequence of frames (i) where i is the timestamp over a linearly ordered set. Each frame i contains data object data_j∈(i) in a data-structure (a tuple) format, for example data_j=(ID, Class, Probability, BBOX (Bounding Box), . . . ). This data structure depends on the ML algorithm. The present method assumes that for each ML algorithm there exists a customized retrieve function which can extract and access the information of the corresponding data object (data_j). In addition, there exists a customized attribute with quality metric which can be evaluated by a quality function to provide us the quality of the data objects in the stream.

In the case study, it is assumed that each data object of the ML algorithm has the following format data_j=(ID, Class, Probability, BBOX), where ID∈ is a number uniquely identifying an object, Probability ∈∩[0, 1], Class∈{Car, Cyclist, Pedestrian}, and BBOX is a tuple of four integers [top, left, bottom, right]∈⁴ representing the coordinates of the bounding box in pixels. For example, a data stream for 3 different detected objects would look like:

$\mathrm{Data}\mathrm{Frame}0:𝒟\left(0\right)=\left\{\left(1,{ }^{″}{\mathrm{Car}}^{″},0.9,\left[283,181,413,231\right]\right)\right\}\mathrm{Data}\mathrm{Frame}1:𝒟\left(1\right)=\left\{\left(2,{ }^{″}{\mathrm{Car}}^{″},0.8,[\dots ]\right),\left(1,{ }^{″}{\mathrm{Car}}^{″},0.9,[\dots ]\right)\right\}\mathrm{Data}\mathrm{Frame}2:𝒟\left(2\right)=\left\{\left(3,{ }^{″}{\mathrm{Car}}^{″},0.95,[\dots ]\right),\left(2,{ }^{″}{\mathrm{Pedestrian}}^{″},0.9,[\dots ]\right)\right\}⋮$

It is also assumed that there exists a function (short for Set of Objects) which can retrieve the object IDs from a data frame (i). For the above streaming data , the function SO returns the following values of object IDs: ((0))={1}, ((1))={1, 2}, ((2))={2, 3}. In order to retrieve the other fields (non-ID) from the data objects, it is assumed that these fields are available through the function . The present system uses object-oriented notation “.” to retrieve the “XYZ” fields (attributes) from (i) for object ID. So, ((I), id)·XYZ retrieves the “XYZ” attribute of object id from frame i. For the above vision stream example, ((1), 2)·Class=Car, and ((1), 2). Probability=0.8.

Without loss of generality, it is assumed that each data object in the stream is provided by the ML algorithm with a unique object ID, wherein the specific object ID is unique over different frames of the video. In other words, it is assumed that the ML algorithm can match the objects of different frames and provide a unique ID of the object within the whole data stream. This is a necessary assumption to help us track these objects through frames and to apply temporal reasoning over the specific objects.

Given a finite stream and Quality Temporal Logic formula φ, the goal of the present system is to compute the quality value of formula φ with respect to stream . Throughout this disclosure, the notation ϕ is used to represent the quality value of formula ϕ. Finally, it is assumed that the quality can be quantified by a real-valued number similar to the robust semantics of temporal logics.

Timed Quality Temporal Logic

In this section, the important aspects of stream reasoning for object detection algorithms are considered. Also, the corresponding syntax and semantics will be provided to address the problem of quality reasoning. One important differentiating factor of quality monitoring to Signal Temporal Logic (STL) monitoring is that the number of objects in the video is dynamically changing. Therefore, the present method introduces Existential and Universal quantifiers to reason about the dynamically changing number of data objects. In addition, the present system needs to be able to record the time in order to extract the data objects using function at different timestamps.

Timed Quality Temporal Logic (TQTL) is defined to reason about a stream . Assume that ={π₁, π₂, . . . , π_n} is a set of predicates which define assertions about data objects. Each π_j corresponds to π_j≡ƒ(j₁, . . . , j_n, id₁, . . . , id_n)˜c, where ƒ(j₁, . . . , j_n, id₁, . . . , id_n) is a scoring function which extracts/processes information about data objects id₁, . . . , id_n from frames j₁, . . . , j_n, and compares it with a constant c to resolve the quality of the data objects. It should be noted that c can be a string, integer, real, enumerator, or any constant data type that the ML algorithm uses to represent data. The symbols ˜∈{=, ≥, >, <, ≤} are relational operators.

Definition 1 (TQTL Syntax). The set of TQTL formulas ϕ over a finite set of predicates , a finite set of time variables (V_t), and a finite set of object indexes (V_o) is inductively defined according to the following BNF grammar:
φ::=T|π|x·ϕ|∃id@x,ϕ|x≤y+n|−¬ϕ|ϕ₁∨ϕ₂|ϕ₁Uϕ₂
where π∈, T is true, x, y∈V_t, n∈, id∈V_o, U is the Until operator. The time constraints of TQTL are represented in the form of x≤y+n. The freeze time quantifier x·ϕ assigns the current time i to time variable x before processing the subformula ϕ. The Existential quantifier is denoted as ∃. The Universal quantifier is defined as ∀id@x,ϕ≡¬(∃id@x,¬ϕ). For TQTL formulas ψ, φ, define ψ∧φ≡¬(¬ψ∨¬ϕ), ⊥≡¬T (False), ψ→ϕ≡¬∨ϕ (ψ Implies ϕ), ⋄ψ≡TUψ (Eventually ψ), □ψ≡¬⋄¬ψ (Always ψ) using syntactic manipulation. The semantics of TQTL is defined over an evaluation function ϵ:V_t∪V_o→ which is an environment for the time variables and object IDs.

Definition 2 (TQTL Semantics). Consider the data stream , i∈ is the index of current frame, π∈, ϕ, ϕ₁, ϕ₂∈TQTL and evaluation function ϵ:V_t∪V_o→. The quality value of formula ϕ with respect to at frame i with evaluation ∈ is recursively assigned as follows:

$〚T〛\left(𝒟,i,ϵ\right):=+\infty 〚\pi 〛\left(𝒟,i,ϵ\right):=〚f_{\pi }\left(j_1,\dots ,j_n,{\mathrm{id}}_1,\dots ,{\mathrm{id}}_n\right)~c〛\left(𝒟,i,ϵ\right)〚x,\varphi 〛\left(𝒟,i,ϵ\right):=〚\varphi 〛\left(𝒟,i,ϵ\left[x\Leftarrow i\right]\right)〚\exists \mathrm{id}@x,\varphi 〛\left(𝒟,i,ϵ\right):=\underset{k\in \mathrm{𝒮𝒪}\left(𝒟\left(e\left(x\right)\right)\right)}{\max }\left(〚\varphi 〛\left(𝒟,i,ϵ\left[\mathrm{id}\Leftarrow k\right]\right)\right)〚x\le y+n〛\left(𝒟,i,ϵ\right):=\{\begin{array}{cc}+\infty & \mathrm{if}ϵ\left(x\right)\le ϵ\left(y\right)+n\\ -\infty & \mathrm{otherwise}\end{array}〚\neg \varphi 〛\left(𝒟,i,ϵ\right):=-〚\varphi 〛\left(𝒟,i,ϵ\right)〚\text{⁠}{\varphi }_1\vee \text{⁠}{\varphi }_2〛\text{⁠}\left(𝒟,\text{⁠}i,\text{⁠}ϵ\right):=\text{⁠}\max \left(〚\text{⁠}{\varphi }_1〛\text{⁠}\left(𝒟,\text{⁠}i,\text{⁠}ϵ\right),〚\text{⁠}{\varphi }_2〛\text{⁠}\left(𝒟,\text{⁠}i,\text{⁠}ϵ\right)\right)〚{\varphi }_{1}U{\varphi }_2〛\left(𝒟,i,ϵ\right):=\underset{i\le j}{\max }(\min (〚{\varphi }_2〛\left(𝒟,j,ϵ\right),\underset{i\le k\le j}{\min }〚{\varphi }_1〛\left(𝒟,k,ϵ\right)))$

Here, ϵ[x⇐a] assigns the value a into the variable x∈V in the environment ϵ. Given a variable ϵ∈V and a value q∈, the environment ϵ′=ϵ[x⇐q] is designed to be equivalent to the environment ϵ on all variables in V except variable x which now has value q. It is said that satisfies φ(|=φ) iff ϕ (, 0, ϵ₀)>0, where ϵ₀ is the initial environment. On the other hand, a data stream ′ does not satisfy a TQTL formula ϕ (denoted by ′|≠ϕ), iff ϕ (, 0, ϵ₀)≤0. The quantifier ∃id@x is the maximum operation on the quality values of formula ϕ corresponding to the objects IDs=ϵ(id) that are detected at frame ϵ(x).

It is assumed that for each ML algorithm there exists a corresponding retrieve function to extract the values corresponding to data objects. The set of predicates ={π₁, π₂, . . . , π_n} evaluate object data values and return a quality value in ∪{±∞}. Each quality predicate π has an associated scoring function ƒ_π, The scoring function ƒ_π(j₁, . . . , j_n, id₁, . . . , id_n) extracts specific information about object id_k at frame j_k for each k∈{1, . . . , n} and compares it with c to compute the quality value of the predicate π represented as ƒ_π(j₁, . . . , j_n, id₁, . . . , id_n)˜c∈∪{±∞} similar to robustness semantics. The scoring functions of the quality predicates depend on the ML algorithm, , data fields of , operator ˜, and type of c. Each scoring function ƒ_π uses the application dependent customized function h_π to compute the quality of the corresponding objects. The function h_π then returns a value about the quality of the data objects which will be used by ƒ_π to compute the quality value of the predicate π which is denoted as π.

In general, ƒ_π(j₁, . . . , j_n, id₁, . . . , id_n)˜c may be of two types. The first returns a Boolean result of comparing values from sets without scalar metrics, i.e., comparing the object class of pedestrian with respect to {car, cyclist}. For example, in this case, for equality =, it can be defined as:

$〚f_e\left(\stackrel{.}{j_1},\dots ,j_n,{\mathrm{id}}_1,\dots ,\stackrel{.}{{\mathrm{id}}_n}\right)=\stackrel{.}{c}〛\left(\stackrel{.}{𝒟,i,ϵ}\right):=\{\begin{array}{cc}+\infty & \begin{array}{c}\mathrm{if}{h}_e\left([ℛ\left(𝒟\left(ϵ\left(j_k\right)\right),ϵ\left({\mathrm{id}}_k\right)\right){]}_{k=1}^n\right)=c\wedge \forall k\in \left\{1,\dots ,n\right\},\\ ϵ\left({\mathrm{id}}_k\right)\in \mathrm{𝒮𝒪}\left(𝒟\left(ϵ\left(j_k\right)\right)\right)\end{array}\\ -\infty & \mathrm{otherwise}\end{array}$

The second type is for predicates comparing values from sets with well-defined metrics similar to the various temporal logic robust semantics. For example, for “greater than”, the present system could define:

$〚f_n\left(j_1,\dots ,j_n,{\mathrm{id}}_1,\dots ,{\mathrm{id}}_n\right)>c〛\left(𝒟,i,ϵ\right):=\{\begin{array}{cc}{h}_n\left({\left[ℛ\left(𝒟\left(ϵ\left(j_k\right)\right),ϵ\left({\mathrm{id}}_k\right)\right)\right]}_{k=1}^n\right)-c& \begin{array}{c}\mathrm{if}\forall k\in \left\{1,\dots ,n\right\},\\ ϵ\left({\mathrm{id}}_k\right)\in \mathrm{𝒮𝒪}\left(𝒟\left(ϵ\left(j_k\right)\right)\right)\end{array}\\ -\infty & \mathrm{otherwise}\end{array}$
wherein k is index of the kth object variable (id_k) and j_k is the kth time variable of the formula. Here, h_e and h_n are application dependent functions on the data fields of , which process the retrieved values of data objects ϵ(id_k) at ϵ(j_k) if ϵ(id_k)∈((ϵ(j_k)). The second predicate type can return finite real values.

TQTL Example: Referring to FIG. 1. When a car is following a cyclist, it is important that the cyclist is correctly classified in order to utilize the appropriate predictive motion model. The present system considers the following vision quality requirement: “At every time step, for all the objects (id₁) in the frame, if the object class is cyclist with probability more than 0.7, then in the next 5 frames the object id₁ should still be classified as a cyclist with probability more than 0.6′. The requirement can be formalized in TQTL as follows:

${\varphi }_1=\text{⁠}\square \left(\begin{array}{c}x·\forall {\mathrm{id}}_1@x,\left(C\left(x,{\mathrm{id}}_1\right)=\mathrm{Cyclist}\wedge P\left(x,{\mathrm{id}}_1\right)>0.7\right)\to \\ \square \left(y·\left(\left(x\le y\wedge y\le x+5\right)\to C\left(y,{\mathrm{id}}_1\right)=\mathrm{Cyclist}\wedge P\left(y,{\mathrm{id}}_1\right)>0.6\right)\right)\end{array}\right)\mathrm{where}〚C\left(x,{\mathrm{id}}_1\right)=\mathrm{Cyclist}〛\left(𝒟,i,ϵ\right):=\{\begin{array}{cc}+\infty & ϵ\left({\mathrm{id}}_1\right)\in \mathrm{𝒮𝒪}\left(𝒟\left(ϵ\left(x\right)\right)\right)\mathrm{and}ℛ\left(𝒟\left(ϵ\left(x\right)\right),ϵ\left({\mathrm{id}}_1\right)\right)·\mathrm{Class}=\mathrm{Cyclist}\\ -\infty & ϵ\left({\mathrm{id}}_1\right)\notin \mathrm{𝒮𝒪}\left(𝒟\left(ϵ\left(x\right)\right)\right)\mathrm{or}\\ & ϵ\left({\mathrm{id}}_1\right)\in \mathrm{𝒮𝒪}\left(𝒟\left(ϵ\left(x\right)\right)\right)\mathrm{and}ℛ\left(𝒟\left(ϵ\left(x\right)\right),ϵ\left({\mathrm{id}}_1\right)\right)·\mathrm{Class}\ne \mathrm{Cyclist}\end{array}$

Experimental Results

The implementation of a TQTL monitor of the present system is based on the publicly available S-TALIRO toolbox. The TQTL monitor was evaluated using the KITTI benchmark dataset. The KITTI dataset is a well-known benchmark for autonomous driving which contains real traffic situations from Karlsruhe, Germany. A processor ran the SqueezeDet object detection algorithm on some KITTI data streams. It should be noted that the object matching is manually annotated in these data streams since it is assumed that the objects are matched correctly. However, automated reliability reasoning about object matching throughout video streams is a challenging problem and it is the focus of on-going research.

In FIG. 1, the results of SqueezeDet on a KITTI data stream (following a cyclist) are provided. The TQTL formula ϕ₁ from the previous section with respect to was evaluated. The monitor tool obtained a negative result when evaluating ϕ₁, i.e., |≠ϕ₁. This was because the data stream D does not contain an object classified as cyclist in Frames 84 and 85 in FIG. 1. A closer investigation of Frames 84 and 85 shows that, although a cyclist is not detected (yellow box), a pedestrian is detected at the position of the cyclist with purple color. If the motion prediction algorithm had the capability to tolerate an object classification change, e.g., from cyclist to pedestrian and back, then the formal specification should be able to reflect that. To specify such a behavior, the second □ of ϕ₁ should be changed to:
□(y·((x≤y∧y≤x+5)→C(y,id₁)=Cyclist ∧P(y,id₁)>0.6
∨∃id₂@y,(C(y,id₂)=Pedestrian ∧dist(x,y,id₁,id₂)<40 ∧P(y,id₂)>0.6)))
where the scoring function of dist extracts the coordinates of the bounding boxes of object id₁ at frame x and object id₂ at frame y for computing the center to center distance between these boxes. The requirement is now satisfied by .
Persephone

One embodiment of this framework, referred to herein as “Persephone”, is implemented within a larger autonomous driving testing system called Sim-ATAV. FIG. 2 refers to a “simulation execution” block in which Persephone implements the TQTL monitoring logic in MATLAB. This embodiment of the framework continually verifies the validity of the Sim-ATAV perception module. Sim-ATAV is available as an open-source project at this link: https://sites.google.com/a/asu.edu/s-taliro/sim-atav. Persephone is available as an open-source project at this link: https://bitbucket.org/cps_vida/persephone/src/master/.

Computing System

An example of a suitable computing system 100 used to implement various aspects of the present system and methods for evaluating perception systems for autonomous vehicles using quality temporal logic are discussed below and shown in FIG. 3. Example embodiments described herein may be implemented at least in part in electronic circuitry; in computer hardware executing firmware and/or software instructions; and/or in combinations thereof. Example embodiments also may be implemented using a computer program product (e.g., a computer program tangibly or non-transitorily embodied in a machine-readable medium and including instructions for execution by, or to control the operation of, a data processing apparatus, such as, for example, one or more programmable processors or computers). A computer program may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a subroutine or other unit suitable for use in a computing environment. Also, a computer program can be deployed to be executed on one computer, or to be executed on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Certain embodiments are described herein as including one or more modules 112. Such modules 112 are hardware-implemented, and thus include at least one tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. For example, a hardware-implemented module 112 may comprise dedicated circuitry that is permanently configured (e.g., as a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented module 112 may also comprise programmable circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software or firmware to perform certain operations. In some example embodiments, one or more computer systems (e.g., a standalone system, a client and/or server computer system, or a peer-to-peer computer system) or one or more processors may be configured by software (e.g., an application or application portion) as a hardware-implemented module 112 that operates to perform certain operations as described herein.

Accordingly, the term “hardware-implemented module” encompasses a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware-implemented modules 112 are temporarily configured (e.g., programmed), each of the hardware-implemented modules 112 need not be configured or instantiated at any one instance in time. For example, where the hardware-implemented modules 112 comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware-implemented modules 112 at different times. Software may accordingly configure a processor 102, for example, to constitute a particular hardware-implemented module at one instance of time and to constitute a different hardware-implemented module 112 at a different instance of time.

Hardware-implemented modules 112 may provide information to, and/or receive information from, other hardware-implemented modules 112. Accordingly, the described hardware-implemented modules 112 may be regarded as being communicatively coupled. Where multiple of such hardware-implemented modules 112 exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware-implemented modules. In embodiments in which multiple hardware-implemented modules 112 are configured or instantiated at different times, communications between such hardware-implemented modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware-implemented modules 112 have access. For example, one hardware-implemented module 112 may perform an operation, and may store the output of that operation in a memory device to which it is communicatively coupled. A further hardware-implemented module 112 may then, at a later time, access the memory device to retrieve and process the stored output. Hardware-implemented modules 112 may also initiate communications with input or output devices.

As illustrated, the computing system 100 may be a general purpose computing device, although it is contemplated that the computing system 100 may include other computing systems, such as personal computers, server computers, hand-held or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, digital signal processors, state machines, logic circuitries, distributed computing environments that include any of the above computing systems or devices, and the like.

Components of the general purpose computing device may include various hardware components, such as a processor 102, a main memory 104 (e.g., a system memory), and a system bus 101 that couples various system components of the general purpose computing device to the processor 102. The system bus 101 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

The computing system 100 may further include a variety of computer-readable media 107 that includes removable/non-removable media and volatile/nonvolatile media, but excludes transitory propagated signals. Computer-readable media 107 may also include computer storage media and communication media. Computer storage media includes removable/non-removable media and volatile/nonvolatile media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules or other data, such as RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information/data and which may be accessed by the general purpose computing device. Communication media includes computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media may include wired media such as a wired network or direct-wired connection and wireless media such as acoustic, RF, infrared, and/or other wireless media, or some combination thereof. Computer-readable media may be embodied as a computer program product, such as software stored on computer storage media.

The main memory 104 includes computer storage media in the form of volatile/nonvolatile memory such as read only memory (ROM) and random access memory (RAM). A basic input/output system (BIOS), containing the basic routines that help to transfer information between elements within the general purpose computing device (e.g., during start-up) is typically stored in ROM. RAM typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processor 102. For example, in one embodiment, data storage 106 holds an operating system, application programs, and other program modules and program data.

Data storage 106 may also include other removable/non-removable, volatile/nonvolatile computer storage media. For example, data storage 106 may be: a hard disk drive that reads from or writes to non-removable, nonvolatile magnetic media; a magnetic disk drive that reads from or writes to a removable, nonvolatile magnetic disk; and/or an optical disk drive that reads from or writes to a removable, nonvolatile optical disk such as a CD-ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media may include magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The drives and their associated computer storage media provide storage of computer-readable instructions, data structures, program modules and other data for the general purpose computing device 100.

A user may enter commands and information through a user interface 140 or other input devices 145 such as a tablet, electronic digitizer, a microphone, keyboard, and/or pointing device, commonly referred to as mouse, trackball or touch pad. Other input devices 145 may include a joystick, game pad, satellite dish, scanner, or the like. Additionally, voice inputs, gesture inputs (e.g., via hands or fingers), or other natural user interfaces may also be used with the appropriate input devices, such as a microphone, camera, tablet, touch pad, glove, or other sensor. These and other input devices 145 are often connected to the processor 102 through a user interface 140 that is coupled to the system bus 101, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 160 or other type of display device is also connected to the system bus 101 via user interface 140, such as a video interface. The monitor 160 may also be integrated with a touch-screen panel or the like.

The general purpose computing device may operate in a networked or cloud-computing environment using logical connections of a network interface 103 to one or more remote devices, such as a remote computer. The remote computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the general purpose computing device. The logical connection may include one or more local area networks (LAN) and one or more wide area networks (WAN), but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

When used in a networked or cloud-computing environment, the general purpose computing device may be connected to a public and/or private network through the network interface 103. In such embodiments, a modem or other means for establishing communications over the network is connected to the system bus 101 via the network interface 103 or other appropriate mechanism. A wireless networking component including an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a network. In a networked environment, program modules depicted relative to the general purpose computing device, or portions thereof, may be stored in the remote memory storage device.

It should be understood from the foregoing that, while particular embodiments have been illustrated and described, various modifications can be made thereto without departing from the spirit and scope of the invention as will be apparent to those skilled in the art. Such changes and modifications are within the scope and teachings of this invention as defined in the claims appended hereto.

Claims

1. A framework for evaluating the quality of a vehicle perception system, the framework comprising:

a processor operable for executing instructions comprising:

extracting an object index and a set of data attributes from a frame of an object detection data stream, wherein the object index and the set of data attributes are associated with an object detected by a vehicle perception system;

receiving a set of quality predicates descriptive of expected behavior of the vehicle perception system over a plurality of frames, wherein each of the set of quality predicates is associated with a frame, wherein each of the set of quality predicates is associated with a scoring function of a plurality of scoring functions and wherein the set of quality predicates descriptive of expected behavior of the vehicle perception system over the plurality of frames are defined in terms of a timed quality temporal logic syntax;

evaluating each of the plurality of scoring functions based on observed behavior of the vehicle perception system over the plurality of frames, wherein each of the plurality of scoring functions extracts information associated with the object using the set of data attributes and compares the information with a constant quality value and wherein a result of the scoring function is descriptive of observed behavior of the vehicle perception system with respect to the associated quality predicate; and

identifying one or more quality predicates of the plurality of quality predicates that the vehicle perception system does not satisfy based on the evaluation of the plurality of scoring functions indicative of observed behavior of the vehicle perception system with respect to expected behavior of the vehicle perception system over the plurality of frames.

2. The framework of claim 1, wherein each of the plurality of scoring functions comprises an application-specific quality function, wherein the application-specific quality function returns a quality value about the set of data attributes indicative of observed behavior of the vehicle perception system over the plurality of frames with respect to expected behavior of the vehicle perception system over the plurality of frames as defined by the associated quality predicate.

3. The framework of claim 2, wherein the quality value returned from the application-specific quality function is used by the scoring function to determine the quality value associated with the quality predicate of the set of quality predicates, wherein the quality value is indicative of how the observed behavior of the vehicle perception system adheres to expected behavior of the vehicle perception system over the plurality of frames.

4. The framework of claim 1, wherein the object detection data stream is provided by a perception algorithm of the vehicle perception system.

5. The framework of claim 4, wherein a set-of-objects function is operable to retrieve the object index from a frame of the object detection stream and wherein the set-of-objects function is specific to the perception algorithm.

6. The framework of claim 4, wherein a retrieval function retrieves the set of data attributes from a frame of the object detection stream and wherein the retrieval function is specific to the perception algorithm.

7. The framework of claim 4, wherein the object detection stream is a tuple comprised of a plurality of data types.

8. The framework of claim 7, wherein the tuple comprises an object index and a plurality of data attributes, wherein the plurality of data attributes comprises a probability associated with the object, a class associated with the object, and a set of bounding box coordinates associated with the object.

9. The framework of claim 2, wherein the application-specific quality function may be defined by a user for a particular application.

10. The framework of claim 1, wherein the scoring function returns a Boolean result when comparing data attributes from sets without scalar metrics.