A system and method may provide assistance to programmer during programming to detect and predict the existence of errors in code and, in some aspects, predict fixes for erroneous code. In some aspects, the system and method may use artificial intelligence to learn based on edits made by programmers, by observing code changes that cause errors and code changes that fix errors, or based on other data.
|
1. A computer-implemented method comprising:
monitoring a communication channel, the communication channel comprising information of an execution environment, compiler, or interpreter, the execution environment configured to execute computer code;
parsing the information in the communication channel and identifying an error, wherein the information comprises computer code and standard output, execution state, execution status, logs or streams of data generated as a result of execution, compiling or interpreting of the computer code;
identifying and storing error text and error context of the error, wherein the error context comprises portions of computer code responsible for the error, configuration settings, or execution state at the time of the error;
extracting one or more features from the error text and error context;
predicting, based on the one or more extracted features, one or more error types of the error, wherein the predicting comprises determining a likelihood for each of the one or more error types;
classifying, based on the determined likelihood of the one or more error types, the error into one or more error types;
predicting, based on the error type, error text and error context, a suggested fix, wherein the suggested fix comprises one or more corrected code segments;
incorporating the suggested fix in the computer code; and
determining whether the error was handled successfully by the incorporating of the suggested fix.
2. The computer-implemented method of
3. The computer-implemented method of
4. The computer-implemented method of
5. The computer-implemented method of
6. The computer-implemented method of
7. The computer-implemented method of
|
This application claims the benefit of U.S. Provisional Patent Application No. 62/735,026 filed Sep. 22, 2018, which is hereby incorporated by reference in its entirety.
The present disclosure relates generally to methods and systems for error recovery during programming and development.
Computer programmers spend a large portion of their time debugging code and fixing errors. It is often time consuming for programmers to debug code, even when facing common-place errors. In the current state of the art, the programmer may have to research online or in a local codebase to determine the reasons for an error. It would be desirable to provide better tools to help programmers eliminate errors in their code.
In some embodiments, a programming co-pilot system is provided that may integrate with a code editor. The programming co-pilot system may provide functionality such as error detection, error classification, error prediction, fix detection, fix prediction, training data generation, or community and team discussion of errors.
One embodiment relates to detection of errors in source code. A configuration request may be received to monitor one or more communication channels of a running computer program. The communication channels may be monitored and content of the communication channels may be parsed to identify an error. Error context of the error may be identified and stored. An error status of the error may be evaluated, and it may be determined whether the error was handled successfully or needs attention.
One embodiment relates to classifying errors into error types. A plurality of training examples may be provided, which may comprise error text, additional error content, and an error type label. One or more features may be extracted from the error text and error content, and a machine learning model may be trained based on the training examples. An unseen error text and error content may be provided and classified into an error type according to the machine learning model.
One embodiment relates to predicting whether a portion of code contains an error. A portion of source code may be provided and a plurality of features extracted from the source code. The features may be input into a machine learning model that has been trained on one or more labeled training examples, where the training examples comprise code portions and labels indicative of errors. The machine learning model may predict from the features the existence of an error and the type of error.
One embodiment relates to detecting fixes to source code during programming by a programmer. A code editor may monitor and store a first set of edits made to source code by a programmer. An error may be detected in the source code. A second set of edits may be made by the programmer and tracked. The source code may be run again and the error determined to no longer be present. The second set of edits made by the programmer, or a portion thereof, may be determined to be a fixing set of changes that fix the error.
One embodiment relates to predicting fixes to source code. A portion of source code may be provided, which contains an error. Moreover, error information may be provided about the error. The source code and error information may be input into a machine learning model, where the machine learning model has been trained on training examples of erroneous code portions and fixes to correct the corresponding code portions. The machine learning model may generate a predicted fix for the error in the source code.
One embodiment relates to generating artificial training examples for code portions having errors. An error-free code portion may be provided. The code portion may then be modified to introduce an error. A fix for the error may be determined using one of several methods. The erroneous code portion, the resulting error, and the fix for the error may be used as a training example for error detection, error prediction, fix prediction, or other methods.
One embodiment relates to presenting a useful user interface to a user to assist with fixing errors in code. The user interface may present a variety of information about an error to assist the user, including crowd-sourced information about the error, suggested fixes for the error, and mechanisms for sharing information from the user to others about the error and potential fixes.
The present disclosure will become better understood from the detailed description and the drawings, wherein:
In this specification, reference is made in detail to specific embodiments of the invention. Some of the embodiments or their aspects are illustrated in the drawings.
For clarity in explanation, the invention has been described with reference to specific embodiments, however it should be understood that the invention is not limited to the described embodiments. On the contrary, the invention covers alternatives, modifications, and equivalents as may be included within its scope as defined by any patent claims. The following embodiments of the invention are set forth without any loss of generality to, and without imposing limitations on, the claimed invention. In the following description, specific details are set forth in order to provide a thorough understanding of the present invention. The present invention may be practiced without some or all of these specific details. In addition, well known features may not have been described in detail to avoid unnecessarily obscuring the invention.
In addition, it should be understood that steps of the exemplary methods set forth in this exemplary patent can be performed in different orders than the order presented in this specification. Furthermore, some steps of the exemplary methods may be performed in parallel rather than being performed sequentially. Also, the steps of the exemplary methods may be performed in a network environment in which some steps are performed by different computers in the networked environment.
Some embodiments are implemented by a computer system. A computer system may include a processor, a memory, and a non-transitory computer-readable medium. The memory and non-transitory medium may store instructions for performing methods and steps described herein.
Local network 150 may connect to network 140 through gateway 152. In some embodiments, the local network 150 may be private and access controlled so that entities on the network 140 cannot generally access the resources on local network 140. However, entities on the local network 150 may access and share at least some of the resources on the local network 150. Code storage 153 may comprise code stored on the local network 150 after having been web scraped from external code sources 110, 111. Code storage 154 may exist on the local network 150 and may store code from a team of programmers working from clients 157, 158, 159 on the local network 150. In an embodiment, a code storage 155 is an individual code storage that stores code of just one of the programmers on the team. The code storage 155 may be separate from code storage 154 or may be, for example, a subset of code storage 154. Code storage may be any kind of storage. In some embodiments, a code storage comprise a codebase, which is a collection of code for building one or a set of software systems, applications, or software components. Moreover, in some embodiments, a codebase comprises a code repository, where a repository keeps track of changes in the codebase over time and may allow version control and allowing checking in and checking out of code. In some embodiments, a code storage comprises a database. A database is any kind of storage and no particular type of database is required. For example, a database may comprise storage of files in memory or permanent storage. Server 156 may exist on the local network 150 and run a program comprised of code from the team of programmers. The code may be team code stored in code storage 154 or an individual branch or subset stored on code storage 155. Server 156 may generate logs or output during the execution of the program, and the logs or output may be retrieved by clients 157, 158, 159 for monitoring or debugging of the program.
Additional servers, clients, computer systems, and local networks may be connected to network 140. It should be understand that where the terms server, client, or computer system are used, this includes the use of networked arrangements of multiple devices operating as a server, client, or computer system. For example, distributed or parallel computing may be used.
The machine learning model 200 has internal parameters that determine its decision boundary and that determine the output that the machine learning model 200 produces. After each training iteration, comprising inputting the input object 210 of a training example in to the machine learning model 200, the actual output 208 of the machine learning model 200 for the input object 210 is compared to the desired output value 212. One or more internal parameters 202 of the machine learning model 200 may be adjusted such that, upon running the machine learning model 200 with the new parameters, the produced output 208 will be closer to the desired output value 212. If the produced output 208 was already identical to the desired output value 212, then the internal parameters 202 of the machine learning model 200 may be adjusted to reinforce and strengthen those parameters that caused the correct output and reduce and weaken parameters that tended to move away from the correct output.
The machine learning model 200 output may be, for example, a numerical value in the case of regression or an identifier of a category in the case of classifier. A machine learning model trained to perform regression may be referred to as a regression model and a machine learning model trained to perform classification may be referred to as a classifier. The aspects of the input object that may be considered by the machine learning model 200 in making its decision may be referred to as features.
After machine learning model 200 has been trained, a new, unseen input object 220 may be provided as input to the model 200. The machine learning model 200 then produces an output representing a predicted target value 204 for the new input object 220, based on its internal parameters 202 learned from training.
Machine learning model 200 may be, for example, a neural network, support vector machine (SVM), Bayesian network, logistic regression, logistic classification, decision tree, ensemble classifier, or other machine learning model. Machine learning model 200 may be supervised or unsupervised. In the unsupervised case, the machine learning model 200 may identify patterns in unstructured data 240 without training examples 206. Unstructured data 240 is, for example, raw data upon which inference processes are desired to be performed. An unsupervised machine learning model may generate output 242 that comprises data identifying structure or patterns.
A neural network may be comprised of a plurality of neural network nodes, where each node includes input values, a set of weights, and an activation function. The neural network node may calculate the activation function on the input values to produce an output value. The activation function may be a non-linear function computed on the weighted sum of the input values plus an optional constant. In some embodiments, the activation function is logistic, sigmoid, or a hyperbolic tangent function. Neural network nodes may be connected to each other such that the output of one node is the input of another node. Moreover, neural network nodes may be organized into layers, each layer comprising one or more nodes. An input layer may comprise the inputs to the neural network and an output layer may comprise the output of the neural network. A neural network may be trained and update its internal parameters, which comprise the weights of each neural network node, by using backpropagation.
A convolutional neural network (CNN) may be used in some embodiments and is one kind of neural network and machine learning model. A convolutional neural network may include one or more convolutional filters, also known as kernels, that operate on the outputs of the neural network layer that precede it and produce an output to be consumed by the neural network layer subsequent to it. A convolutional filter may have a window in which it operates. The window may be spatially local. A node of the preceding layer may be connected to a node in the current layer if the node of the preceding layer is within the window. If it is not within the window, then it is not connected. A convolutional neural network is one kind of locally connected neural network, which is a neural network where neural network nodes are connected to nodes of a preceding layer that are within a spatially local area. Moreover, a convolutional neural network is one kind of sparsely connected neural network, which is a neural network where most of the nodes of each hidden layer are connected to fewer than half of the nodes in the subsequent layer.
A recurrent neural network (RNN) may be used in some embodiments and is one kind of neural network and machine learning model. A recurrent neural network includes at least one back loop, where the output of at least one neural network node is input into a neural network node of a prior layer. The recurrent neural network maintains state between iterations, such as in the form of a tensor. The state is updated at each iteration, and the state tensor is passed as input to the recurrent neural network at the new iteration.
In some embodiments, the recurrent neural network is a long short-term (LSTM) memory neural network. In some embodiments, the recurrent neural network is a bi-directional LSTM neural network.
A feed forward neural network is another type of a neural network and has no back loops. In some embodiments, a feed forward neural network may be densely connected, meaning that most of the neural network nodes in each layer are connected to most of the neural network nodes in the subsequent layer. In some embodiments, the feed forward neural network is a fully-connected neural network, where each of the neural network nodes is connected to each neural network node in the subsequent layer.
A gated graph sequence neural network (GGSNN) is a type of neural network that may be used in some embodiments. In a GGSNN, the input data is a graph, comprising nodes and edges between the nodes, and the neural network outputs a graph. The graph may be directed or undirected. A propagation step is performed to compute node representations for each node, where node representations may be based on features of the node. An output model maps from node representations and corresponding labels to an output for each node. The output model is defined per node and is a differentiable function that maps to an output.
Neural networks of different types or the same type may be linked together into a sequential or parallel series of neural networks, where subsequent neural networks accept as input the output of one or more preceding neural networks. The combination of multiple neural networks may comprise a single neural network and may be trained from end-to-end using backpropagation from the last neural network through the first neural network.
A compiler or interpreter 320 may compile the code 310 into executable instructions or an intermediate representation, or interpret the source code 310 for execution. The compiler/interpreter 320 may comprise a namespace 322 that can be used to store symbols, such as identifiers and types, and to allow for name resolution 330. In some embodiments, the compiler/interpreter 320 may comprise a scanner 324, parser 326, semantic checker 328, name resolver 330, and code generator 332. Scanner 324 may accept as input the source code 310 and split expressions and language statements into tokens that can be processed by the parser 326 to determine the grammatical structure of a program. A token may be a single element of a programming language such as a constant, identifier, operator, separator, reserved word, or other element. In some embodiments, a token is atomic and is the smallest semantic unit of a programming language, such that the token cannot be broken down further into units with semantic meaning in the language. The parser 326 may parse the tokens and organize them according to a grammar of a programming language. In some embodiments, parser 326 builds a parse tree. Semantic checker 328 may perform semantic checking of a computer program and may identify and throw errors that are semantic in nature. The name resolver 330 may resolve names in the parse tree to elements of the namespace 322. Code generator 332 may translate the parse tree, or other intermediate representation of the source code, into a target language. The target language may be executable instructions, such as a binary executable, or an intermediate language that may be interpreted for execution. In an execution environment 370, code may be executed, such as for testing or production.
Programming co-pilot system 340 may interact with the programming environment 300, source code 310, and compiler/interpreter 320 to provide programming assistance to the programmer. Programming co-pilot 340 may include a monitoring system 380 to monitor user actions in an editor 302 and system events such as inputs, outputs, and errors. Programming co-pilot 340 may also include a journal 382, which may comprise a digital record of the history of data, such as sequential changes to and versions of source code, user interactions in the editor 302, user interactions in other parts of a system such as a terminal or web browser, system events, and other data. The journal 382 may record data sequentially so that a sequence of events may be exactly reconstructed. Programming co-pilot may include functionalities such as error detection system 342, error prediction system 344, error classification system 346, fix detection system 348, fix prediction system 350, training data generator 352, and other features. Programming co-pilot 340 may include machine learning model 384 to power its functionality, including learning algorithms 386 that learn from data or rule-based systems 388 that use hard-coded rules or heuristics. Although illustrated as one unit, multiple machine learning models 384 may be used in practice to perform or implement different functionality. For example, each function may have a separate machine learning model. Programming co-pilot system 340 may interface with the programming environment 300 through API calls, data streams, inter-process messages, shared data structures, or other methods. In some embodiments, the programming co-pilot system 340 is a separate program from the programming environment 300. In other embodiments, the programming co-pilot is a sub-program or component of the programming environment 300.
An embodiment of a programming co-pilot system 340 and its various functionality will be described herein. The programming co-pilot system 340 may include various combinations of the features described herein. In some embodiments, it includes all the functionalities described herein, and, in other embodiments, it includes only a subset of the functionalities described.
Embodiments may operate on any kind of source code including imperative programming languages, declarative code, markup languages, scripting languages. and other code. For example, source code may be Python, Perl, PHP, Javascript, Java, C, C++, HTML, reStructuredText, Markdown, CSS, shell scripts (such as bash, zsh, etc.), and so on.
In an embodiment, the programming co-pilot system 340 includes an error detection system 342. The error detection system 342 may monitor the programming environment 300, the compiler 320, and execution environment 370, identify when errors are reported, record error messages and context such as the portion of the source code 310 which is likely to have created and the error or the state of the computer program at the time of the error, and identify whether the error is properly handled or requires attention. Errors may include syntax errors such as misspelling or incorrect sequencing of tokens, semantic errors such as type mismatch or calling variables out of scope, execution errors such as index out of bounds or out of memory, logic errors such as providing the wrong output for a test case or running an infinite loop, or other errors. Errors may be identified at compile time during compilation of the code by the compiler 320, at run time during execution of compiled code, or at test time when execution results are compared against default values or user defined test cases.
In an embodiment, the programming co-pilot system includes an error classification system 346. The error classification system 346 may take an error indication as input and may predict one or more error types to which the error indication may belong. In some embodiments, the error indication may be an error message returned by the programming environment 300, execution environment 370, or the compiler 320, such as the error message identified in step 403. In some embodiments, the error indication may include additional information such as the error context captured by the error detection system 342 in step 404. The error types to be predicted may be defined by a user, defined by configuration settings, or automatically clustered by the error classification system 346.
In an embodiment, the programming co-pilot system includes an error prediction system 344. The error prediction system 344 may take a portion of code as input and predict the likelihood that an error may occur. In addition, some or all of the codebase that contains the portion of code may be input to the error prediction system 344. The codebase as a whole may be used in the prediction process. In some embodiments, the error prediction system 344 may also predict one or more types of error that may occur. In an embodiment, the error types to be predicted comprise the same error types which may be identified through classification by the error classification system 346 in method 500. In some embodiments, the error prediction system 344 may also assign a confidence interval to its predictions. Moreover, the error prediction system 344 may also predict a location in the code portion that is causing the error, such as by identifying line numbers or positions.
In an embodiment, the programming co-pilot system includes a fix detection system 348. The fix detection system tracks edits to the code, monitors for errors, and records further edits to the code which are associated with fixing the errors.
In an embodiment, further methods may be used to identify a change that fixed the error in the code. In the course of receiving edits from a user, there may be many edits that are made between two versions of code, and it would be desirable to isolate the edit that causes an error to be resolved. As described above, a first copy of code having an error and a second copy of code not having the error may be identified based on compiling and running the code. However, there may be multiple changes between the first copy of code and second copy of code. A method of identifying the change between the first copy of code and second copy of code that fixed the error may be performed automatically or manually.
In one embodiment, the change that fixed the code may be identified automatically. A data set of examples of code where the error was resolved may be stored. These code examples may be analyzed using heuristics, machine learning, or statistical methods to identify common changes that were made to each of these examples. The common changes may be likely to be the relevant portion of code that resolved the error. Changes made in these code examples that are not common between the code examples may be more likely to not be the portions of code that resolved the error and may be ignored, in some embodiments.
In one embodiment, the change that fixed the code may be identified with user interface elements for manual input. In one embodiment, one or more user interface components may be displayed to the user for identifying which one of a plurality of changes to code was the one that fixed the error. In some embodiments, the user interface components may be selectors or menus. In other embodiments, the user interface components may allow entering an identifier of the code that fixed the error.
In one embodiment, the error prediction system 344 includes an option for users who are fixing their code to anonymize their fix for sharing. If the option is selected, the error prediction system 344 may automatically remove private information from the source code, or a portion thereof, that are not required for the fix. Using a share option, the fix, including the source code or a portion thereof, may be shared with other users such as in a community or discussion board. Private information that may be removed or changed may include, for example, variable or function names, removal of portions of code, changing or redacting of portions of code, and so on.
In one embodiment, the error prediction system 344 prompts the user to enter a written explanation of the fix. The written explanation may be stored with the text. It may be shared in an online community or discussion board. It may also be stored for further reference by the user.
In an embodiment, the programming co-pilot system includes a fix prediction system 350. The fix prediction system may take a code sample and optionally additional error information as input and provide a user with one or more options to fix errors in the code sample.
In an embodiment, the programming co-pilot system may include a training data generator 352. The training data generator may create additional training examples comprising corresponding sets of code samples with errors, additional error information, and corrected code.
In an embodiment, sharing the fixes may store the fixes for display in a community. The community may be public or private and may be accessible through, for example, a website, an application, or other methods. The community may be a part of the programming co-pilot system 340 or may be a separate component. The shared fixes in the community may comprise a data set that may provide examples of fixes that users may reference in order to address future errors and fixes. The fixes may include associated data such as source code or portions thereof that include the fix. In addition, fixes may include associated user discussion or user-written explanations about the fixes.
The community may include storage of any or all of the data collected by the programming co-pilot system 340. In an embodiment, the community stores a data set of error messages, stack traces, and code of users. This information may be used for error canonicalization and error prediction. In addition, the raw data about subsequent edits may be stored, which may be used for isolating fixes in code and may help with fix prediction. Moreover, manually isolated fixes in code may also be stored.
The community may also include tools for finding other users who can act as experts about an error or fix. In an embodiment, the community may include a display of users who have previously encountered an error or have used a particular fix. In one embodiment, the community may store the specific fix used by a user. In addition, an edit stream showing the edits that a user made to the source code to implement the fix may be stored and be available for display. One or more methods of filtering users may be provided. For example, users performing a query may filter other users, who are potential experts, based on how recently they got an error or used the fix, how frequently they got an error or used the fix, on their experience with parts of the codebase that contain the error or fix, the amount of overlap of experience between the querying user and the expert user, whether the querying user and the expert user have worked together before, whether the expert user is in the querying user's chat list, and whether the expert user is online.
In an embodiment, the programming co-pilot system 340 may also store a journal for the user. In an embodiment, the journal may comprise the private data of the user, some of which may be automatically collected by the programming co-pilot system 340. For example, the journal may store information about each of the prior errors encountered by the user and the associated fixes. For each error encountered by the user, the system may store an associated fix that fixed the error. It may also stored an edit stream of the edits that led to the fix. The user may then query to find past errors and how he or she fixed them. In some embodiments, a manual query is not required. When the system detects that the user has encountered an error, it may automatically search the journal for prior instances where the user encountered the error. It may then return an associated fix or other information such as a prior stack trace, relevant discussions or explanations, web links to relevant information, and so on.
The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.
The example computer system 1100 includes a processing device 1102, a main memory 1104 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a static memory 1106 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 1118, which communicate with each other via a bus 1130.
Processing device 1102 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, or the like. More particularly, the processing device may be complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 1102 may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 1102 is configured to execute instructions 1126 for performing the operations and steps discussed herein.
The computer system 1100 may further include a network interface device 1108 to communicate over the network 1120. The computer system 1100 also may include a video display unit 1110 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 1112 (e.g., a keyboard), a cursor control device 1115 (e.g., a mouse), a graphics processing unit 1122, a signal generation device 1116 (e.g., a speaker), graphics processing unit 1122, video processing unit 1128, and audio processing unit 1132.
The data storage device 1118 may include a machine-readable storage medium 1124 (also known as a computer-readable medium) on which is stored one or more sets of instructions or software 1126 embodying any one or more of the methodologies or functions described herein. The instructions 1126 may also reside, completely or at least partially, within the main memory 1104 and/or within the processing device 1102 during execution thereof by the computer system 1100, the main memory 1104 and the processing device 1102 also constituting machine-readable storage media.
In one implementation, the instructions 1126 include instructions to implement functionality corresponding to the components of a device to perform the disclosure herein. While the machine-readable storage medium 1124 is shown in an example implementation to be a single medium, the term “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure. The term “machine-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media and magnetic media.
It will be appreciated that the present disclosure may include any one and up to all of the following examples.
Example 1: A computer-implemented method comprising: monitoring a communication channel, the communication channel for collecting error information of an execution environment, compiler, or interpreter, the execution environment configured to execute computer code; parsing content in the communication channel and identifying an error; identifying and storing error context of the error; determining whether the error was handled successfully or needs attention.
Example 2: The computer-implemented method of Example 1, wherein identifying an error in the content of the communication channel is performed by a machine learning algorithm,
Example 3: The computer-implemented method of Example 2, wherein the machine learning algorithm is a neural network.
Example 4: The computer-implemented method of Example 1, wherein the error context comprises a stack trace.
Example 5: The computer-implemented method of Example 4, wherein the stack trace indicates a portion of source code and the error context further includes the indicated portion of source code.
Example 6: The computer-implemented method of Example 1, wherein evaluating the error status of the error is performed by a machine learning algorithm.
Example 7: The computer-implemented method of Example 6, wherein the machine learning algorithm is a neural network.
Example 8: A computer-implemented method comprising: providing a plurality of training examples, each training example comprising error text and an error type label; determining one or more features from the error text in the training examples, wherein the features are based on one or more words or tokens in the error text; training a machine learning model using the plurality of training examples; providing an unseen error text; inputting the unseen error text into the machine learning model; generating, by the machine learning model, a prediction of an error type label of the unseen error text.
Example 9: The computer-implemented method of Example 8, wherein the machine learning model is a neural network.
Example 10: The computer-implemented method of Example 8, wherein the error type labels of the plurality of training examples are generated by a clustering algorithm.
Example 11: The computer-implemented method of Example 8, wherein the error type labels of the training examples are extracted from error text.
Example 12: The computer-implemented method of Example 8, wherein the error type labels of the training examples are generated by hand labeling.
Example 13: The computer-implemented method of Example 8, further comprising: wherein the training examples further comprise a stack trace; determining one or more features from the stack traces in the training examples.
Example 14: A computer-implemented method comprising: providing a portion of source code in a code editor; processing the portion of source code to extract a plurality of features of the source code, wherein the plurality of features are based on the source code; inputting the features into a machine learning model; generating, by the machine learning model, a prediction of whether the portion of source code contains an error.
Example 15: The computer-implemented method of Example 14, wherein the machine learning model is a neural network.
Example 16: The computer-implemented method of Example 14, further comprising: when the machine learning model predicts that the source code contains an error, predicting an error type of the error.
Example 17: The computer-implemented method of Example 14, further comprising: when the machine learning model predicts that the source code contains an error, predicting an error text of the error.
Example 18: The computer-implemented method of Example 14, wherein the machine learning model is trained using labeled training examples that are web scraped from online data sources, wherein the online data sources comprise crowd-sourced data.
Example 19: The computer-implemented method of Example 18, wherein a plurality of the labeled training examples include an identification of a type of error in the training code portion.
Example 20: The computer-implemented method of Example 14, further comprising generating a prediction of a location in the source code containing the error.
Example 21: The computer-implemented method of Example 14, further comprising generating a prediction of a stack trace of the error
Example 22: The computer-implemented method of Example 14, further comprising: displaying the prediction and an interface element for accepting the prediction; when the prediction is not accepted in a threshold amount of time, dismissing the prediction.
Example 23: The computer-implemented method of Example 22, further comprising: generating a training set for the machine learning model based on a plurality of predictions being accepted or not accepted.
Example 24: A computer-implemented method comprising: launching a code editor; loading source code from memory and displaying it in the code editor; monitoring and storing a first set of edits made to the source code by a programmer; receiving a first request to compile the source code into an executable program; receiving an indication of an error in the source code as a result of running the executable program; monitoring and storing a second set of edits made to the source code by the programmer; receiving a second request to compile the source code into an executable program, and successfully running the executable program; displaying the second set of edits and an indication that the second set of edits corrects the error in the source code.
Example 25: A computer-implemented method comprising: providing a portion of source code, wherein the portion of source code contains an error; providing error information about the error in the source code; inputting the portion of source code and error information into a machine learning model, the machine learning model trained on a plurality of training examples, the plurality of training examples comprising code portions, information about errors in the code portions, and fixes of the errors, wherein the fixes of the errors comprise sequences of changes to the code portions; generating, by the machine learning model, a predicted fix for the error in the source code.
Example 26: The computer-implemented method of Example 25, wherein the machine learning model is a neural network.
Example 27: The computer-implemented method of Example 20, wherein the machine learning model is trained using backpropagation.
Example 28: The computer-implemented method of Example 20, wherein the error is a runtime error.
Example 29: The computer-implemented method of Example 20, wherein the error is a compile-time error.
Example 30: A computer-implemented method comprising: receiving a code portion, wherein the code portion is error-free; making a sequences of pseudo-random changes to the code portion to create an error-containing code portion; compiling the error-containing code portion and receiving a compile-time error, the compile-time error including error information; reverting the pseudo-random changes to obtain a sequence of fixing changes; training a machine learning model to predict fixing changes for the compile-time error by using the sequence of fixing changes, the compile-time error, the error information, and the error-containing code portion as a training example.
Example 31: The computer-implemented method of Example 30, further comprising: training the machine learning model using a plurality of training examples.
Example 32: The computer-implemented method of Example 30, wherein the machine learning model is a neural network.
Example 33: A computer-implemented method comprising: displaying a code editor; monitoring for user input and modifying source code in the code editor; detecting an error in the source code; displaying links to a plurality of online resources related to the error, the plurality of online resources including crowd-sourced information about the error; displaying, in a suggested fix container, a list of suggested changes to the source code to fix the error; displaying, in a fix history container, past changes made by a first group of other users to fix the error.
Example 34: The computer-implemented method of Example 33, wherein the list of suggested changes to the source code are shown using markup.
Example 35: The computer-implemented method of Example 33, further comprising providing an option to share changes made to the source code to share the changes with a second group of other users.
Example 36: The computer-implemented method of Example 35, wherein at least some of the source code is anonymized before sharing the changes with the second group of other users.
Example 37: The computer-implemented method of Example 35, further comprising presenting an interface for receiving user annotations about changes made to the source code.
Example 38: A computer-implemented method comprising: providing a portion of source code in a code editor; processing the portion of source code to extract a plurality of features of the source code, wherein the plurality of features are based on the source code; inputting the features into a machine learning model; generating, by the machine learning model, a prediction of whether the portion of source code contains an error.
Example 39: The computer-implemented method of Example 38, further comprising: inputting the features and the error into a second machine learning model for fix prediction; generating, by the second machine learning model, a prediction of a fix to the error.
Example 40: A computer-implemented method comprising: monitoring a communication channel, the communication channel for collecting error information of an execution environment, compiler, or interpreter, the execution environment configured to execute computer code; parsing content in the communication channel and identifying an error; identifying and storing error context of the error; determining whether the error was handled successfully or needs attention.
Example 41: The computer-implemented method of claim 41, further comprising: storing a record of the error in a journal.
Example 42: The computer-implemented method of claim 40, further comprising: detecting, at a second time, an error; searching the journal for prior occurrences of the error; displaying one or more of the prior occurrences to a user.
Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “identifying” or “determining” or “executing” or “performing” or “collecting” or “creating” or “sending” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage devices.
The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the intended purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.
Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the method. The structure for a variety of these systems will appear as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the disclosure as described herein.
The present disclosure may be provided as a computer program product, or software, that may include a machine-readable medium having stored thereon instructions, which may be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as a read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices, etc.
A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.
Smith, Adam, Hung, Daniel, Lozano, Juan, Upadhyaya, Tarak
Patent | Priority | Assignee | Title |
11132512, | Nov 08 2019 | International Business Machines Corporation; The Board of Trustees of the University of Illinois | Multi-perspective, multi-task neural network model for matching text to program code |
11226858, | Dec 24 2020 | SALESFORCE, INC | Root cause analysis of logs generated by execution of a system |
11494291, | Dec 20 2019 | UiPath, Inc.; UIPATH, INC | System and computer-implemented method for analyzing test automation workflow of robotic process automation (RPA) |
11580012, | Jul 07 2020 | Accenture Global Solutions Limited | Smart test case generator |
11663070, | Dec 24 2020 | SALESFORCE, INC | Root cause analysis of logs generated by execution of a system |
11928473, | Sep 29 2020 | BEIJING TSINGMICRO INTELLIGENT TECHNOLOGY CO., LTD. | Instruction scheduling method and system for reconfigurable array processor |
11966326, | Jul 20 2022 | Bank of America Corporation | Development test automation framework |
Patent | Priority | Assignee | Title |
10067815, | Jun 21 2016 | International Business Machines Corporation | Probabilistic prediction of software failure |
10459695, | Oct 05 2017 | SAP SE | Correction of code errors using machine learning |
10482000, | Apr 24 2017 | Microsoft Technology Licensing, LLC | Machine learned decision guidance for alerts originating from monitoring systems |
7228404, | Jan 18 2000 | ADVANCED SILICON TECHNOLOGIES, LLC | Managing instruction side-effects |
7519871, | Nov 16 2005 | GLOBALFOUNDRIES Inc | Plug-in problem relief actuators |
7877642, | Oct 22 2008 | GLOBALFOUNDRIES Inc | Automatic software fault diagnosis by exploiting application signatures |
9069737, | Jul 15 2013 | Amazon Technologies, Inc | Machine learning based instance remediation |
20060026467, | |||
20060143548, | |||
20070168753, | |||
20090013208, | |||
20100269091, | |||
20100325618, | |||
20110072418, | |||
20130152047, | |||
20140033174, | |||
20150227838, | |||
20150278074, | |||
20160306689, | |||
20170039132, | |||
20170063910, | |||
20180082183, | |||
20180239658, | |||
20180276562, | |||
20190391904, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Nov 13 2018 | HUNG, DANIEL | Manhattan Engineering Incorporated | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 051664 | /0543 | |
Nov 27 2018 | SMITH, ADAM | Manhattan Engineering Incorporated | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 051664 | /0543 | |
Nov 27 2018 | LOZANO, JUAN | Manhattan Engineering Incorporated | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 051664 | /0543 | |
Dec 17 2018 | UPADHYAYA, TARAK | Manhattan Engineering Incorporated | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 051664 | /0543 | |
Sep 23 2019 | Manhattan Engineering Incorporated | (assignment on the face of the patent) | / | |||
Jun 01 2021 | Manhattan Engineering Incorporated | AFFIRM, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 056548 | /0888 |
Date | Maintenance Fee Events |
Sep 23 2019 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Sep 23 2019 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Oct 07 2019 | SMAL: Entity status set to Small. |
Oct 07 2019 | SMAL: Entity status set to Small. |
Jul 22 2024 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Date | Maintenance Schedule |
May 11 2024 | 4 years fee payment window open |
Nov 11 2024 | 6 months grace period start (w surcharge) |
May 11 2025 | patent expiry (for year 4) |
May 11 2027 | 2 years to revive unintentionally abandoned end. (for year 4) |
May 11 2028 | 8 years fee payment window open |
Nov 11 2028 | 6 months grace period start (w surcharge) |
May 11 2029 | patent expiry (for year 8) |
May 11 2031 | 2 years to revive unintentionally abandoned end. (for year 8) |
May 11 2032 | 12 years fee payment window open |
Nov 11 2032 | 6 months grace period start (w surcharge) |
May 11 2033 | patent expiry (for year 12) |
May 11 2035 | 2 years to revive unintentionally abandoned end. (for year 12) |