International Business Machines Corporation
Automatically extracting feature engineering knowledge from execution traces

Abstract:

A method is provided for creating a directed graph. The method obtains an execution trace having a sequence of invocations of table transformations. Each invocation is classified as a call-with-object invocation or call-with-column invocation. The call-with-object invocation indicates that an invoked function has object IDs as at least one of input and output information. The call-with-column invocation indicates that the invoked function has column names as the at least one of input and output information. The method processes the trace to build the graph. Each node indicates an object or a column and each edge indicates data flow. The method performs, in response to all invocations being processed, an automated graph reduction process by eliminating given nodes indicating a respective object such that that every two nodes that connect with each other, via a respective one of the given nodes as an intermediate node, can keep the connection therebetween.