Functionality engineering occupies a unique place in the field of data science. For most supervised and unsupervised learning deployments (which include the majority of enterprise cognitive computing efforts), this process of determining the characteristics of the training data that influence the accuracy of predictive modeling is the guardian to unlock the wonders of statistical artificial intelligence.

Other processes before and after feature generation (such as data preparation or model management) are required to ensure accurate machine learning models. Yet without knowing which data traits are critical to achieving a model’s goal – such as predicting a candidate’s risk of default on a loan – organizations can’t move on to the later stages of data science, making the previous ones unnecessary.

Therefore, feature engineering is one of the most indispensable tasks in creating machine learning models. The demanding nature of this process is based on:

• Labeled training data: The large amounts of training data for supervised and unsupervised learning is one of its business inhibitors. This concern is compounded by the lack of training data labeled for specific model objectives.
• Data preparation: Even when there is enough training data available, simply cleaning, transforming, integrating, and modeling that data is one of the most labor-intensive data science tasks.
• Engineering manipulations: There is a comprehensive range of data science tools and techniques to determine the characteristics, which also require a lot of work.

Each of these factors makes feature engineering a long and tedious process, without which most automatic learning is impossible. As such, there are a number of emerging and established approaches to data science to overcome this barrier or make it much less intrusive.

According to Cambridge semantics CTO Sean Martin, “In some ways feature engineering is starting to be less interesting because nobody wants to do that hard work.” This sentiment is particularly significant in light of graph database approaches to speed up the feature engineering process, or to avoid it altogether with the incorporation of graphs, to achieve the same results faster, faster, and more. cheaper.

The integration alternative

The incorporation of graphs enables organizations to overcome the difficulties of feature engineering while discerning the characteristics of the data with the greatest influence on the accuracy of advanced analysis models. With “incorporating graphs, you don’t need to do a lot of feature engineering for this,” Martin revealed. “You basically use the functionality of the chart as is to learn embedding.” According to Martin, embedding graphs is the process of transforming a graph into vectors (numbers) that correctly capture the connections or topology of the graph so that data scientists can perform the mathematical transformations supporting machine learning.

For example, if there is a graph of knowledge about mortgages and risk, data scientists can use integration to vectorize that data, and then use those vectors for machine learning transformations. Thus, they learn the characteristics of the model from the graphic vectors while eliminate the critical need for labeled training data—One of the biggest barriers to machine learning. Frameworks like Apache Arrow can cut and paste graphical data into data science tools that perform embedding; eventually, users will be able to embed directly into competing knowledge graph solutions.

Faster feature engineering

The underlying graphical environment supporting this embedding process is also helpful in transforming the efficiency of traditional feature engineering, making it much more accessible to the business. Part of this utility comes from the data graphics modeling capabilities. Semantic graph technology is based on on standardized data models all types of data adhere, which is crucial to speeding up aspects of the data preparation phase, because “you can more easily integrate data from multiple sources,” observed Martin. This ease of integration is directly responsible for including more sources for machine learning training datasets and determining their relationships to each other, which provides additional inputs not gleaned from individual sources.

“You now get more signal sources and integrating them can give you a signal that you won’t get in separate data sources,” Martin said. Additionally, the inherent nature of graph parameters – they provide a rich and nuanced contextualization of relationships between nodes – is extremely useful for identifying features. Martin pointed out that in graph environments, features are potentially links or connections between entities and their attributes, both of which are described with semantic techniques. The simple analysis of these connections leads to significant inputs for machine learning models.

Speed ​​up feature engineering

In addition to integrating graphs and analyzing links between entities to verify functionality, data integration and analysis readiness platforms built on graph databases provide query capabilities. automatic to speed up the feature engineering process. According to Martin, this process usually involves creating an attribute table from relevant data and “one of those columns is the one you want to make predictions on.”

Automatic generation of queries speeds up this business because it “allows you to quickly engineer features against a combination of data,” Martin said. “You can quickly create extractions from your chart, where each column is part of your entity that you are modeling.” Automated queries also allow users to visually create large tables from different parts of the chart, allowing them to work with their data faster. The result is an improved ability to “experience the functionality you want to extract more quickly,” Martin said.

Automatic data profiling

The ability to automatically generate queries for feature engineering is the ability to automatically profile data in graphical environments to speed up the feature selection process. Data profiling “shows you what kind of data is in the chart and it gives you very detailed statistics on each dimension of that data, as well as samples,” Martin noted. Automated data profiling naturally speeds up this dimension of data science This is often necessary to simply understand how data can relate to a specific machine learning use case. This form of automation naturally complements that relating to the generation of queries. A data scientist can take this statistical information “and that can be used when you start to create your entity table that you are going to extract,” Martin said. “You can do this kind of work hand in hand by looking at data profiling.”

The future of features

Features are the definitive data characteristics that enable machine learning models to accurately make predictions and prescriptions. In this regard, they form the foundation of the statistical branch of AI. However, the effort, time, and resources to generate these features can become obsolete by simply learning them with graph integration so that data scientists are no longer dependent on hard-to-find labeled training data. The ramifications of this development could potentially extend the use cases of supervised and unsupervised learning, making machine learning much more common in the enterprise than it is currently.

Alternatively, there are other ways for graphical platforms to speed up feature engineering (depending on their integration, automatic data profiling, and automatic query generation mechanisms) so that it requires much less time. , energy and resources than before. Both approaches make machine learning more practical and useful for organizations, expanding the value of data science as a discipline. “The biggest problem of all is putting the data together, cleaning it up, and extracting it so that you can engineer the functionality on it,” Martin said. “An accelerator for your machine learning project is essential.”

About the Author

Jelani Harper is an editorial consultant serving the information technology market. He specializes in data-driven applications focused on semantic technologies, data governance and analytics.

Sign up for free insideBIGDATA newsletter.

Join us on Twitter: @ InsideBigData1 – https://twitter.com/InsideBigData1