feature engineering
In plain wordsTurning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'.
The design of a model's input variables (featuresfeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself.) from raw data, drawing on knowledge of the problem and on exploratory analysis to create, transform and select them. Typical operations include encoding categories (one-hot encodingone-hot encoding, n.: Turning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers., target encoding); scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units. and transforming numbers (logarithms, binning, polynomial terms); filling gaps (imputationimputation, n.: Filling in missing values in a dataset with sensible estimates, such as the typical value or a prediction from the other columns, so the data can be analyzed or used for training.); extracting parts of dates, text and coordinates; aggregating histories, such as a customer's purchases in the last 30 days; forming ratios and interactions between variables; and selectingfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate. the most useful subset. For tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this. and classical models such as gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., feature engineering is often the largest single source of accuracy gains. Deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract. shifted much of this work into representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand., letting networks learn features from raw pixels, audio or text, but engineered features remain central in tabular, time-series and production systems, where they are often computed and shared through a feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live.. Careless feature construction is a frequent cause of data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints..
Most of the lift came from feature engineering: days since last login and the ratio of refunds to orders.
Origin
The practice is as old as statistical modeling, in which analysts have always hand-crafted variables. The phrase became common in machine learning in the late 2000s and 2010s, spread especially by Kaggle competitions; Pedro Domingos's widely read 2012 essay 'A Few Useful Things to Know about Machine Learning' argued that feature engineering is the key to most successful projects.
Usage note
Contrast with representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand., in which a model learns its own features from raw data instead of having them designed by people.
Related
- featurefeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself.
- feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate.
- one-hot encodingone-hot encoding, n.: Turning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers.
- feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units.
- imputationimputation, n.: Filling in missing values in a dataset with sensible estimates, such as the typical value or a prediction from the other columns, so the data can be analyzed or used for training.
- feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live.
- data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
Sources
- Domingos, P. (2012). A Few Useful Things to Know about Machine Learning. Communications of the ACM 55(10).
- Zheng, A. & Casari, A. (2018). Feature Engineering for Machine Learning. Sebastopol: O'Reilly Media.
Mentioned in
deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract., AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., data cleaningdata cleaning, n.: Finding and fixing errors in a dataset — typos, duplicates, missing values, impossible numbers, wrong labels — before using it, because flawed data produces flawed results., data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints., end-to-end learningend-to-end learning, n.: Training one AI system to go straight from raw input, such as camera images, to the final output, such as steering commands, instead of chaining hand-designed steps., featurefeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself., feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., feature selectionfeature selection, n.: Choosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate., imputationimputation, n.: Filling in missing values in a dataset with sensible estimates, such as the typical value or a prediction from the other columns, so the data can be analyzed or used for training., KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science., machine learning engineermachine learning engineer, n.: An engineer who turns machine-learning ideas into working systems: preparing data, training and testing models, and running them reliably inside real products., one-hot encodingone-hot encoding, n.: Turning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers., pandaspandas, prop. n.: A free Python library for loading, cleaning, combining and summarizing tables of data, like a programmable spreadsheet; it is a standard first tool for preparing data for machine learning., representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand., tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this., feature storefeature store, n.: A shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live.