pandas
In plain wordsA free Python library for loading, cleaning, combining and summarizing tables of data, like a programmable spreadsheet; it is a standard first tool for preparing data for machine learning.
An open-source Python library for data analysis built around the DataFrame, a two-dimensional table with labeled rows and typed columns, and the Series, a labeled one-dimensional array. It reads and writes CSV, Excel, JSON, SQL databases, Parquet and other formats, and provides vectorized operations for selecting, filtering, joining and reshaping data, handling missing values, grouping and aggregating ('split–apply–combine'), and working with time series. Built on NumPyNumPy, prop. n.: The basic free Python library for working quickly with large grids of numbers; nearly every data-science and machine-learning tool in Python is built on it or copies its style. and increasingly on Apache Arrow, pandas is the usual tool for exploratory analysis and feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'. in notebooksJupyter notebook, n.: An interactive document, opened in a web browser, that mixes runnable code with its results, charts and notes, so people can experiment step by step and share their work. before data passes to libraries such as scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way.. Its main limits are that data must fit in one machine's memory and most operations use a single CPU core; newer DataFrame libraries such as Polars, and distributed engines such as Apache Spark and Dask, address scale while often imitating its interface.
Load the logs into pandas, group by user and count sessions before building features.
Origin
Created by Wes McKinney in 2008 at the investment firm AQR Capital Management for financial data analysis and released as open source in 2009. The name derives from 'panel data', an econometrics term for multidimensional data sets.
Usage note
Conventionally written in lowercase. Its DataFrame concept follows the data frame of the R statistical language.
Related
- NumPyNumPy, prop. n.: The basic free Python library for working quickly with large grids of numbers; nearly every data-science and machine-learning tool in Python is built on it or copies its style.
- scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way.
- Jupyter notebookJupyter notebook, n.: An interactive document, opened in a web browser, that mixes runnable code with its results, charts and notes, so people can experiment step by step and share their work.
- feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'.
- data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material.
Sources
- McKinney, W. (2010). Data Structures for Statistical Computing in Python. Proceedings of the 9th Python in Science Conference (SciPy 2010).