This content is not included in your SAE MOBILUS subscription, or you are not logged in.

Statistical Characterization, Pattern Identification, and Analysis of Big Data

Journal Article
2017-01-0236
ISSN: 1946-3979, e-ISSN: 1946-3987
Published March 28, 2017 by SAE International in United States
Statistical Characterization, Pattern Identification, and Analysis of Big Data
Sector:
Citation: Wei, Z. and Nikbin, K., "Statistical Characterization, Pattern Identification, and Analysis of Big Data," SAE Int. J. Mater. Manf. 10(3):293-305, 2017, https://doi.org/10.4271/2017-01-0236.
Language: English

Abstract:

In the Big Data era, the capability in statistical and probabilistic data characterization, data pattern identification, data modeling and analysis is critical to understand the data, to find the trends in the data, and to make better use of the data. In this paper the fundamental probability concepts and several commonly used probabilistic distribution functions, such as the Weibull for spectrum events and the Pareto for extreme/rare events, are described first. An event quadrant is subsequently established based on the commonality/rarity and impact/effect of the probabilistic events. Level of measurement, which is the key for quantitative measurement of the data, is also discussed based on the framework of probability. The damage density function, which is a measure of the relative damage contribution of each constituent is proposed. The new measure demonstrates its capability in distinguishing between the extreme/rare events and the spectrum events. Several case studies including vehicle reliability, vehicle road test score, warranty, salary distribution of an institution, the city population distribution in 3 countries, and the earthquake distribution worldwide and in the USA, are provided to demonstrate the role of the statistical and probabilistic approaches in the characterization and analysis of the big data.