Big Data: It's Not Just About the Raw Data

Big Data technologies have become quite ubiquitous in the last years, allowing for the storage of substantial amounts of data, typically flight test data as recorded by the flight test installation. On recent helicopter prototypes, we generate in excess of 50 GB of raw data per flight hour, usually in a format not adequate for efficient large-scale processing. With some specific optimizations and the setup of a specialized infrastructure, there are now practicable means to store timeseries in ways that allow for requests spanning hundreds or thousands of flights to complete within minutes, opening the way to some substantial savings and new insights. However, to make the most of these data and make informed decisions it is often quite important to store contextual data that go beyond the pure timeseries data, typically on helicopters where optional installations can have a significant impact on aircraft performance or behavior. This paper explores the various kinds of data and metadata related with flight tests, how to collect them and relate them with one another in order to maximize raw data value and avoid some common pitfalls. We define more accurately the data of interest, where to collect them and some ideas to further improve data collection in the future.