In this paper, we study a problem of control system design for small-scale helicopter that has been applied to a robotic helicopter project. The structure of the mathematical models of single-rotor helicopter and the description of its constituent elements are presented. The general mathematical model of a helicopter is a complex multivariable system. This model consists of nonlinear differential equations of the helicopter dynamics, the kinematics and auxiliary equations. The control forces and moments, and also the external disturbances, that affecting on helicopter flight, are in the right side of the dynamic equations. It is necessary to have experimental data for helicopter flight parameters to get adequate auxiliary equations. Those equations have been applied to associate the control forces and moments, to control positions of actuators. In this paper we present the experimental results, estimation algorithms and data-processing. The experimental study includes a series of test flights combining with sets of basic flight configurations, such as take-off/landing, take-off/hovering and etc. The control signals have been recorded during flight tests. There are a set of signals connected to actuators of collective/cyclic pitch mechanism, tail rotor and engine. In postprocessing procedure we are suggesting approaches of data smoothing and filtering. The procedure includes estimation algorithms of the helicopter mass center, forces and moments that affecting on the helicopter. Those estimations are based on helicopter dynamics. This paper presents the results of helicopter mathematical model study, algorithms of helicopter control system synthesis and autopilot structure. The helicopter control system is based on algorithms of position-trajectory control. The hardware and software parts of autopilot are described in the paper. This paper shows results of hardware-in-the-loop (HIL) simulation of helicopter autopilot performance including the mathematical model and external wind disturbances.