With the recent advancement in sensing and controller technologies architecture design of an autonomous driving system becomes an important issue. Researchers have been developing different sensors and data processing technologies to solve the issues associated with fast processing, diverse weather, reliability, long distance recognition performance, etc. Necessary considerations of diverse traffic situations and safety factors of autonomous driving have also increased the complexity of embedded software as well as architecture of autonomous driving. In these circumstances, there are almost countless numbers of possible architecture designs. However, these design considerations have significant impacts on cost, controllability, and system reliability. Thus, it is crucial for the designers to make a challenging and critical design decision under several uncertainties during the conceptual design phase. This paper proposes an optimization-based robust architecture design framework for an autonomous driving system. The proposed framework focuses mainly on two design processes. The first one deals with the hardware integration issue. In this process, processors and buses need to be selected from an available hardware list and connected to realize the hardware system. The second one addresses the issue of proper allocation of software to the integrated components. In this process, computational tasks are allocated to the selected processors. Similarly, message transmission between different processors is also assigned through the selected buses. These architecture design issues are formulated as an integer programming (IP) problem to manage them simultaneously. Since the architecture design involves multiple objectives, the design issues are solved as a multi-objective optimization problem in order to identify the set of compromise optimal solutions, which ultimately minimize hardware cost as well as end-to-end latency under constrains of feasibility and safety. Furthermore, the proposed framework is expanded to a robust design method against software uncertainty. As a result, the risk of becoming an infeasible design is reduced by 75%.