Causal inference from observational data, particularly the estimation of a treatment’s causal effect on an outcome, has long been challenging, primarily because it hinges on correctly identifying confounders. This is typically accomplished in two main ways within causal inference frameworks: either by using causal discovery algorithms to recover the underlying causal structure through a causal graph, or by assuming that the relevant confounders are already known. Both approaches have been shown to be unreliable or simply infeasible in practical applications. Although large language models (LLMs) are advancing rapidly, their emerging capabilities in causal inference have only recently begun to receive significant attention. Nevertheless, LLMs currently lack the ability to directly interpret structured tabular data, which is widely used in causal inference. To address this limitation, we introduce a novel framework, CauExecutor, for causal inference. Our framework enables a novel combination of the semantic reasoning strength of LLM with the accurate estimation capacity of off-the-shelf statistical tools to more accurately estimate the causal effect from observational structured data. The CauExecutor first uses the semantic understanding and the reasoning power of LLMs to help find potential mediators and separate them from the confounders. It subsequently leverages off-the-shelf tools to programmatically handle tabular data and estimate causal effects by the optimized adjustment set. On several benchmark datasets, we observe CauExecutor outperforms all other LLM-based methods by correctly identifying more mediators and producing more accurate causal effect estimates. Additional experiments show that CauExecutor’s decision to disqualify mediators from the adjustment set, rather than qualifying any variable that meets the backdoor criterion, is beneficial to successfully minimizing mediator-induced bias and attaining improved estimation performance.