Causal reasoning is the task to identify causal relations between a pair of events in a given context. However, causal reasoning in natural language remains a challenging task for large language models (LLMs), since they tend to mix correlation and causality and exhibit bias in their reasoning, especially by mistaking temporal proximity for causal relations. The problem is exacerbated by the models’ propensity to generate spurious justifications that confuses co-occurrence rather than actual causal relationships. Although CoT prompting has shown effectiveness in enhancing multi-step reasoning, it is prone to hallucination and spurious inferences, which generally dampens their capability to provide correct causal explanations. The variant of CoT, CoT-SC, is a more promising attempt at yielding consistent outputs by randomly sampling multiple reasoning paths, and voting for the most probable answer. However, for its implementation, CoT-SC also demands expensive computations. The prompting strategy that we propose in the current work, named as CF-CoT, aims to enable the LLMs to leverage causal reasoning on the explicit consideration of counterfactual worlds that they may have ignored to facilitate the reasoning task, mirroring how humans use “what-if” reasoning in day-to-day thinking. Our framework motivates models to distinguish real causal relations from mere correlations by studying how changing things hypothetically would affect them. Comprehensive experiments on two event-level causal reasoning benchmarks—EventStoryLine and e-CARE—show that CF-CoT, to some extent, enhance the performance in terms of causal inference accuracy and robustness over standard CoT and other baselines.