Move libffi configuration after basic toolchain setup