Hi WWF-Caroline,
Based on my knowledge, it should work.
As a tip, we need to make sure that the secondary ADFS server can also connect to the domain controllers successfully.
Moreover, for further professional assistance, you can also post the question in our Directory Service TechNet forum. I believe that the staff in the TechNet forum should be more professional about this kind of question.
Thanks,
Claud