We booked a mover, cancelled now he is threatening to sue us. Help?

Hi there, a moving company is threating to sue us and reposses our furniture. We had him booked to move on a Sunday in Calgary AB. We decided to move our selves and not use the mover. We did not tell him right away. He asked us to call to confirm the move on the Saturday before the booked move. This is when we told him we would not require his services. He said ok. Then a few hours later he told us he is going to sue us and repossess our furniture to recoup losses. We never signed anything. We sent him 2 emails stating our addresses and times we would like to move. He never replied to the emails. We did talk on the phone prior to the move and this is when he told us to call Saturday to confirm. We did not realize it was a confirm or be sued situation. He never told us a cancellation policy nor does his website state one. He never emailed us, or told us over the phone. We did not sign a contract but agreed on the phone. He is saying the emails we went confirms the contract not the phone call he asked us to make to confirm. What do you think? We are waiting right now to see what he does (if anything) before contacting a lawyer.