This was the problem that I couldn't solve in the past with IIS FTP server. Then I gave up using IIS and turned to FileZilla Server instead. Unfortunately, I have to use IIS FTP server again and this time I have no choice of migrating to others. Well, after figuring out for a while, I found that the reason why FileZilla couldn't retrieve directory listing is Windows Firewall is blocking its request. Since FileZilla uses passive mode as a default, it need to initiates data connection to random port which > 1023 and specified by server. Here, the problem occurs; Windows Firewall blocked anything if they are not in the exception. That's why we have such a delay before Error occurs in FileZilla.
Well, the easiest solution here is forcing FileZilla to use 'active transfer mode' instead. Thus, client (FileZilla client) will not have to initiate data connection. It's server's job to do so. In brief, firewall is not a problem no more here and we are happy using FTP although it's not that secure protocol, but gets the job done quite well.
Alternatively, we could get passive transfer mode work here too, but we need to open so many ports [of course, it's random] waiting for initiation from clients. We could specify the range by command:-
C:\Inetpub\AdminScript\adsutil.vbs /MSFTPSVC/PassivePortRange "8400-8420"
Then add exception for all those ports (8400-8420 as example above). I just don't think it's a good way to add too many exceptions to firewall :-P It's just a choice; choose what suits best to yours.