Linux will remember your passphrase after you have entered it once. Unfortunately, this is not the case in Cygwin. Thanks to this post and this post, I found the following workaround which lets Cygwin asks for the passphrase only once at login. This is not exactly the same behavior as in Linux, when the passphrase is asked for when needed (not at every login), but it is close.
Just add the following script to your .bash_profile: