The first prototype is when linked against libmcrypt 2.2.x, the second when linked against libmcrypt 2.4.x or higher. The mode should be either MCRYPT_ENCRYPT or MCRYPT_DECRYPT.
This function should not be used anymore, see mcrypt_generic() and mdecrypt_generic() for replacements.