modReLU is an activation that is a modification of a ReLU. It is a pointwise nonlinearity, $\sigma\_{modReLU}\left(z\right) : C \rightarrow C$, which affects only the absolute value of a complex number, defined as: $$ \sigma\_{modReLU}\left(z\right) = \left(|z| + b\right)\frac{z}{|z|} \text{ if } |z| + b \geq 0 $$ $$ \sigma\_{modReLU}\left(z\right) = 0 \text{ if } |z| + b \leq 0 $$ where $b \in \mathbb{R}$ is a bias parameter of the nonlinearity. For a $n\_{h}$ dimensional hidden space we learn $n\_{h}$ nonlinearity bias parameters, one per dimension.
A Unitary RNN is a recurrent neural network architecture that uses a unitary hidden to hidden matrix. Specifically they concern dynamics of the form: $$ h\_{t} = f\left(Wh\_{t−1} + Vx\_{t}\right) $$ where $W$ is a unitary matrix $\left(W^{†}W = I\right)$. The product of unitary matrices is a unitary matrix, so $W$ can be parameterised as a product of simpler unitary matrices: $$ h\_{t} = f\left(D\_{3}R\_{2}F^{−1}D\_{2}PR\_{1}FD\_{1}h\_{t−1} + Vxt\right) $$ where $D\_{3}$, $D\_{2}$, $D\_{1}$ are learned diagonal complex matrices, and $R\_{2}$, $R\_{1}$ are learned reflection matrices. Matrices $F$ and $F^{−1}$ are the discrete Fourier transformation and its inverse. P is any constant random permutation. The activation function $f\left(h\right)$ applies a rectified linear unit with a learned bias to the modulus of each complex number. Only the diagonal and reflection matrices, $D$ and $R$, are learned, so Unitary RNNs have fewer parameters than LSTMs with comparable numbers of hidden units. Source: Associative LSTMs