Thanks Kirk, I already received some additional info from Larry.
The encoder seems to be ok, the problem is in the software setup. There are two specific issues:
1) The Magic RDS 4 currently does not support multiple network adapters in the same PC. In other words, you cannot choose the adapter directly in the application, the application uses such adapter for the connection which is assigned by Windows routing table. (The route can be set up in Windows if necessary.)
Thus, you should verify that the connection is really established between the Magic RDS 4 and the encoder. The Magic RDS 4 must indicate 'Connected' and the encoder must indicate the connection by the TCP LED. This is essential.
2) Although the encoder supports several methods of Ethernet setup, it is preferable to use the method as described in the manual (section 6.3.2), i.e. using the USB cable and the Magic RDS 4 - Device Setup - LAN Settings. The reason is simple - the Magic RDS 4 will not allow invalid setup for some advanced parameters (for example, the UART speed must be 19200, not 115200).
3) It was not mentioned yet (and I do not really believe that this is applicable here) but the power adapter may be wrong.
4) Theoretically, you may be experiencing Ethernet auto negotiation problem. Although I have never experienced it personally, auto negotiation failure is known to be relativelly frequent. It happens when link speed and duplex are not correctly arranged between the network devices, which causes dropped packets, slower throughput and session drops. Not all devices allow the auto negotiation configuration so common solution is to add an extra Ethernet switch between them (different brand/model).