Hi,
here are some information :
1) If you want to replicate from MS SQL Server you need to configure the Replication Agent for MS SQL , that comes with the heterogeneous edition of Replication Server but you can also get it as Replication Server Option for MS SQL.
The Replication Agent for MS SQL is a seperate process that runs on Windows. There is a quick starting guide that explains the steps how to set it up.
The error message occurs because you are using the rs_init utility and the "add new database" option does work only for SAP ASE as a source database and for MS SQL as source you have to define the new connection manually in the SRS.
2) If you need to replicate also from an Oracle instance , then you do not need another Replication Server, but you need here the specific Replication Agent for Oracle. ( Replication Server Option for Oracle.
3) If SAP IQ is the target you need also the license key for the Real Time Load ( RTL ) Option.
Hope that helps,
Thomas .