osmo_bsc_main: integrate MGW pooling into osmo-bsc

Since the libosmo-mgcp-client now supports MGW pooling, lets use this
feature in osmo-bsc. Large RAN installations may benefit from
distributing the RTP voice stream load on multiple media gateways.

Depends: osmo-mgw Icaaba0e470e916eefddfee750b83f5f65291a6b0
Change-Id: I8f33ab2cea04b545c403a6fe479aa963a0fc0d0d
Related: SYS#5091

9 files changed, 330 insertions, 18 deletions

diff --git a/doc/manuals/chapters/mgwpool.adoc b/doc/manuals/chapters/mgwpool.adoc
new file mode 100644
index 000000000..7d9e9a681
--- /dev/null
+++ b/doc/manuals/chapters/mgwpool.adoc
@@ -0,0 +1,243 @@
+[[mgw_pooling]]
+== MGW Pooling
+
+OsmoBSC is able to use a pool of media gateway (MGW) instances since mid 2021.
+The aim of MGW pooling is to evenly distribute the RTP voice stream load across
+multiple MGW instances. This can help to scale out over multiple VMs or physical
+machines. Until osmo-mgw includes multithreading support, it may also be used to
+scale-out to multiple cores on a single host.
+
+The load distribution is managed in such a way that when a new call is placed,
+the pool will automatically assign the call to the MGW with the lowest load.
+
+MGW pooling is recommended for larger RAN installations. For small networks and
+lab installations the classic method with one MGW per BSC offers sufficient
+performance.
+
+=== VTY configuration
+
+In OsmoBSC the MGW is controlled via an MGCP-Client. The VTY commands to
+configure the MGCP-Client are part of the 'msc' node due to historical reasons.
+Unfortunately this concept does not allow to configure multiple MGCP-Client
+instances as required by MGW pooling. In order to support MGW pooling a new
+'mgw' node has been added under the 'network' node.
+
+=== Existing configuration files
+
+Existing OsmoBSC configuration files will continue to work, but as soon as an
+MGW pool is configured the 'mgw' settings under the 'msc' node will be ignored.
+
+Example configuration with only one MGCP-Client under the 'msc' node:
+----
+msc 0
+ mgw remote-ip 127.0.0.1
+ mgw remote-port 2428
+----
+
+=== MGW pool configuration
+
+To setup an MGW pool, the user must first install multiple OsmoMGW instances, so
+that they won’t interfere with each other. This can be done using different
+local host IP addresses or different ports. When OsmoMGW is installed from
+packages, the systemd configuration may also require adjustment.
+
+The VTY settings under the 'mgw' node works the same way as the VTY settings for
+the MGW under the 'msc' node.
+
+Example configuration with two MGCP-Client instances in a pool:
+----
+ mgw 0
+  mgw remote-ip 127.0.0.1
+  mgw remote-port 2432
+  mgw local-ip 127.0.0.1
+  mgw local-port 2431
+  mgw endpoint-domain mgw0 <1>
+ mgw 1
+  mgw remote-ip 127.0.0.1
+  mgw remote-port 2430
+  mgw local-ip 127.0.0.1
+  mgw local-port 2429
+  mgw endpoint-domain mgw1 <1>
+----
+
+<1> When working with multiple MGW / MGCP-Client instances, the domain name for
+each MGW should be different. Otherwise it won't be possible to distinguish the
+endpoint names in the log. It should also be noted that the domain name must
+match the configuration of the related OsmoMGW instance.
+
+=== MGW pool management
+
+While it was not possible to change the MGCP-Client configuration under the
+“msc” node at runtime, the pool is fully runtime-manageable. The only limitation
+is that an MGCP-Client can not be restarted or removed as long as it is serving
+calls (see also: <<mgw_pooling_blocking>>).
+
+==== MGW pool status
+
+The VTY implements a 'show mgw-pool' command that lists the currently configured
+MGW pool members, their status and call utilization.
+
+----
+OsmoBSC> show mgw-pool
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 1
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 0
+----
+
+==== Adding an MGW / MGCP-Client to the MGW pool
+
+To add a new MGCP-Client to the pool, the 'mgw' node is used. Like with the
+'bts' or the 'msc' node a reference number is used that usually starts at 0.
+However it is still possible to assign any number from 0-255. The enumeration
+also may contain gaps.
+
+----
+OsmoBSC> enable
+OsmoBSC# configure terminal
+OsmoBSC(config)# network
+OsmoBSC(config-net)# mgw 2
+OsmoBSC(config-mgw)# mgw
+  local-ip         local bind to connect to MGW from
+  local-port       local port to connect to MGW from
+  remote-ip        remote IP address to reach the MGW at
+  remote-port      remote port to reach the MGW at
+  endpoint-domain  Set the domain name to send in MGCP messages, e.g. the part 'foo' in 'rtpbridge/*@foo'.
+  reset-endpoint   Add an endpoint name that should be reset (DLCX) on connect to the reset-endpoint list,e.g. 'rtpbridge/*'
+----
+
+The newly added MGW will immediately appear in the mgw-pool list but it won't
+be used until its configuration finished by reconnecting it.
+
+----
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 2
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 3
+%  MGW 2
+%   mgcp-client:   disconnected
+%   service:       unblocked
+%   ongoing calls: 0
+----
+
+==== Reconnecting an MGW / MGCP-Client
+
+It may become necessary to reconnect an MGCP-Client. This is the case when the
+VTY configuration was changed at runtime. In order to make the changes effective
+the MGW configuration must be reloaded by reconnecting the MGW connection. Also
+newly created MGW instances require a reconnect once their configuration is
+done.
+
+To reconnect an MGCP-Client use the 'reconnect' VTY command:
+----
+OsmoBSC# mgw 2 reconnect
+----
+
+The mgcp-client status should immediately change to 'connected'. The MGW is now
+ready to be used for new calls.
+
+----
+OsmoBSC# show mgw-pool
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 2
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 3
+%  MGW 2
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 0
+----
+
+It should be noted that MGCP a protocol is used via UDP, the connect only
+happens locally to forward the UDP datagrams properly. Also (unless a reset
+endpoint is configured like in the example config above) there will be no
+immediate interaction with the MGW. However, the log should at least confirm
+the the connect worked and the MGCP client has been created successfully.
+
+----
+Mon Aug  2 17:15:00 2021 DLMGCP mgcp_client.c:788 MGCP client: using endpoint domain '@mgw'
+Mon Aug  2 17:15:00 2021 DLMGCP mgcp_client.c:908 MGCP GW connection: r=127.0.0.1:2427<->l=127.0.0.1:2727
+----
+
+It is strongly advised to check the activity on the related MGW and to follow
+the log in order to see that the communication between OsmoBSC and the MGW is
+working correctly.
+
+[[mgw_pooling_blocking]]
+==== Blocking an MGW / MGCP-Client
+
+If it becomes apparent that an MGCP-Client must be restarted or removed from
+the config (maintenance) the operator can put that MGCP-Client into a blocked
+mode. A blocked MGCP-Client will still serve the ongoing calls but it will not
+be picked for the assignment of new calls.
+
+To block an MGCP-Client use the 'block' VTY command:
+----
+OsmoBSC# mgw 2 block
+OsmoBSC# show mgw-pool
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 11
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 12
+%  MGW 2
+%   mgcp-client:   connected
+%   service:       blocked
+%   ongoing calls: 10
+----
+
+When the number of ongoing calls has tapered off, the MGW / MGCP-Client can be
+restarted or removed if necessary.
+
+----
+OsmoBSC# show mgw-pool
+% MGW-Pool:
+%  MGW 0
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 15
+%  MGW 1
+%   mgcp-client:   connected
+%   service:       unblocked
+%   ongoing calls: 14
+%  MGW 2
+%   mgcp-client:   connected
+%   service:       blocked
+%   ongoing calls: 0
+----
+
+If the blockade should be reverted, the 'unblock' VTY command can be used in
+the same way to remove the blockade. (Reconnecting will not remove the
+blockade.)
+
+==== Removing an MGW / MGCP-Client
+
+An MGCP-Client is removed from the pool using the 'no mgw' command from the
+configure terminal. The MGCP-Client instance will automatically be terminated
+and the related resources are freed. The only requirement is that there are no
+ongoing calls on the selected instance.
+
+----
+OsmoBSC# configure terminal
+OsmoBSC(config)# network
+OsmoBSC(config-net)# no mgw 2
+----
diff --git a/doc/manuals/chapters/running.adoc b/doc/manuals/chapters/running.adoc
index 327e0bd7f..8e1c8e94a 100644
--- a/doc/manuals/chapters/running.adoc
+++ b/doc/manuals/chapters/running.adoc
@@ -143,6 +143,9 @@ msc 0
 DLCX to the media gateway. This helps to clear lingering calls from the
 media gateway when the OsmoBSC is restarted.
 
+NOTE: OsmoBSC is also able to handle a pool of media gateways for load
+distribution. See also <<mgw_pooling>>.
+
 
 ==== Configure Lb to connect to an SMLC
 
diff --git a/doc/manuals/osmobsc-usermanual.adoc b/doc/manuals/osmobsc-usermanual.adoc
index c7589cc35..120fe2c21 100644
--- a/doc/manuals/osmobsc-usermanual.adoc
+++ b/doc/manuals/osmobsc-usermanual.adoc
@@ -32,6 +32,8 @@ include::{srcdir}/chapters/smscb.adoc[]
 
 include::{srcdir}/chapters/mscpool.adoc[]
 
+include::{srcdir}/chapters/mgwpool.adoc[]
+
 include::{srcdir}/chapters/smlc.adoc[]
 
 include::./common/chapters/qos-dscp-pcp.adoc[]
diff --git a/include/osmocom/bsc/gsm_data.h b/include/osmocom/bsc/gsm_data.h
index db0f2497f..bed36f3e9 100644
--- a/include/osmocom/bsc/gsm_data.h
+++ b/include/osmocom/bsc/gsm_data.h
@@ -1203,8 +1203,15 @@ struct gsm_network {
 	struct osmo_timer_list bts_store_uptime_timer;
 
 	struct {
+		/* Single MGCP client configuration under msc node (also required for
+		 * MGCP proxy when sccp-lite is used) */
 		struct mgcp_client_conf *conf;
-		struct mgcp_client *client;
+
+		/* MGW pool, also includes the single MGCP client as fallback if no
+		 * pool is configured. */
+		struct mgcp_client_pool *mgw_pool;
+
+		/* Timer definitions, the same for all MGW pool members */
 		struct osmo_tdef *tdefs;
 	} mgw;
 
diff --git a/include/osmocom/bsc/vty.h b/include/osmocom/bsc/vty.h
index 9ea983541..70b973db3 100644
--- a/include/osmocom/bsc/vty.h
+++ b/include/osmocom/bsc/vty.h
@@ -23,6 +23,7 @@ struct buffer *vty_argv_to_buffer(int argc, const char *argv[], int base);
 
 enum bsc_vty_node {
 	GSMNET_NODE = _LAST_OSMOVTY_NODE + 1,
+        MGW_NODE,
 	BTS_NODE,
 	TRX_NODE,
 	TS_NODE,
diff --git a/src/osmo-bsc/bsc_subscr_conn_fsm.c b/src/osmo-bsc/bsc_subscr_conn_fsm.c
index afb633dc3..1e61d3f78 100644
--- a/src/osmo-bsc/bsc_subscr_conn_fsm.c
+++ b/src/osmo-bsc/bsc_subscr_conn_fsm.c
@@ -45,6 +45,7 @@
 #include <osmocom/bsc/assignment_fsm.h>
 #include <osmocom/bsc/codec_pref.h>
 #include <osmocom/mgcp_client/mgcp_client_endpoint_fsm.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 #include <osmocom/core/byteswap.h>
 #include <osmocom/bsc/lb.h>
 #include <osmocom/bsc/lcs_loc_req.h>
@@ -518,39 +519,47 @@ struct osmo_mgcpc_ep *gscon_ensure_mgw_endpoint(struct gsm_subscriber_connection
 						uint16_t msc_assigned_cic, struct gsm_lchan *for_lchan)
 {
 	const char *epname;
+	struct mgcp_client *mgcp_client = NULL;
 
 	if (conn->user_plane.mgw_endpoint)
 		return conn->user_plane.mgw_endpoint;
 
+	if (gscon_is_sccplite(conn) || gscon_is_aoip(conn)) {
+		/* Get MGCP client from pool */
+		mgcp_client = mgcp_client_pool_get(conn->network->mgw.mgw_pool);
+		if (!mgcp_client) {
+			LOGPFSML(conn->fi, LOGL_ERROR,
+				 "cannot ensure MGW endpoint -- no MGW configured, check configuration!\n");
+			conn->user_plane.mgw_endpoint = NULL;
+			return NULL;
+		}
+	}
+
 	if (gscon_is_sccplite(conn)) {
 		/* derive endpoint name from CIC on A interface side */
 		conn->user_plane.mgw_endpoint =
 			osmo_mgcpc_ep_alloc(conn->fi, GSCON_EV_FORGET_MGW_ENDPOINT,
-					    conn->network->mgw.client,
+					    mgcp_client,
 					    conn->network->mgw.tdefs,
 					    conn->fi->id,
 					    "%x@%s", msc_assigned_cic,
-					    mgcp_client_endpoint_domain(conn->network->mgw.client));
+					    mgcp_client_endpoint_domain(mgcp_client));
 		LOGPFSML(conn->fi, LOGL_DEBUG, "MGW endpoint name derived from CIC 0x%x: %s\n",
 			 msc_assigned_cic, osmo_mgcpc_ep_name(conn->user_plane.mgw_endpoint));
 
 	} else if (gscon_is_aoip(conn)) {
-
 		if (is_ipaccess_bts(for_lchan->ts->trx->bts))
 			/* use dynamic RTPBRIDGE endpoint allocation in MGW */
-			epname = mgcp_client_rtpbridge_wildcard(conn->network->mgw.client);
+			epname = mgcp_client_rtpbridge_wildcard(mgcp_client);
 		else {
-			epname = mgcp_client_e1_epname(conn, conn->network->mgw.client, for_lchan->ts->e1_link.e1_nr,
+			epname = mgcp_client_e1_epname(conn, mgcp_client, for_lchan->ts->e1_link.e1_nr,
 						       for_lchan->ts->e1_link.e1_ts, 16,
 						       for_lchan->ts->e1_link.e1_ts_ss*2);
 		}
 
 		conn->user_plane.mgw_endpoint =
-			osmo_mgcpc_ep_alloc(conn->fi, GSCON_EV_FORGET_MGW_ENDPOINT,
-					    conn->network->mgw.client,
-					    conn->network->mgw.tdefs,
-					    conn->fi->id,
-					    "%s", epname);
+			osmo_mgcpc_ep_alloc(conn->fi, GSCON_EV_FORGET_MGW_ENDPOINT, mgcp_client,
+					    conn->network->mgw.tdefs, conn->fi->id, "%s", epname);
 	} else {
 		LOGPFSML(conn->fi, LOGL_ERROR, "Conn is neither SCCPlite nor AoIP!?\n");
 		return NULL;
@@ -801,6 +810,12 @@ void gscon_forget_lchan(struct gsm_subscriber_connection *conn, struct gsm_lchan
 
 static void gscon_forget_mgw_endpoint(struct gsm_subscriber_connection *conn)
 {
+	struct mgcp_client *mgcp_client;
+
+	/* Put MGCP client back into MGW pool */
+	mgcp_client = osmo_mgcpc_ep_client(conn->user_plane.mgw_endpoint);
+	mgcp_client_pool_put(mgcp_client);
+
 	conn->user_plane.mgw_endpoint = NULL;
 	conn->user_plane.mgw_endpoint_ci_msc = NULL;
 	conn->ho.created_ci_for_msc = NULL;
diff --git a/src/osmo-bsc/bsc_vty.c b/src/osmo-bsc/bsc_vty.c
index 17fdb45ca..5ccd78487 100644
--- a/src/osmo-bsc/bsc_vty.c
+++ b/src/osmo-bsc/bsc_vty.c
@@ -37,6 +37,7 @@
 #include <osmocom/gsm/gsm0502.h>
 
 #include <osmocom/mgcp_client/mgcp_client_endpoint_fsm.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 
 #include <osmocom/bsc/vty.h>
 #include <osmocom/bsc/gsm_data.h>
@@ -3384,6 +3385,7 @@ int bsc_vty_init(struct gsm_network *network)
 	install_element(GSMNET_NODE, &cfg_net_nri_null_del_cmd);
 
 	bts_vty_init();
+        mgcp_client_pool_vty_init(GSMNET_NODE, MGW_NODE, " ", vty_global_gsm_network->mgw.mgw_pool);
 
 	install_element(ENABLE_NODE, &drop_bts_cmd);
 	install_element(ENABLE_NODE, &restart_bts_cmd);
diff --git a/src/osmo-bsc/osmo_bsc_main.c b/src/osmo-bsc/osmo_bsc_main.c
index b3bd365f9..14cbc2260 100644
--- a/src/osmo-bsc/osmo_bsc_main.c
+++ b/src/osmo-bsc/osmo_bsc_main.c
@@ -57,6 +57,7 @@
 #include <osmocom/vty/cpu_sched_vty.h>
 
 #include <osmocom/mgcp_client/mgcp_client_endpoint_fsm.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 
 #include <osmocom/abis/abis.h>
 #include <osmocom/bsc/abis_om2000.h>
@@ -70,6 +71,7 @@
 #include <osmocom/bsc/bsc_stats.h>
 
 #include <osmocom/mgcp_client/mgcp_client.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 
 #include <osmocom/sigtran/xua_msg.h>
 
@@ -867,6 +869,41 @@ extern void *tall_fle_ctx;
 extern void *tall_tqe_ctx;
 extern void *tall_ctr_ctx;
 
+static int bsc_mgw_setup(void)
+{
+	struct mgcp_client *mgcp_client_single;
+	unsigned int pool_members_initalized;
+
+	/* Initalize MGW pool. This initalizes and connects all MGCP clients that are currently configured in
+	 * the pool. Adding additional MGCP clients to the pool is possible but the user has to configure and
+	 * (re)connect them manually from the VTY. */
+	pool_members_initalized = mgcp_client_pool_connect(bsc_gsmnet->mgw.mgw_pool);
+	if (pool_members_initalized) {
+		LOGP(DNM, LOGL_NOTICE,
+		     "MGW pool with %u pool members configured, (ignoring MGW configuration in VTY node 'msc').\n",
+		     pool_members_initalized);
+		return 0;
+	}
+
+	/* Initialize and connect a single MGCP client. This MGCP client will appear as the one and only pool
+	 * member if there is no MGW pool configured. */
+	LOGP(DNM, LOGL_NOTICE, "No MGW pool configured, using MGW configuration in VTY node 'msc'\n");
+        mgcp_client_single = mgcp_client_init(bsc_gsmnet, bsc_gsmnet->mgw.conf);
+	if (!mgcp_client_single) {
+		LOGP(DNM, LOGL_ERROR, "MGW (single) client initalization failed\n");
+		return -EINVAL;
+	}
+	if (mgcp_client_connect(mgcp_client_single)) {
+		LOGP(DNM, LOGL_ERROR, "MGW (single) connect failed at (%s:%u)\n",
+		     bsc_gsmnet->mgw.conf->remote_addr,
+		     bsc_gsmnet->mgw.conf->remote_port);
+		return -EINVAL;
+	}
+	mgcp_client_pool_register_single(bsc_gsmnet->mgw.mgw_pool, mgcp_client_single);
+
+	return 0;
+}
+
 int main(int argc, char **argv)
 {
 	struct bsc_msc_data *msc;
@@ -897,6 +934,7 @@ int main(int argc, char **argv)
 	}
 
 	bsc_gsmnet->mgw.conf = talloc_zero(bsc_gsmnet, struct mgcp_client_conf);
+	bsc_gsmnet->mgw.mgw_pool = mgcp_client_pool_alloc(bsc_gsmnet);
 	mgcp_client_conf_init(bsc_gsmnet->mgw.conf);
 
 	bts_init();
@@ -1000,14 +1038,8 @@ int main(int argc, char **argv)
 		}
 	}
 
-	bsc_gsmnet->mgw.client = mgcp_client_init(bsc_gsmnet, bsc_gsmnet->mgw.conf);
-
-	if (mgcp_client_connect(bsc_gsmnet->mgw.client)) {
-		LOGP(DNM, LOGL_ERROR, "MGW connect failed at (%s:%u)\n",
-		     bsc_gsmnet->mgw.conf->remote_addr,
-		     bsc_gsmnet->mgw.conf->remote_port);
+	if (bsc_mgw_setup() != 0)
 		exit(1);
-	}
 
 	if (osmo_bsc_sigtran_init(&bsc_gsmnet->mscs) != 0) {
 		LOGP(DNM, LOGL_ERROR, "Failed to initialize sigtran backhaul.\n");
diff --git a/tests/handover/handover_test.c b/tests/handover/handover_test.c
index f76a70071..a355d3b0e 100644
--- a/tests/handover/handover_test.c
+++ b/tests/handover/handover_test.c
@@ -52,6 +52,7 @@
 #include <osmocom/bsc/bts.h>
 #include <osmocom/bsc/paging.h>
 #include <osmocom/bsc/vty.h>
+#include <osmocom/mgcp_client/mgcp_client_pool.h>
 
 #include "../../bscconfig.h"
 
@@ -1617,6 +1618,12 @@ int main(int argc, char **argv)
 	if (!bsc_gsmnet)
 		exit(1);
 
+	/* The MGCP client which is handling the pool (mgcp_client_pool_vty_init) is used from the bsc_vty_init, so
+	 * we must allocate an empty mgw pool even though we do not need it for this test. */
+	bsc_gsmnet->mgw.mgw_pool = mgcp_client_pool_alloc(bsc_gsmnet);
+	if (!bsc_gsmnet->mgw.mgw_pool)
+		exit(1);
+
 	vty_init(&vty_info);
 	bsc_vty_init(bsc_gsmnet);
 	ho_test_vty_init();