The evolution of mobile wireless networks towards their Fifth Generation (5G) establishment and beyond (Andrews et al., 2014; Nikolich et al., 2017) has seen an increasingly massive number of connected devices ensuing from the Internet of Things (IoT) and Industry 4.0 phenomena. According to the report in (Lucero, 2016), the amount of wireless equipments will outreach 75 billions by 2025. Among many key enabling technologies (Boccardi et al., 2014) such as massive multi-input multi output (MIMO), millimeter wave (mmWave), full-duplex (FD), 5G promotes Ultra Dense Network (UDN) technology (Ge et al., 2016) as one significant candidate to fulfill its promises on enhancing user’s throughput, catering seamless coverage, and reducing overall system power. In general, UDNs spatially deploys a massive number of small cell base stations (BSs) in the proximity of local users to reduce communication range, thus can ubiquitously guarantee undisrupted services to cell edge users, significantly improve area spectral efficiency (SE) and efficiently conserve transmit power. Despite many inherited benefits, dense deployment of classical small cell architecture quickly raises the operational cost (OPEX) due to a huge amount of fiber backhaul establishment to the core networks, which causes new economical and deployment challenges.

Recently, wireless backhaul (WB) technology has emerged as a cost-effective and pragmatic architectural solution, which enables the transportation of backhaul data over-the-air to the remote small cell without the needs of fiber connections. Therefore, it can potentially sustain the operation of dense networks. In the context of 5G, WB in dense heterogeneous networks (HetNets) is found most applicable and advantageous when being accommodated under the mmWave, microwave, and sub–6 Gigahertz spectrum bands to exploit both Line-of-Sight (LoS) and Non-Line-of-Sight (NLoS) communications. Especially in the last case, NLoS WB can fully exploit the availability of existing hardwares and developed wireless access (WA) technologies for immediate WB’s experiment towards its future deployment. Despite the introduced advantages, this new WB scheme must ensure the quality of backhaul rate as high as the conventional wired ones so as to assist the WA communications between the small cells and local users. Note that in the sub–6 Gigahertz case, simultaneous WB communications are concurrently operated over the same spectral channel, also known as Point-to-Multi-Point (PtMP) in-band WB. Under such circumstance, WB signals are severely interfered by other concurrent WB and WA transmissions. This becomes a fundamental bottleneck of PtMP WB HetNets and makes it more difficult to manage a proper system resource allocation to respect the stringent WB-WA rate constraint while optimizing the network performance. Motivated by these observations, this dissertation aims at constructing several novel designs which jointly optimize the resource allocation in NLoS PtMP WB HetNets to improve WB’s operation and consolidate its viability in 5G dense networks.

Network Densification
The term network densification refers to the dense deployment of small cells over the cellular coverage (Bhushan et al., 2014). Small cell architecture was first proposed to deploy small size, low power, short range base stations, capable of serving indoor and local users, to coexist with the underlying macrocell BSs which are mainly dedicated to serve outdoor mobile users (Chandrasekhar et al., 2008). Towards the establishment of 5G and its subsequent generations, small cell and macrocell BSs are likely to be equipped with a large scale of antenna array, also known as massive MIMO, embraced by mmWave technologies to cope with the expected increase of 1000× user’s data traffic, 10 − 20× SE and system energy-efficiency (EE) (Andrews et al., 2014). In fact, due to the characteristics of wireless signal propagation, improving wireless transmissions by massive MIMO operated at mmWave band is compatible with the short-range communications, which subsequently leads to small cell coverage shrinkage (Ge et al., 2014). Since this shrinkage creates several out-of-service spots (Ge et al., 2014),  Upon the newest 3GPP Release 15 of 5G standard along with the proliferation of IoT and Industry 4.0, UDNs embraced by massive MIMO and mmWave will remain as a dominant candidate to support the expected user’s demand of diverse QoS requirements.

In small cell HetNets, the essential component which maintains the backbone of the entire network is the backhaul architecture. Backhaul generally represents the family of connections which spans from the BSs to the core networks. It is important to note that the backhaul traffic of small cell BSs are often aggregated at the macrocell BS (donor node (Siddique et al., 2015b)) via wired or wireless backhaul connectivities before being routed to the core networks.

Wired Backhaul
A classical wired backhaul connection prefers fiber optic material to bridge the necessary links between the BSs and core networks since fiber backhaul can support high capacity and low delay (Ranaweera et al., 2013). At the macrocell tier, wired backhaul connections can be distinguished based on the physical connection distances between the BSs and the core networks as follows (Wang et al., 2015b):
– More than ten kilometers: optical transport network (OTN) with wavelength-division multiplexing (WDM).
– Less than ten kilometers: point-to-point (PtP) or PtMP unified passive optical network (UniPON) which uses an optical splitter to aggregate WDM signals from the backhaul links of multiple cells.

At the small cell tier, small cell BSs can take advantage of the existing indoor wired infrastructure such as digital subscriber loop (DSL) over copper wire (Dahrouj et al., 2015), which is capable of providing up to 1 Gb/s of data rate when more sophisticated technology such as G.fast is employed (Wang et al., 2015b). If there exists a fiber optic connectivity nearby, the small cell traffic via DSL link can be multiplexed into the main fiber optic link and transported towards the aggregated MBS node before forwarding to the core network. In most situations, ubiquitous wired backhaul connections are often deployed less ubiquitously due to their expensive cost. For a dense small cell network, this becomes excessive and impractical, and hence a more viable backhaul solution is always anticipated.

Wireless Backhaul
WB emerges as a replaceable and cost-effective technology which more flexibly and efficiently adapts with the backhauling demands compared to the wired backhaul. In fact, WB concept is not entirely new and was used to relay information between the MBS and core networks in the traditional homogeneous cellular networks. In particular, WB LoS transmissions based on PtP microwave or free space optic (FSO) (Dahrouj et al., 2015) have been shown to achieve comparatively high capacity as the fiber optic ones; and in some cases, it can be used for small cell backhaul if LoS condition is feasible. In general, when small cells are densely deployed in urban area with several surrounding giant buildings and blockages, NLoS transmissions are more favorable and practical for WB small cell BSs. However, NLoS transmissions often occur in the sub–6 GHz band where conventional wireless access transmissions preoccupy. Thus, accommodating WB technology here could create many concurrent transmissions and degrade each transmission’s quality without a proper network resource planning.