Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection
Reinforcement Learning (RL) is a learning paradigm suitable for problems in which an agent has to maximize a given reward, while interacting with an ever-changing environment. This class of problem appears in several research topics of the 5th Generation (5G) and the 6th Generation (6G) of mobile ne...
| Autor principal: | BORGES, João Paulo Tavares |
|---|---|
| Grau: | Dissertação |
| Idioma: | eng |
| Publicado em: |
Universidade Federal do Pará
2024
|
| Assuntos: | |
| Acesso em linha: |
https://repositorio.ufpa.br/jspui/handle/2011/16551 |
| id |
ir-2011-16551 |
|---|---|
| recordtype |
dspace |
| spelling |
ir-2011-165512024-10-25T15:53:15Z Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection Simulações híbridas CAVIAR e aprendizagem por reforço aplicada para sistemas 5G: Experimentos com escalonamento e seleção de feixe BORGES, João Paulo Tavares KLAUTAU JÚNIOR, Aldebaro Barreto da Rocha http://lattes.cnpq.br/1596629769697284 Aprendizado por reforço Tecnologia 5G Simuladores híbridos Reinforcement learning Hybrid simulators CAVIAR (Communication networks, artificial intelligence and computer vision with 3D computer-generated imagery CNPQ::ENGENHARIAS::ENGENHARIA ELETRICA::TELECOMUNICACOES INTELIGÊNCIA COMPUTACIONAL TELECOMUNICAÇÕES Reinforcement Learning (RL) is a learning paradigm suitable for problems in which an agent has to maximize a given reward, while interacting with an ever-changing environment. This class of problem appears in several research topics of the 5th Generation (5G) and the 6th Generation (6G) of mobile networks. However, the lack of freely available data sets or environments to train and assess RL agents are a practical obstacle that delays the widespread adoption of RL in 5G and future networks. These environments must be able to close the so-called reality gap, where reinforcement learning agents, trained in virtual environments, are able to generalize their decisions when exposed to real, never before seen, situations. Therefore, this work describes a simulation methodology named CAVIAR, or Communication Networks, Artificial Intelligence and Computer Vision with 3D Computer-Generated Imagery, tailored for research on RL methods applied to the physical layer (PHY) of the wireless communications systems. In this work, this simulation methodology is used to generate an environment for the tasks of user scheduling and beam selection, where, at each time step, the RL agent needs to schedule a user and then choose the index of a fixed beamforming codebook to serve it. A key aspect of this proposal is that the simulation of the communication system and the artificial intelligence engine must be closely integrated, such that actions taken by the agent can reflect back on the simulation loop. This aspect makes the trade-off of processing time versus realism of the simulation, an element to be considered. This work also describes the modeling of the communication systems and RL agents used for experimentation, and presents statistics concerning the environment dynamics, such as data traffic, as well as results for baseline systems. Finally, it is discussed how the methods described in this work can be leveraged in the context of the development of digital twins. CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior Aprendizado por reforço, do inglês Reinforcement Learning (RL), é um paradigma de aprendizagem adequado para problemas em que um agente tem que maximizar uma determinada recompensa, enquanto interage com um ambiente em constante mudança. Esta classe de problema aparece em diversos tópicos de pesquisa da 5a Geração (5G) e da 6a Geração (6G) das redes móveis. No entanto, a falta de conjuntos de dados ou ambientes disponíveis gratuitamente para treinar e avaliar os agentes de RL é um obstáculo prático que atrasa a adoção de RL em redes 5G e futuras. Esses ambientes devem ser capazes de fechar o chamado reality gap, onde os agentes de aprendizagem por reforço, treinados em ambientes virtuais, são capazes de generalizar suas decisões quando expostos a situações reais, nunca antes vistas. Portanto, este trabalho descreve uma metodologia de simulação denominada CAVIAR, ou Communication Networks, Artificial Intelligence and Computer Vision with 3D Computer-Generated Imagery, voltada para pesquisa sobre métodos de RL aplicados à camada física (PHY) dos sistemas de comunicações sem fio. Neste trabalho, essa metodologia de simulação é utilizada para gerar um ambiente para as tarefas de escalonamento de usuários e seleção de feixes, onde, a cada passo, o agente RL precisa escalonar um usuário e então escolher o índice de um codebook de beamforming para atendê-lo. Um aspecto fundamental desta proposta é que a simulação do sistema de comunicação e o software de inteligência artificial devem estar intimamente integrados, de modo que as ações realizadas pelo agente possam refletir de volta no loop de simulação. Esse aspecto torna a compensação de tempo de processamento versus realismo da simulação, um elemento a ser considerado. Este trabalho também descreve a modelagem dos sistemas de comunicação e agentes RL usados para experimentação, e apresenta estatísticas sobre a dinâmica do ambiente, como tráfego de dados, bem como resultados para sistemas baseline. Por fim, é discutido como os métodos descritos neste trabalho podem ser aproveitados no contexto do desenvolvimento de gêmeos digitais. 2024-10-24T17:17:15Z 2024-10-24T17:17:15Z 2022-01-28 Dissertação BORGES, João Paulo Tavares. Hybrid CAVIAR simulations and reinforcement learning applied to 5G systems: experiments with scheduling and beam selection. Orientador: Aldebaro Barreto da Rocha Klautau Júnior. 2022. 73 f. Dissertação (Mestrado em Engenharia Elétrica) - Instituto de Tecnologia, Universidade Federal do Pará, Belém, 2022. Disponível em: https://repositorio.ufpa.br/jspui/handle/2011/16551 . Acesso em:. https://repositorio.ufpa.br/jspui/handle/2011/16551 eng Acesso Aberto Attribution-NonCommercial-NoDerivs 3.0 Brazil http://creativecommons.org/licenses/by-nc-nd/3.0/br/ application/pdf Universidade Federal do Pará Brasil Instituto de Tecnologia UFPA Programa de Pós-Graduação em Engenharia Elétrica 1 CD-ROM Disponível na internet via correio eletrônico: bibliotecaitec@ufpa.br |
| institution |
Repositório Institucional - Universidade Federal do Pará |
| collection |
RI-UFPA |
| language |
eng |
| topic |
Aprendizado por reforço Tecnologia 5G Simuladores híbridos Reinforcement learning Hybrid simulators CAVIAR (Communication networks, artificial intelligence and computer vision with 3D computer-generated imagery CNPQ::ENGENHARIAS::ENGENHARIA ELETRICA::TELECOMUNICACOES INTELIGÊNCIA COMPUTACIONAL TELECOMUNICAÇÕES |
| spellingShingle |
Aprendizado por reforço Tecnologia 5G Simuladores híbridos Reinforcement learning Hybrid simulators CAVIAR (Communication networks, artificial intelligence and computer vision with 3D computer-generated imagery CNPQ::ENGENHARIAS::ENGENHARIA ELETRICA::TELECOMUNICACOES INTELIGÊNCIA COMPUTACIONAL TELECOMUNICAÇÕES BORGES, João Paulo Tavares Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection |
| topic_facet |
Aprendizado por reforço Tecnologia 5G Simuladores híbridos Reinforcement learning Hybrid simulators CAVIAR (Communication networks, artificial intelligence and computer vision with 3D computer-generated imagery CNPQ::ENGENHARIAS::ENGENHARIA ELETRICA::TELECOMUNICACOES INTELIGÊNCIA COMPUTACIONAL TELECOMUNICAÇÕES |
| description |
Reinforcement Learning (RL) is a learning paradigm suitable for problems in which an agent has to maximize a given reward, while interacting with an ever-changing environment. This class of problem appears in several research topics of the 5th Generation (5G) and the 6th Generation (6G) of mobile networks. However, the lack of freely available data sets or environments to train and assess RL agents are a practical obstacle that delays the widespread adoption of RL in 5G and future networks. These environments must be able to close the so-called reality gap, where reinforcement learning agents, trained in virtual environments, are able to generalize their decisions when exposed to real, never before seen, situations. Therefore, this work describes a simulation methodology named CAVIAR, or Communication Networks, Artificial Intelligence and Computer Vision with 3D Computer-Generated Imagery, tailored for research on RL methods applied to the physical layer (PHY) of the wireless communications systems. In this work, this simulation methodology is used to generate an environment for the tasks of user scheduling and beam selection, where, at each time step, the RL agent needs to schedule a user and then choose the index of a fixed beamforming codebook to serve it. A key aspect of this proposal is that the simulation of the communication system and the artificial intelligence engine must be closely integrated, such that actions taken by the agent can reflect back on the simulation loop. This aspect makes the trade-off of processing time versus realism of the simulation, an element to be considered. This work also describes the modeling of the communication systems and RL agents used for experimentation, and presents statistics concerning the environment dynamics, such as data traffic, as well as results for baseline systems. Finally, it is discussed how the methods described in this work can be leveraged in the context of the development of digital twins. |
| author_additional |
KLAUTAU JÚNIOR, Aldebaro Barreto da Rocha |
| author_additionalStr |
KLAUTAU JÚNIOR, Aldebaro Barreto da Rocha |
| format |
Dissertação |
| author |
BORGES, João Paulo Tavares |
| title |
Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection |
| title_short |
Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection |
| title_full |
Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection |
| title_fullStr |
Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection |
| title_full_unstemmed |
Hybrid CAVIAR Simulations and Reinforcement Learning Applied to 5G Systems: Experiments with Scheduling and Beam Selection |
| title_sort |
hybrid caviar simulations and reinforcement learning applied to 5g systems: experiments with scheduling and beam selection |
| publisher |
Universidade Federal do Pará |
| publishDate |
2024 |
| url |
https://repositorio.ufpa.br/jspui/handle/2011/16551 |
| _version_ |
1816948613956239360 |
| score |
11.755432 |