Soichiro Nishimori

CV / Google Scholar / GitHub / Twitter (X)

portrait.png

About Me

I am Soichiro Nishimori, a PhD student at Sugiyama-Yokoya-Ishida lab supervised by Prof. Sugiyama. Also, I am working as a research part-timer in Imperfect-information Learning Team at RIKEN AIP.

I previously interned at OMRON SINIC X Corporation supervised by Dr. Yoshitaka Ushiku and Dr. Atsushi Hashimoto and Matsuo-Iwasawa lab at The University of Tokyo supervised by Prof. Paavo Parmas.

Mahjong 🀄️ and Tennis 🎾 lover.

Research Interests

I am interested in scalable reinforcement learning (RL) from three perspectives:

  • Scale to Data: The amount and quality of data can be traded off. I have worked on offline RL with weak supervision, including domain-unlabeled data RLC2025 and noisy preferences TMLR2026.
  • Scale to Computation: I am especially interested in GPU-accelerated RL frameworks, both for simulators, including board games ♟️ Pgx, NeurIPS2023 (Co-authored) and Riichi Mahjong 🀄️ Mahjax, arXiv2026, and for algorithm codebases, such as offline RL JAX-CORL.
  • Scale to Trial: Even if an agent can gain experience through massive data or computation, it is not truly scalable unless it can learn efficiently. From this perspective, I have worked on exploration in RL. I focus on a novel exploration objective called ReMax, based on the Retry idea (ICML2026, TMLR2026).

publications

2026

  1. Mahjax: A GPU-Accelerated Mahjong Simulator for Reinforcement Learning in JAX
    Soichiro Nishimori, Shinri Okano, Keigo Habara, and 3 more authors
    arXiv preprint arXiv:2605.20577 2026
    arXiv PDF Code
  2. Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
    Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada, and 4 more authors
    International Conference on Machine Learning (ICML2026) 2026
    arXiv PDF Code
  3. On Symmetric Losses for Policy Optimization with Noisy Preferences
    Soichiro Nishimori, Yu-Jie Zhang, Thanawat Lodkaew, and Masashi Sugiyama
    Transactions on Machine Learning Research (TMLR2026) 2026
    PDF Code
  4. Mitigating Reward Hacking in RLHF via Advantage Sign Robustness
    Shinnosuke Ono, Johannes Ackermann, Soichiro Nishimori, and 2 more authors
    arXiv preprint arXiv:2604.02986 2026
    arXiv PDF
  5. Finite-Time Regret Analysis of Retry-Aware Bandits
    Bingkui Tong, Junpei Komiyama, Soichiro Nishimori, and Paavo Parmas
    arXiv preprint arXiv:2605.20854 2026
    arXiv PDF
  6. On Advantage Estimates for Max@K Policy Gradients
    Shota Takashiro*, Soichiro Nishimori*, Paavo Parmas*, and 6 more authors
    arXiv preprint arXiv:2606.06080 (* Equal contribution) 2026
    arXiv PDF
  7. OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
    Paavo Parmas, Yongmin Kim, Kohsei Matsutani, Shota Takashiro, Soichiro Nishimori, and 3 more authors
    arXiv preprint arXiv:2606.06096 2026
    arXiv PDF Code
  8. Retry Policy Gradients in Continuous Action Spaces
    Soichiro Nishimori and Paavo Parmas
    Transactions on Machine Learning Research (TMLR2026) 2026
    PDF Code
  9. Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests
    Thanawat Lodkaew, Johannes Ackermann, Soichiro Nishimori, and 3 more authors
    Advances in Neural Information Processing Systems (NeurIPS2026) 2026
    arXiv

2025

  1. Recursive Reward Aggregation
    Yuting Tang, Yivan Zhang, Johannes Ackermann, Yu-jie Zhang, Soichiro Nishimori, and 2 more authors
    Reinforcement Learning Conference (RLC2025) 2025
    arXiv PDF
  2. Offline Reinforcement Learning with Domain-Unlabeled Data
    Soichiro Nishimori, Xin-Qiang Cai, Johannes Ackermann, and Masashi Sugiyama
    Reinforcement Learning Conference (RLC2025) 2025
    arXiv PDF Code

2024

  1. A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees
    Toshinori Kitamura, Tadashi Kozuno, Masahiro Kato, Yuki Ichihara, Soichiro Nishimori, and 4 more authors
    Reinforcement Learning Conference Workshop 2024
    arXiv
  2. A Batch Sequential Halving Algorithm without Performance Degradation
    Sotetsu Koyamada, Soichiro Nishimori, and Shin Ishii
    Reinforcement Learning Conference (RLC2024) 2024
    arXiv
  3. JAX-CORL: A single-file repository for offline reinforcement learning
    Soichiro Nishimori
    github 2024
    Code

2023

  1. Pgx: Hardware-accelerated parallel game simulators for reinforcement learning
    Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, and 4 more authors
    Advances in Neural Information Processing Systems (NeurIPS2023) 2023
    arXiv Code
  2. End-to-End Policy Gradient Method for POMDPs and Explainable Agents
    Soichiro Nishimori, Sotetsu Koyamada, and Shin Ishii
    arXiv preprint 2023

2022

  1. Mjx: A framework for Mahjong AI research
    Sotetsu Koyamada, Keigo Habara, Nao Goto, Shinri Okano, Soichiro Nishimori, and Shin Ishii
    2022 IEEE Conference on Games (CoG) 2022
    PDF

news

Sep 23, 2026 Our paper Retry Policy Gradients in Continuous Action Spaces is accepted to TMLR!
Jun 05, 2026 Preprints on Regret analysis of ReMax, ReMax in continous actions, Baseline for Max@K are out!
May 09, 2026 Our ReMax paper is accepted to ICML 2026!
Apr 27, 2026 1 paper is accepted to TMLR 2026.
Dec 26, 2025 I releaced JAX-based Mahjong simulator, Mahjax 🀄️

latest posts

Apr 09, 2024 個人ページ作りました．