Shinnosuke Ono

I am applying to PhD programs for Fall 2027. I hold the Toyota Riken Overseas PreDoctoral Fellowship, which funds two years of doctoral study abroad with up to three further years of support.

I am a second-year master’s student at the University of Tokyo, advised by Prof. Masashi Sugiyama and Prof. Takashi Ishida. Through December 2026 I am a visiting research student at Nanyang Technological University, hosted by Prof. Bo An. I am also a research assistant at the Matsuo-Iwasawa Lab and the lead engineer at EQUES.

personal

Outside of research, I love:

running. Best half marathon: 1:43:38 (Yamanakako 2026).
music. I also play (a bit of) the guitar.
sports. I played soccer (was a 5′6″ goalkeeper). Also love watching various sports.
my dear three cats.

research interests

My research goal is to develop reliable AI systems that humans can benefit from working with, and to help our society adapt to them. I am working on foundation models:

Reward hacking in RLHF: True human preferences are hard to specify via rule-based rewards, so we need learned reward models (RMs). RMs are vulnerable to reward hacking, where language models exploit imperfect RMs during RL. TMLR2026 quantifies this vulnerability via adversarial robustness and proposes a practical, lightweight algorithm to mitigate reward hacking.
Application: IJCNLP-AACL2025 develops a Japanese pharmaceutical LLM via continual pre-training and model merging, and proposes three new evaluation benchmarks for this field. arXiv2025 introduces a large-scale hierarchical mobile manipulation dataset for VLA models.
Societal perspective: How can we oversee the evolution of almost superhuman AI systems? Work coming soon!

I am always happy to chat about research. Reach out!

news

Oct 24, 2026	Our paper Mitigating Reward Hacking in RLHF via Advantage Sign Robustness was accepted to Transactions on Machine Learning Research (TMLR) 🎉
Jul 07, 2026	Awarded the Overseas PreDoctoral Fellowship of the Toyota Physical and Chemical Research Institute, funding doctoral study abroad from 2027 🎉
Jul 01, 2026	Started a six-month visiting research stay at Nanyang Technological University with Prof. Bo An, supported by the Heiwa Nakajima Foundation 🇸🇬
May 19, 2026	Our paper Mitigating Reward Hacking in RLHF via Advantage Sign Robustness was accepted to the EIML Workshop at ICML 2026 🎉
Apr 09, 2026	Presented a poster and chaired a session at the Statistical Safeguarding Workshop 2026, hosted by the University of Tokyo and the RIKEN AIP Imperfect Information Learning Team.

selected publications

TMLR
Mitigating Reward Hacking in RLHF via Advantage Sign Robustness

Shinnosuke Ono, Johannes Ackermann, Soichiro Nishimori, and 2 more authors

Transactions on Machine Learning Research, Oct 2026

Abs DOI arXiv Bib HTML PDF

Reward models (RMs) used in reinforcement learning from human feedback (RLHF) are vulnerable to reward hacking: as the policy maximizes a learned proxy reward, true quality plateaus or degrades. We make the assumption that reward hacking is often caused by flipped advantage signs: instead of reducing the likelihood of a bad response, a flipped sign causes the update to increase it. By considering an adversarial perturbation in the RM parameter space, we can derive a certified sign-preservation radius, which is the smallest perturbation that can flip the advantage sign during policy optimization. Based on this formulation, we propose Sign-Certified Policy Optimization (SignCert-PO), down-weighting non-robust completions in the policy gradient update. Unlike prior approaches that require multiple RMs or access to the RM training data, SignCert-PO is lightweight and operates purely at the policy optimization stage using only the RM parameters and on-policy completions. On TL;DR summarization and AlpacaFarm benchmarks, SignCert-PO achieves a better win rate than baselines in most cases and reduces reward hacking.
@article{ono2026signcert, title = {Mitigating Reward Hacking in {RLHF} via Advantage Sign Robustness}, author = {Ono, Shinnosuke and Ackermann, Johannes and Nishimori, Soichiro and Ishida, Takashi and Sugiyama, Masashi}, journal = {Transactions on Machine Learning Research}, year = {2026}, month = oct, doi = {10.48550/arXiv.2604.02986}, }
AACL
A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP

Shinnosuke Ono, Issey Sukeda, Takuro Fujii, and 2 more authors

In Proceedings of the 4th International Joint Conference on Natural Language Processing and the 14th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL)equal contribution with I. Sukeda , Dec 2025

Abs DOI arXiv Bib HTML PDF Supp Code Website

We present a Japanese domain-specific language model for the pharmaceutical field, developed through continual pretraining on 2 billion Japanese pharmaceutical tokens and 8 billion English biomedical tokens. To enable rigorous evaluation, we introduce three new benchmarks: YakugakuQA, based on national pharmacist licensing exams; NayoseQA, which tests cross-lingual synonym and terminology normalization; and SogoCheck, a task designed to assess consistency reasoning between paired statements. We evaluate our model against both open-source medical language models and commercial models, including GPT-4o. Results show that our domain-specific model outperforms existing open models and achieves competitive performance with commercial ones, particularly on terminology-heavy and knowledge-based tasks. Even GPT-4o performs poorly on SogoCheck, suggesting that cross-sentence consistency reasoning remains an open challenge. Our benchmark suite offers a broader diagnostic lens for pharmaceutical NLP, covering factual recall, lexical variation, and logical consistency.
@inproceedings{ono2025japanese, title = {A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical {NLP}}, author = {Ono, Shinnosuke and Sukeda, Issey and Fujii, Takuro and Buma, Kosei and Sasaki, Shunsuke}, booktitle = {Proceedings of the 4th International Joint Conference on Natural Language Processing and the 14th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL)}, year = {2025}, month = dec, doi = {10.48550/arXiv.2505.16661}, }