About me

I am a Senior Machine Learning Engineer at Gamania Digital Entertainment, where I lead the development of Muztrix, an AI-native manga and webtoon creation platform, and build multimodal AI products spanning LLM agents, image and audio generation, and video understanding. Previously, I was a Visiting Researcher at Sony Computer Science Laboratories.

With a Master's degree in Data Science from National Taiwan University, I have presented research at conferences such as ICASSP, ISMIR, and DAFx. I enjoy taking ideas from research to production: from audio processing and brain decoding to full-stack generative AI services running in the cloud.

What I do

  • Generative AI & LLM Agents

    Building agentic AI products with tool calling, multimodal generation, and production-grade cloud deployment.

  • Medical Imaging & fMRI Analysis

    Applying self-supervised learning techniques to decode brain signals and analyze medical images.

  • Audio, Speech & MIR

    Music transcription, audio effects, voice cloning, and expressive text-to-speech for virtual characters.

  • Multimodal & Computer Vision

    Combining detection, segmentation, speech, and vision-language models to understand images and video.

Resume

Education

  1. National Taiwan University

    Feb 2023 — Jun 2024

    M.S. in Data Science
    Thesis: Whole‑Brain Feature Selection Methods for Decoding from fMRI Data

  2. National Taiwan University

    Sep 2019 — Jan 2022

    B.S. in Computer Science and Information Engineering (CSIE)

Experience

  1. Senior Machine Learning Engineer @ Gamania Digital Entertainment

    Jul 2025 — Present

    AI Manga & Webtoon Creation Platform (Muztrix)
    • Led development of Muztrix, a public AI platform where creators plan stories with a conversational agent and turn them into page manga or vertical-scroll webtoons.
    • Built an agentic story chat (Ask/Agent modes, tool calling, web search, file uploads) that generates and edits characters, locations, and storyboards, with reference-image binding and versioned images for visual consistency.
    • Integrated multiple generation models for images, line art, image-to-3D, and music, and added an AI game studio where coding agents build and publish web games inside isolated cloud sandboxes.
    • Shipped production infrastructure: Next.js + FastAPI + PocketBase on GCP Cloud Run with CI/CD, monitoring, usage-based credits, OAuth sign-in, and 4-language i18n.
    Multimodal AI Applications
    • Prototyped retail and restaurant video analytics combining segmentation/detection, speech recognition, and vision-language models.
    • Built an AI comic-diary LINE Bot for older adults with LLM story writing, image generation, and TTS narration.
    • Developed AI-generated sound effects and music for immersive webtoon reading, plus voice conversion and expressive TTS for virtual characters.

  2. Visiting Researcher @ Sony Computer Science Laboratories

    Jun 2025 — Oct 2025

    Research Topics: Music Information Retrieval, Automatic Music Transcription
    • Proposed VioPTT, a lightweight cascade model for violin technique-aware transcription with a novel synthetic dataset (MOSA-VPT). Published at ICASSP 2026 [1].
    • Investigated transfer learning for violin AMT, showing training from scratch on instrument-specific data can match fine-tuned piano-pretrained models. Published at ISMIR 2025 Late Breaking/Demo [2].

  3. AI Engineer @ Gate.io

    Oct 2024 — May 2025

    • Developed a Text-to-SQL AI agent enabling non-technical teams to access internal data, boosting query efficiency by 20%.
    • Developed a fund flows anomaly detection system with LLMs and tree-based models, enhancing financial security.

  4. Research Assistant @ Academia Sinica (MCTLAB)

    Mar 2022 — Oct 2024

    Research Topics: Self‑Supervised Learning, Medical Imaging
    • Proposed a Transformer-based self-supervised learning method for decoding brain signals (fMRI), achieving a 77% reduction in memory footprint.
    • Conducted distributed training experiments on high-resolution 4D medical images (fMRI) using TWCC HPC.
    • Proposed a whole-brain feature selection method for decoding musical pitch from fMRI.

  5. AI Engineer Intern @ Tomofun

    Mar 2023 — Jul 2024

    Research Topics: Computer Vision, Large Language Models, Multimodal Learning
    • Developed an automatic short music video generation system for daily pet clips.
    • Fine-tuned visual language models (e.g., BLIP), achieving a 20.6% improvement in visual question answering.
    • Enhanced LLaVA image inference speed by 250% with only a 3% accuracy reduction.
    • Developed APIs for visual language models using llama.cpp/ollama for image-caption pair datasets.

Skills

  • Python & Deep Learning

    PyTorch, TensorFlow, Scikit-learn, Hugging Face

  • Generative AI & Agents

    LLM Agents, Tool Calling, RAG, Vision-Language Models, Image / Audio Generation, Gemini, Claude, Vertex AI

  • Machine Learning

    Self-Supervised Learning, Computer Vision, NLP, Speech & Audio, Medical Imaging

  • Full-Stack & Cloud

    Next.js, React, FastAPI, PocketBase, Docker, GCP (Cloud Run), Cloudflare Workers, CI/CD

  • Data Engineering

    Pandas, SQL, Distributed Training (Slurm), HPC

  • Programming Languages

    Python, TypeScript, JavaScript, Go, C++, C, Linux

Download CV

Projects

Research & Projects

  1. VioPTT Demo Page

    VioPTT: Violin Technique-Aware Transcription

    Research Project

    Developed VioPTT, a violin transcription system that recognizes playing techniques (pizzicato, tremolo, harmonics, etc.) from audio via synthetic data augmentation. Supports end-to-end audio-to-MIDI with per-note technique labels.

    💻 Code 🎻 Demo
  2. Guitar Effect Removal Demo

    Guitar Effect Removal

    Collaboration with Positive Grid ML Team

    Proposed a two-stage method to remove distortion effects from guitar recordings using Positive Grid VST plugins. Achieved 20% higher audio quality than the best baseline, rated by 26 professional guitarists. Published in DAFx 2024.

    📄 Paper 🎵 Demo
  3. AutoFMRI GitHub

    Whole Brain fMRI Feature Selection (AutoFMRI)

    Academia Sinica - MCTLAB

    Proposed a two-stage automatic thresholding method to extract whole-brain fMRI features and predict musical pitch. Demonstrated 2-fold improvement over ROI-based feature selection. Available as a pip package (pip install autofmri). Published in ICASSP 2023.

    📄 Paper 💻 Code
  4. MIDI Rhythm Master

    MIDI Rhythm Master

    Side Project

    A browser-based rhythm game that reads any MIDI file and turns it into a playable 4-lane game. Features real-time synth playback via Web Audio API, difficulty settings, hold notes, and a global leaderboard powered by Cloudflare Workers + D1.

    💻 Code 🎮 Play
  5. Match-3 Puzzle Game

    Match-3 Puzzle Game

    Side Project

    A match-3 puzzle game built with Godot Engine and deployed to the web. Features gem-matching mechanics, cascading combos, and shader effects.

    💻 Code 🎮 Play

Publications

Publications

  1. VioPTT: Violin Technique-Aware Transcription from Synthetic Data Augmentation

    ICASSP 2026

    Wang, T. K.*, Peng, Y. P.*, Su, L., & Cheung, V. K. M.
    Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing 2026 (ICASSP'26). (* equally contributed)

    📄 Paper

  2. Is Transfer Learning Necessary for Violin Transcription?

    ISMIR 2025 Late Breaking

    Peng, Y. P.*, Wang, T. K.*, Su, L., & Cheung, V. K. M.
    Int. Society for Music Information Retrieval Conf. 2025 (ISMIR'25) - Late Breaking/Demo. (* equally contributed)

    📄 Paper

  3. Whole-Brain Transferable Representations from Large-Scale fMRI Data Improve Task-Evoked Brain Activity Decoding

    arXiv 2025

    Peng, Y. P., Cheung, V. K. M., & Su, L.
    arXiv preprint arXiv:2507.22378

    📄 Paper

  4. Distortion Recovery: A Two-Stage Method for Guitar Effect Removal

    DAFx 2024

    Lee, Y. S.*, Peng, Y. P.*, Wu, J. T., Cheng, M., Su, L., & Yang, Y. H.
    Proc. Int. Conf. Digital Audio Effects 2024 (DAFx'24). (* equally contributed)

    📄 Paper 🎵 Demo

  5. Decoding Musical Pitch from Human Brain Activity with Automatic Voxel-Wise Whole-Brain FMRI Feature Selection

    ICASSP 2023

    Cheung, V. K.*, Peng, Y. P.*, Lin, J. H., & Su, L.
    Proc. IEEE Int. Conf. on Acoustics, Speech, and Signal Processing 2023 (ICASSP'23). (* equally contributed)

    📄 Paper