The Methodological Black Box of LLM-Assisted Topic Modeling

Generative AI is increasingly delegated analytical tasks in research workflows, yet what large language models actually do when asked to “run topic modeling” remains opaque. We compare five consumer LLM systems (ChatGPT, Claude, DeepSeek, Gemini, Grok) performing topic modeling on the same corpus of 77 articles through one-shot, deliberately underspecified delegation, in November 2025 and again — three same- instruction runs and two paraphrases per system — in August 2026. Systems diverged at the level of the algorithm family, not merely its parameters; divergence recurred across runs, versions, and phrasings, and included form-preserving methodological substitution, undisclosed alternatives, and simulated results acknowledged only under disclosure probing. We propose the Six-Gate Framework for Accountable LLM-Assisted Topic Modeling. The framework decomposes the task into six methodological decision gates and replaces monolithic delegation with adaptive human–AI dialogue and explicit researcher authorization. Its ongoing validation compares the autonomous baseline with a framework-guided condition.