This study uses a fully crossed within-subjects repeated-measures design across 4 file-format conditions and 4 LLMs. Each CME article serves as its own control across file formats, yielding a within-item repeated-measures structure.

To account for the nondeterministic nature of LLMs, each model-format-article combination will be run 3 times, with primary and secondary outcomes being generated from recorded responses to CME test questions.

The study follows the Declaration of Helsinki and Good Clinical Practice principles. Although no human participants are involved, the study adheres to these ethical principles to ensure transparency, scientific integrity, and responsible AI evaluation. Ethics approval was granted by the Witten/Herdecke University Ethics Committee (S-260/2025; dated October 8, 2025). The study is prospectively registered with the Open Science Framework (OSF; OSF.IO/V96R5).

The sampling frame will consist of expired CME modules made available by the 3 prespecified major German medical publishers included in this study (Deutscher Ärzteverlag GmbH, Springer Medizin, and Georg Thieme Verlag). Eligible modules must contain (1) the complete CME article text, (2) the associated multiple-choice questions, and (3) the official answer key required for scoring. Only expired modules will be included to ensure that CME credit can no longer be claimed retrospectively. Modules that are still active, incomplete, duplicated, or technically unsuitable for standardized file-format conversion will be excluded.

To ensure structured coverage of major clinical domains, the eligible modules will be grouped according to 6 prespecified specialties: internal medicine, surgery, pediatrics, gynecology, neurology, and anesthesiology. Article selection will then be performed using a computer-generated randomization procedure from the eligible pool, with balanced allocation across specialties and publisher sources. This approach is intended to provide a heterogeneous multicenter sample from major German CME providers for a methodological comparison of file-format effects.

The aim of this sampling strategy is not to establish formal statistical representativeness of all German CME materials in Germany but rather to assemble a transparent, reproducible, and clinically diverse sample from major national CME providers under standardized experimental conditions.

All CME articles and the associated multiple-choice questions used in this study are protected by German copyright law (Urheberrechtsgesetz) and remain the intellectual property of the respective publishers (Deutscher Ärzteverlag GmbH, Springer Medizin, and Georg Thieme Verlag). Prior to study initiation, all 3 publishers were formally contacted; were informed in writing about the nature, scope, and purpose of the study; and provided the requested expired CME modules together with explicit permission for their use in this research project. No specific conditions were imposed by the publishers. The corresponding correspondence is on file with the study team and is available upon reasonable request.

Only expired CME modules are included, that is, modules for which CME credit can no longer be claimed retrospectively. This ensures that the study cannot influence the active CME credit market, does not interfere with the publishers’ ongoing commercial interests, and excludes any possibility that LLM-generated answers could be misused to obtain valid CME credits. No original article text, question stems, distractors, or answer keys are reproduced in the manuscript, the supplementary materials, or the OSF repository. CME modules are identified only by anonymized internal codes together with their specialty and publication year range, and quantitative performance data are reported in aggregated form. All technically necessary reproductions generated for the file-format conversions (protected, raster, and vector PDFs) are used exclusively as input for the LLMs under evaluation. The CME materials are entered into the 4 LLMs (GPT-5, Claude Sonnet 4, Grok-4, and Gemini 3) via their official consumer web chat interfaces, in accordance with the CME providers’ terms of service for noncommercial research use. Where available, the interfaces are configured to disable use of submitted content for model training to prevent leakage of copyrighted material into future model versions.

Four document types will be generated for each CME article to enable within-article comparisons of file-format effects on LLM performance:

Protected PDFs will be generated by applying standardized security settings to the original publisher-provided files using 256-bit AES encryption. These settings will uniformly restrict content modification, printing, and extraction of text, images, and metadata across all documents. The applied protection parameters (including encryption level, permission flags, and software toolchain; eg, Adobe Acrobat [Adobe Inc] or equivalent) will be documented in detail in the appendix or OSF materials to ensure transparency and reproducibility.

The aim of the protected condition is not to alter the visual representation of the document but to assess whether access restrictions alone affect LLM performance despite the unchanged underlying content. Therefore, each protected PDF will undergo a verification step prior to model evaluation. This will include manual inspection to confirm that visual fidelity matches the source document and to verify that text extraction is effectively restricted or blocked under standard conditions. Successful protection will be defined as maintaining full human readability while preventing or substantially limiting direct programmatic access to the textual content.

Raster PDFs will be generated using fixed rendering settings that are standardized across all articles, including prespecified image resolution and compression parameters, to ensure consistent image quality across the study. These settings will be documented in detail in the appendix or OSF materials to enable reproducibility.

The aim of the raster condition is not to create artificially unreadable files but to generate nonsearchable image-based PDFs that remain visually legible for human readers while limiting direct machine-readable text access. Therefore, each rasterized PDF will undergo a quality control check before model evaluation to confirm that page content remains readable and that rendering quality is not degraded to a level at which trivial optical character recognition (OCR) failure would be expected purely because of inadequate technical conversion settings.

In the vector PDF condition, the goal is to preserve a “vector-encoded” visual representation while minimizing direct text extractability. To achieve this, all textual content will be converted into vector outlines (ie, text-to-path conversion) so that no selectable text layer remains. Vector PDFs will be generated using a standardized toolchain (eg, Adobe Acrobat Preflight and/or Ghostscript [Artifex Software] or Inkscape [Inkscape Project]). To verify the intended reduction in extractability, each generated vector PDF will undergo a quality control step using automated text-extraction checks (eg, pdftotext [Glyph & Cog LLC] or equivalent); successful conversion will be defined as yielding no meaningful extracted text (eg, empty output or a negligible character count) while maintaining legibility comparable to the source document. These definitions are crucial, as LLMs differ in their ability to parse and extract text from graphical encodings. While raster formats may impede OCR and tokenization, vector formats retain structured information that might still be exploitable by advanced multimodal models [16-18].

The study does not involve human subjects as participants, only 1 “operator” (LÖ). Instead, the 4 “participating” LLMs (ChatGPT [version 5; OpenAI], Claude Sonnet [version 4; Anthropic PBC], Gemini [version 3; Google LLC], and Grok [version 4; xAI]) are treated as experimental agents. Each model will attempt to answer CME questions under 4 distinct format conditions. Responses will be collected and scored according to the official CME answer keys provided by the publishers.

To ensure reproducibility of the LLM experiments, we will provide the exact prompt templates used either verbatim in an appendix or via a permanent OSF appendix link. In brief, the prompt will include standardized task instructions, presentation of the current CME material in the assigned file-format condition, and instructions to answer the associated multiple-choice questions exclusively on the basis of the provided material. Core prompt components will be harmonized across models as far as technically feasible. Thus, the purpose of standardization is not to claim empirically proven prompt invariance across models but to minimize and transparently document prompt-related variation as consistently as possible across all study conditions.

Each model-format-article combination will be executed in 3 independent runs to account for stochastic variability in LLM outputs. These repeated runs will be used to derive the binary pass or fail outcome, as defined below. The primary analysis will use the mean accuracy across runs.

There are 16 model-format combinations, comprising a 4 (file format)×4 (LLM) fully crossed design. The four experimental conditions correspond to the 4 file-format types used in the within-article crossover design:

The validity of the outcome measure depends critically on the LLMs’ ability to correctly parse and interpret the presented CME materials. Different PDF formats introduce systematic variation in this process: searchable PDFs provide a clean, tokenized text structure that can be processed directly, whereas rasterized PDFs require OCR, which is known to introduce transcription errors, reduce token accuracy, and increase noise in model inputs [2]. To reduce the risk that observed performance losses merely reflect avoidable artifacts of excessively poor rasterization quality, raster PDFs will be generated using fixed standardized rendering settings and subjected to a prespecified quality control step before evaluation. Vector-encoded PDFs may further fragment text into glyphs or graphic paths, increasing parsing difficulty and occasionally exceeding tokenization limits in multimodal models [17-19]. These format-dependent processing differences constitute a potential threat to construct validity, as performance may reflect a model’s visual parsing capability rather than its medical reasoning competence. Therefore, evaluating file-format effects is necessary to isolate true diagnostic performance from artifacts introduced by differential input accessibility.

Models will never receive outputs, intermediate reasoning, or corrected answers from prior models. All responses are generated in isolated sessions with cleared context windows. No system prompts contain summaries or previous model outputs. In addition, within-model learning across file formats is minimized by enforcing strict session isolation: the same model never encounters more than 1 format of a given article within a single session, and no feedback on response correctness is provided between runs. Because all prompts start from an empty context window and contain only the current article in a single format, the risk of cumulative learning about individual CME items across formats is substantially reduced.

The primary outcome is the proportion of correctly answered CME questions for each LLM across the 4 file-format conditions (searchable, protected, rasterized, and vector-encoded PDFs). Accuracy will be calculated as the percentage of items answered correctly relative to the official answer key provided by the publishers. For inferential analysis, the 3 repeated runs for each article-model-format combination will be treated as technical replicates and averaged before statistical testing. Thus, the primary inferential unit will be the article-level mean accuracy for each model and file-format condition, rather than the individual run. This approach accounts for stochastic run-to-run variability while avoiding artificial inflation of the effective sample size.

Pass/fail status will be determined using the passing threshold specified by the respective CME module/provider (typically 70%). The threshold for each module will be recorded from the module instructions. For the binary pass/fail outcome, each model-format-article combination will be classified as “pass” if at least 1 of the 3 runs meets the passing threshold; otherwise, it will be classified as “fail.” This classification takes into account the models’ fundamental ability to pass a CME test despite the measures in effect. For inferential analysis, pass or fail status will therefore be evaluated at the article-model-format level rather than at the individual-run level.

This protocol describes a study designed to test the hypothesis that searchable PDF formats will enable the highest LLM performance on German CME assessments, whereas protected, rasterized, and vector-encoded PDFs will reduce performance by limiting direct machine readability. We further expect that the magnitude of this format effect may vary across model families, depending on their multimodal input-processing capabilities and robustness to OCR-related or layout-related degradation.

If confirmed, these findings would suggest that document format is not a neutral technical property but a potentially relevant determinant of AI solvability in postgraduate medical assessment environments. In this sense, the study is expected to contribute not only to the evaluation of LLM performance in medical education but also to the broader question of whether simple file-format modifications could function as low-threshold safeguards against AI-assisted misuse in CME settings.

This study has several methodological limitations that must be acknowledged. First, LLMs are inherently nondeterministic. Even with fixed prompts and identical input materials, models may produce slightly different outputs across runs due to stochastic sampling, internal state variation, or backend optimization routines.

The study does not include a human control group, preventing direct comparison of how physicians perform with protected, rasterized, or vector-based PDFs compared to LLMs. The experimental setup uses standardized prompts and does not examine the impact of advanced prompt engineering or multistep attacks (eg, external OCR followed by LLM processing), which could potentially circumvent format-based restrictions in real-world cheating scenarios. Although the conflict between AI protection and accessibility for visually impaired users is discussed, it is not empirically investigated.

Finally, modern LLMs are subject to frequent backend updates and performance drift over time, even when model names remain unchanged. As a result, the behavior of the evaluated models at the time of data collection (planned for June 2026) may differ from their performance at the time of publication or future replication attempts. We will therefore document all model version identifiers and run time stamps in detail, but residual version drift remains an inherent limitation of AI evaluation studies.

If protected, rasterized, or vector-encoded CME materials can be shown to reduce LLM performance, such formats may constitute a simple technical safeguard for protecting the integrity of CME assessments by limiting machine readability. However, implementing deliberately nonsearchable formats introduces a significant accessibility conflict. Under the German Accessible Information Technology Ordinance (Barrierefreie-Informationstechnik-Verordnung) and other technical guidelines [20-22], educational materials provided by professional bodies should, where feasible, remain accessible to users with visual impairments, age-related reading limitations, or reliance on assistive technologies such as screen readers or text-to-speech systems. Nonsearchable or access-restricted PDF formats may impair these functions by restricting text extraction, semantic navigation, adjustable contrast, and integrated search tools. Consequently, any attempt to constrain LLM capabilities through file-format manipulation must be balanced against the legal and ethical obligation to ensure accessibility and equal participation for human learners. Future studies should explore hybrid countermeasures that combine technical barriers with authentication protocols (eg, CAPTCHA-style test verification). Beyond regulatory implications, this research contributes to broader discussions on AI governance and digital literacy in postgraduate medical education [20,23-27].

Future studies should determine whether any observed file-format effects remain stable across newer model generations, other medical specialties, and additional assessment settings beyond CME article-based testing. It will also be important to investigate whether format-based restrictions can be circumvented through combined workflows, such as external OCR, multimodal preprocessing, or prompt-engineered extraction strategies.

In addition, future research should evaluate human-centered alternatives that preserve accessibility while maintaining assessment integrity, including secure authentication procedures, adaptive item delivery, hybrid anticheating measures, and accessibility-preserving technical safeguards. Comparative studies involving human participants may further clarify whether format manipulations disproportionately affect legitimate learners, particularly those who rely on assistive technologies.

The findings of this study will be disseminated through submission to a peer-reviewed journal and presentation at scientific meetings in the fields of medical education, digital health, and health professions assessment. In addition, the results will be communicated to relevant CME stakeholders, including publishers, professional bodies, and regulatory audiences, to inform ongoing discussions about assessment integrity, accessibility, and responsible AI governance in postgraduate medical education. Where appropriate, study materials relevant to reproducibility will be made available via the OSF in accordance with journal and copyright requirements.

Rather than merely describing a technical comparison of PDF types, this protocol addresses a timely governance question in postgraduate medical education: whether assessment materials can be made less vulnerable to AI-assisted misuse without creating unacceptable barriers for human users. By examining how searchable, protected, rasterized, and vector-encoded formats affect LLM performance, the study is expected to generate evidence relevant to the design of CME assessments at the intersection of validity, accessibility, and responsible AI integration. The findings may help inform future policy and technical decisions by CME providers, publishers, and regulators.