What is already known?

• • Clinical trials are a primary approach for generating evidence on health interventions. Trials are regulated to ensure participants’ well-being, scientific relevance, and transparency.

• • Registers make public rich trial data, but tools for efficiently using the data are lacking, impacting reproducible, deep information synthesis and learning.

What is new?

• • ctrdata, an open-source R package, enables using all publicly available data and documents from four registers (the EU Clinical Trials Information System [CTIS], the EU Clinical Trials Register [EUCTR], the US Clinicaltrials.Gov [CTGOV], and ISRCTN—the UK’s Clinical Study Registry). It supports research steps from identifying and storing trials of interest, deduplication, scrutinising structure, extracting fields, to analysing user- or pre-defined scientific and operational concepts.

Potential impact for RSM readers

• • With ctrdata, researchers can easily implement a programmatic workflow to investigate trials in depth. ctrdata keeps pace with register changes and user requirements. Its databases can be used with any system.

• • Beyond drug development, ctrdata is relevant for patient access, methodology research, health policy, and outcomes research.

2.1. Registers

When the development of ctrdata was started in 2015, making public information on registered clinical trials had been required by US legislation since fifteen years,Reference Tse, Fain and Zarin ¹⁰ by medical journal editors for ten years, ¹¹ and by legislation in the European Union (EU) for five years.Reference Egger, Herold, Rodriguez, Manent, Sweeney and Saint Raymond ²

At that time, CTGOV provided an API, ¹² and both CTGOV and EUCTR used XML schemas (Extensible Markup Language) ¹³ for data models, published them for information of data providers, and continually updated them to meet business requirements and changes in the legislative frameworks. ^{14 , 15}

Package ctrdata supported these two registers since 2015. The ISRCTN is supported since 2021, when it was started to be used for statutory purposes in the UK; it provides an API and XML data. ¹⁶

Since March 2023, ctrdata supports the EU CTIS, both for data made public before and after its relaunch in mid-2024. ¹⁷ CTIS is to be used for new clinical trials since February 2022. Data from CTIS are derived in JSON format (JavaScript Object Notation ¹⁸ ) from its public API that feeds the register’s web interface.

The current main characteristics of registers are summarised in Table 1.

Table 1

Overview of registers supported by ctrdata (numbers rounded to three significant digits; XML, extensible markup language; JSON, JavaScript Object Notation; ‘other types of clinical studies’ refers to studies of medical devices, behavioural and other health interventions, observational, non-interventional, and other studies)

2.2. Principles

Several principles evolved when developing package ctrdata.

All details are downloaded for clinical trials of interest, because only a complete public record is an accurate representation of the trial. Since the registers differ in scope, legal, and regulatory purposes, their content is complementary and ctrdata thus should work with several and an increasing number of registers.

Before analyses, trial information should first be downloaded and stored in a database, because the set of trials of interest is often a union set from different search queries and possibly different registers, and because offline availability of data for analysis is useful.

The data models that are implicit in data as retrieved from the different registers are retained by ctrdata, because differences between the data structure and value sets of different registers can well be handled at the time of analysis, and because any mapping to a putative target data model would be a goal suitable for an international harmonisation organisation.

Against this technical background and principles, ctrdata was implemented in the R environment, ¹⁹ which has a broad user base and has an extensive support for structured data, network operations, dependency management, and quality assurance.

At the same time, the users of ctrdata should be provided with functions that are simple and cover all relevant steps, without duplicating functions in R or one of its many extension packages. Main functions in ctrdata are listed in Table 2 in the order of a potential workflow, together with the number of the use case in Section 3 that exemplifies the function.

Table 2

Main functions in ctrdata in order of a potential workflow

Note: A full list of functions is part of the documentation website at https://rfhb.github.io/ctrdata/.

2.3. Analysis concepts

Package ctrdata includes functions that implement specific analysis concepts (Table 3). Concepts of clinical trials, such as their start date or their number of arms/groups with different investigational medicines, require analysing several fields against various criteria. However, the structure and the value sets of data fields differ between the registers.

Table 3

Overview on clinical trial concept functions implemented in ctrdata

Note: The refinement of some of the concepts is informed by ongoing research and use cases.

To address this situation, 20 trial concepts, pre-defined in ctrdata, are offered to simplify and accelerate a user’s analysis workflow, thereby increasing analysis consistency and reproducibility.

Some trial concepts can build on clear definitions and close similarities of registers; thus, concepts such as the trial phase, trial population, number of sites, and status of recruitment when loading the trial can be calculated with some confidence, yet users should note the respective help texts, which include any caveats such as that EUCTR does not have numbers of sites for non-EEA (European Economic Area) countries .

Where definitions are not closely similar, an operational definition was chosen to create trial concepts of interest; an example is the sponsor type at the level of the trial, for which a new value of ‘mixed’ is calculated if a trial’s sponsors include commercial and non-commercial entities.

Other trial concepts reflect the author’s proposals for temporary approximations of less well-defined concepts, such as trial objectives (e.g. dose-finding, pharmacodynamics) and ‘f.likelyPlatformTrial()’, with the function name flagging the uncertainties.

The trial concepts in ctrdata (all described in Table 3) have not been validated with any formal approach but have been checked for plausibility and against common sense expectations. Where possible, the implementation of a trial concept is based on documented current understanding, on public data models, or on scientific papers, as relevant. Users are invited to note the help texts of the concepts, which mention the register fields and any caveats, to review the function logic in its code (as in Section 3.3), and to raise an issue or to contribute improving any trial concept in the public repository of ctrdata (see data availability statement).

2.4. Storage

For storage of trial data, a document-centric approach was chosen because all data on a particular trial represent a self-standing document, where documents can differ in structure and do not require to pre-specify a schema.

The R package nodbi is used as a connector to document-centric databases and was extended to work, in addition to MongoDB, with PostgreSQL, RSQLite, and DuckDB as backend.Reference Herold, Chamberlain, FitzJohn and Ooms ²⁰ The latter are SQL databases but have functions for handling JSON which are abstracted by nodbi so that all four backends can be used interchangeably, without further changes in R scripts. Since RSQLite and DuckDB are available for all R platforms as local databases, their use with package ctrdata is likely of general interest.

Databases created with ctrdata can be accessed with other R packages and with other languages, such as Python, Julia, or JavaScript. Furthermore, using a MongoDB server enables to execute analyses directly on the server, such as efficient aggregation pipelines as shown in one of ctrdata’s vignettes. ²¹

3.1. Generate queries and count trials

Research often starts with developing a search strategy for information of interest. To facilitate searching in different CTRs, the user can provide high-level search parameters to function ‘ctrGenerateQueries()’. The parameters are translated into the different approaches of the trial registers for producing search results:

The function generates by default queries limited to interventional studies with medicines, referred to as ‘clinical trials’ throughout this article. The function parameter ‘onlyMedIntervTrials’ can be set to FALSE to remove this limitation and find all types of studies available in the register. Other interesting parameters of function ‘ctrGenerateQueries()’ are, for example, ‘countries’, ‘searchPhrase’, ‘condition’, ‘phase’, and various dates.

The function generates a (named) vector of hyperlinks specific to the registers. The links can be used to open the registers’ results pages so that the user can check and refine the queries:

A next step can be to determine the number of trials that can be obtained with the queries. To this end, function ‘ctrLoadQueriesIntoDb()’ is executed, and this emits messages for the user’s information about the data exchange with the CTR, including the number of trials found.

As a more advanced programming pattern, function ‘ctrLoadQueryIntoDb()’ can be applied to all queries to store return values in a list, from which the number of trials can be extracted as follows:

Note that the number of EUCTR records reflects the number of countries involved; it is thus a multiple of the number of trials. Also, note two types of queries are provided for CTGOV, including the register’s expert search page for interactively composing and executing more complicated and nested queries; they result in the same set of trials and thus, one query can be removed for subsequent use cases:

Importantly, users retain full control over queries to match their specific research interest, for example by modifying the query strings as one would modify any other string in R. Function ‘ctrGenerateQueries()’ will remain useful to get started, since searches much differ between registers.

3.2. Download trial data for analyses

The trials that have been identified with a search strategy have to be retrieved and downloaded, in order to refine the set of trials of interest and to analyse any of their details. One of the principles recognised by ctrdata (see Section 3.2) is that a final set of trials of interest often results from complementary queries in the same or in different registers.

First, a connection to a database is created, here SQLite (DuckDb, MongoDb, and PostgreSQL can also be used), for which the corresponding R package needs to be installed. A collection (database table) is specified to hold data of trials of interest:

Second, the queries defined above are used to download the trial data. Function ‘ctrLoadQueryIntoDb()’ here is applied to all queries:

For the total of almost 1700 trial records from four registers, the downloading takes around 100s (at a maximum bandwidth of 10 MB/s; ctrdata throttles the number of requests per time period).

Any number of additional queries can be loaded into the same collection, for example:

Function ‘ctrLoadQueryIntoDb()’ can also repeat a query to load trials that were updated or are new since the last time the query was loaded. Since this is a main function of package ctrdata, its full signature provides an overview of the options with which a user can tailor the data to be loaded to their needs. Some of the options are discussed in the following use cases.

The function returns the number and identifiers of trials that were successfully loaded or failed to load, and the query that was used.

Further for documentation and reproducibility, ctrdata includes metadata in the database collection whenever ‘ctrLoadQueryIntoDb()’ is run so that users can check and re-use:

A second main function of ctrdata has the purpose to provide a user-friendly table (data frame in R) from any data in a database collection, which can then be tabulated, for example:

Since trial data are extensive and hierarchically structured, a user can explore their structure and value sets of individual trials with ctrdata, which provides an interactive browser widget to identify individual data fields of interest:

Fields of interest can also be found across a sample or all trials from different registers in a collection:

3.3. Analyse using pre-defined trial concepts

Beyond individual fields of interest as identified above, ctrdata comes with various trial concepts that are already implemented as functions for selecting and analysing fields from different registers.

It is seemingly simple to calculate the start date of a trial and its current recruitment status, yet it involves more than 20 fields across the registers to calculate the new columns that correspond to these two concepts for the data frame provided by ‘dbGetFieldsIntoDf()’:

The pre-defined trial concepts much simplify a user’s workflow, and an overview on the currently 20 functions is available in Table 3 and here:

The function names indicate if a trial concept can be calculated exactly as above or can only be approximated (e.g. ‘f.likelyPlatformTrial()’). Users can inspect how a concept is calculated by calling the name of the function, for example:

A particular trial may have been registered in more than one register, and in EUCTR one trial has one record for every participating EU Member State. Therefore, ctrdata provides the trial concept ‘f.isUniqueTrial()’, which helps to identify and select only unique trials before further analyses, to avoid double-counting:

Alternatively to the above, at the time of loading trials from EUCTR, it is possible to just load a single record for any trial, by calling function ‘ctrLoadQueryIntoDb()’ with parameter ‘euctrprotocolsall’ set to FALSE. This setting can be useful when there are no questions about differences between Member States’ versions, such as dates, authorisation decisions, ethics opinions, and trial end.

In the example above, phases of medicine development are calculated based on values recorded in registers, and sample sizes are calculated to reflect the planned or the achieved number of participants, depending on the status of recruitment. After calculating the trial concepts, further analyses become reasonably simple, such as exploring associations (Figure 2):

Figure 2

Boxplot of sample size by phase of trial.

3.4. Merge and analyse information

Function ‘dfMergeVariablesRelevel()’ can be used for combining arbitrary fields of the same type from different registers into a new variable, here in an example for country data.

Clinical trials often span countries and even regions, in particular when conditions under study are rare, when a large number of participants is sought, or when the performance of interventions in the context of local health systems is to be analysed. ²³ All trial registers supported by ctrdata provide data on countries, and CTIS and CTGOV provide data on individual sites, including their location and contacts. The respective fields are extracted into a data frame and then are merged into a new variable, concatenated with ‘ / ’:

The new variable can be used for example for a cross-tabulation against all countries involved in the trials of interest, or for counting the sites per country as follows:

Besides analysing country data, function ‘ctrGenerateQueries()’ can be used for searching trials that are conducted in countries specified by the user.

3.5. Analyse text data that describe endpoints

Endpoints or outcomes investigated in clinical studies are so far represented in registers as textual descriptions, and there is no controlled terminology established globally or in any register. A particular endpoint is usually described with components covering a title or short description, its operational definition, and the time points when it is evaluated, and together they roughly correspond to the variable, one of the four estimand components.Reference Pétavy, Guizzaro, dos, Teerenstra and Roes ²⁴ The trial concept ‘f.primaryEndpointDescription()’ can provide the outcome variable in a single string; however, text analysis methods then need to be applied.

For example, questions may concern in how far difference-from-baseline variables continue to be used. Here, for unique efficacy trials with a phase 3 label, the text analysis method is using a regular expression to determine if the primary endpoint of a trial likely corresponds to such a variable or not, cross-tabulating with the type of sponsor:

Other research questions require an abstraction or categorisation of endpoint variables, such as questions about the type of endpoint.Reference Folgori, Bielicki and Ruiz ²⁵ Here, package ctrdata can be used for obtaining and pre-processing endpoint data (shown above), which then could be fed into a suitable large language model to predict the sought category:

3.6. Results-related primary endpoint data

In previous sections, the retrieval from EUCTR did not include result-related data, because this can be quite time-consuming for this register. Results-related data are always retrieved from CTGOV. From CTIS and ISRCTN, there are no results available in a structured format for the foreseeable time.

Retrieving results could have been done already during the first loading of trial data; newly loading trial data that include results overwrites any records of the same trials that were previously loaded (while maintaining user annotations such as those used below):

Package ctrdata provides data on primary endpoints by analysing various fields in different registers. This simplifies a user’s workflow, for example to explore details of results of null hypothesis significance testing (NHST, Figure 3):

Figure 3

Cumulative density of reported p-values (dotted line, p = 0.05).

Similarly, statistical methods used for primary endpoint analysis can be tabulated:

3.7. Changes during trial conduct

With the availability of historic versions of registered trials, changes over time can be identified by comparing data of interest across versions. At this time, only CTGOV directly provides historic versions; in addition, ctrdata can create historic versions also for CTIS, when re-running a previous query. Since retrieving historic versions is time-consuming, it has to be specified by the user when calling function ‘ctrLoadQueryIntoDb()’.

For CTGOV, a user has to specify the parameter ‘ctgov2history’ to be either a number (which loads this number of historic versions, at equal time intervals from the first to the current version), a string such as ‘n:m’ (which loads the n^th to the m^th version) or TRUE (which loads all available versions).

For example, changes in the targeted sample size are an example of the research questions that can be addressed with historic versions:

For CTIS, a user has to set parameter ‘ctishistory’ to TRUE and specify ‘querytoupdate’, which moves a current CTIS record in the database into an array of historic versions in the record, before updating the record from CTIS. Thus, historical versions depend on when a user updates a previous CTIS query; for example, changes in recruitment numbers across current and historic versions can be analysed as follows:

3.8. Linking trial and product data

The research activities in clinical trials can lead to data that show the quality, safety, and efficacy of medicines. Research questions about market and patient access to medicines include the progress from trials to product authorisation. They are examples of questions that require two or more data sources for analysis. Here, data on new molecular entities that are authorised medicinal products are retrieved from openFDA ²⁶ :

Data on trials are retrieved for the new molecular entities one by one, storing the respective name in the user annotation of the retrieved trials:

Therapeutic-confirmatory trials, often referred to as phase 3 trials, are typically needed for regulatory review of applications for marketing authorisation, and here they are merged with product data for a visualisation of trial completion (‘C’) and product authorisation (‘A’, Figure 4):

Figure 4

Completion of phase 3 trials and authorisation of medicines.

3.9. Exploring documents of trials

Study documents offer a rich source of information that is made readily accessible by ctrdata for exploration. Loading documents is activated in function ‘ctrLoadQueryIntoDb()’ by specifying the name of a directory in parameter ‘documents.path’. As a first step for this example, the parameter ‘documents.regexp’ is set to NULL, which causes the function to create empty placeholder files for every document that could be loaded. The names of the files are analysed to obtain an overview of types of available documents.

In a second step, parameter ‘documents.regexp’ can be set to a regular expression that causes downloading the files conforming to the expression.

A broad set tools are available for text analysis of document corpora. A recently published tool chain for a retrieval-augmented generation (RAG) workflow in R is ‘ragnar’.Reference Kalinowski and Falbel ²⁷ This is used in the following example to search for pharmacokinetics in result documents.

3.10. Safety data analysis across registers

Use case 10 is provided in the Supplementary Material to this article.

5.1. Implementation

So far, package ctrdata is based on a single developer, and this situation could impact the quality of its implementation and coding, which in turn may impact code comprehensibility and opportunities for involving other developers. This article contributes to the visibility of ctrdata and to attracting contributors. In addition, the following steps mitigate this potential limitation.

For readability and maintainability, code in package ctrdata follows style conventions, uses standard linting, and is well documented with line and function comments as well as a with a comprehensive website that includes vignettes and examples. Over the years, improvement exercises for already functioning code were repeatedly undertaken, including substantial re-implementation, refactoring, or factoring out, and this improved code quality, limited dependencies, stabilised performance, simplified functions, and helped adding registers or adapting to their changes.

Unit and other tests are written at the time that code is written or issues are fixed, and now more than 630 tests cover more than 94% of the code base. A continuous integration pipeline automates testing on several operating systems and with different database backends.

Since 2015, code was regularly committed and pushed to its public repository. Users from across the world contributed with around 50 issues so far, which are visible in the public repository and were resolved typically in hours or days. Since 2016, package ctrdata is made available in the CRAN, which requires stringent checks.

5.2. Functionality

Limitations could be seen in the functionality of package ctrdata.

For example, ctrdata does not map or translate related data elements from registers to a single common data model for data storage. Another limitation could be seen in the choice of trial registers that are currently supported by ctrdata. The reasons for these two choices that could be perceived as limitations are discussed in Section 1.

Furthermore, even though no formal approach was used for managing business requirements, the current functionality of ctrdata was informed by a broad variety of questions that users sought to answer with trial register data and by wishes for specific functionalities, such as downloading trial-related documents or providing a correspondence matrix of trial identifiers.

5.3. Use for research questions

For research and scientific questions, package ctrdata may be limited if not all relevant registers can be used with ctrdata, or when content of trial registers is incorrect or incomplete. The latter situation can arise from different statutory obligations on trial sponsors. Also, information made publicly available may be ambiguous or not detailed enough for the questions at hand. Early phase trials or trials that were rejected by oversight authorities or ethics committees may have only scarce details or may not be publicly visible at all. Some statutory requirements and information content details are in Table 1.

A challenge with using ctrdata for research questions is that currently interesting topics are often not represented as structured data in trial register information. For example, at this time, platform trials or integrated research platformsReference König, Spiertz and Millar ³⁵ cannot be directly identified in searches or characterised in publicly available data from trial registers. Some registers provide examples for registering trials with such and other less common designs. ³⁶ Package ctrdata offers a set of trial concepts (Table 3) to mitigate the potential limitations of incomplete controlled vocabularies and of evolving research concepts.

For using CTGOV for research, ten common problems were described.Reference Tse, Fain and Zarin ¹⁰ Conceptually, many of the issues concern any trial register. Package ctrdata can help addressing or mitigating many issues, as exemplified in Table 4.

Table 4

Selected issues when using trial registers for research, and how package ctrdata can help