The GlobalSurg 3 team are currently drafting a manuscript describing the methods of the study. This morning we had a planning discussion on Slack – our favourite team collaboration tool.

We’ve decided to share our Slack Chat with you so you can follow our progress as we prepare the manuscript!

Katie Shaw [9:02 AM]

The GlobalSurg 3 team are planning a manuscript describing the methods of the study set up. This morning we’re having an online discussion to plan the paper

The unique selling points of the study are:

1. ORCID ID system for registrations
2. the database architecture
3. the validation study

Riinu Ots [9:04 AM]

Using the ORCID ID is a major difference between GS3, and GS1 or GS2 and it has made managing 3,000 surgeons and medical researchers around the world a lot easier

Katie Shaw [9:05 AM]

This is going to make the mammoth task of arranging the authorship list at the end of the study a lot easier

We hope!

Ewen Harrison [9:07 AM]

Does https://orcid.org provide an ID for hospitals as well, or is it just for collaborators?

Katie Shaw [9:07 AM]

If only!

ORCID ID is for collaborators only but @Riinu has a great database now of curated hospitals throughout the GlobalSurg studies

Riinu Ots [9:08 AM]

Good question and we’ve thought about this a bit – they don’t, they only do people.

And I can understand why – hospitals can be reorganised – grouped together or broken into different units. People are easier to number than hospitals

So yes, we have created our own World hospitals database with unique Hospital IDs, available here: https://github.com/SurgicalInformatics/world_hospitals

A curated list of world hospitals that have taken part in GlobalSurg studies – SurgicalInformatics/world_hospitals

It is still very tricky deciding at which point do two separate buildings but right next to each other become different hospitals/the same hospital

Katie Shaw [9:11 AM]

Yes – big campus style hospitals can be tricky. And places with very similar names.

So we’ve put a lot of work into identifying all the hospitals and naming them in a consistent manner. This should help with consistent naming of hospitals in any future GlobalSurg study too

Ewen Harrison [9:12 AM]

A worldwide registry of healthcare sites would be a fantastic resource. Each hospital and health centre together with the services available there. But as you say, tricky to build and maintain!

Tom Drake [9:12 AM]

An ORCID for hospitals would be very useful, is this something Globalsurg could contribute to? How do the WHO do it?

How to keep it up to date? Would need department of health buy in from most countries

Or maybe not!

What about mobile devices and GPS mapping of sites

Riinu Ots [9:14 AM]

We keep it up to date by running a GlobalSurg project every 2 years – the list includes hospitals, including name changes and updates from 2014 (GS1), 2016 (GS2), 2018 (GS3)

Cameron Fairfield [9:14 AM]

It would be a great resource to have. There would always be a lot of ongoing maintenance of such a resource required. When the neurosurgical unit moves from one hospital and the paediatric unit from another hospital to the Royal in Edinburgh the departments associated with each unique ID would change

Riinu Ots [9:15 AM]

Yes exactly, our own hospital is no exception with new buildings and many different surgical departments moving around

Katie Shaw [9:16 AM]

And in having a name that can be expressed in 2 different ways – Edinburgh Royal Infirmary and Royal Infirmary of Edinburgh

Riinu Ots [9:17 AM]

That is true – we have had collaborators requesting different official names for their hospitals. We usually try to go with what is on their website – but sometimes even different parts of a website give different names (especially if translated from English) for a hospital

Tom Drake [9:18 AM]

Does Google have all the hospitals?

Presumably they have most

Riinu Ots [9:21 AM]

Google tends to have all the hospitals, yes. But it doesn’t know which departments it has, e.g. are general surgery and colorectal surgery in the same unit or not. But we need to know this for validation

Tom Drake [9:22 AM]

And in some places general surgery is everything

Ewen Harrison [9:18 AM]

What is a hospital? Surprisingly difficult to define! A combination of a physical location, the team of healthcare professionals, and the portfolio of diagnostic and treatment services available?

Riinu Ots [9:19 AM]

As well as the health records system! Some big hospitals have different systems set up for different units – this is very important for our validators – who need to be independent from the primary team, but at the same time have access to all the relevant information.

Katie Shaw [9:19 AM]

This is a good example of how something that seems relatively straightforward becomes increasingly complicated on a global scale – a principle that would apply to any organizational unit used during a worldwide study, not just to hospitals.

Ewen Harrison [9:22 AM]

Well it is great to be actively tracking collaborators using ORCID, that is going to be very helpful in the future, not only for publishing. And ORCID must be happy that we are pushing their system so hard internationally! How many collaborators do we have now on GlobalSurg 3?

Riinu Ots [9:22 AM]

Over 3,000 collaborators in GlobalSurg 3!

Riinu Ots [9:23 AM]

Over 80 countries.

Tom Drake [9:25 AM]

Amazing! Impressive how the workflow and @Riinu’s ORCiD system has enabled this.

Katie Shaw [9:22 AM]

Lets talk about validation, since you mention it!

Katie Shaw [9:23 AM]

As far as we know, we are the only international collaborative attempting to validate such a large dataset (>15,000 patients)

This is going to really help improve the quality of the data

Stephen Knight [9:24 AM]

Morning

What are the key steps in validation? Could we develop a CONSORT equivalent for validation of prospectively collected observational data?

Tom Drake [9:25 AM]

With validation particularly difficult to know what’s accurate at time and whether that was recorded accurately for the validation. Is there any way of addressing this?

Ewen Harrison [9:25 AM]

I think a flow chart is a great idea. Best practice in validating observational data collection at scale would be a great paper. (edited)

Tom Drake [9:26 AM]

Yes, a reporting or validation quality system would be a great idea

Cameron Fairfield [9:26 AM]

It raises an interesting question about “gold standard” and how that is measured particularly in a global setting. It also raises a great question about how a project such as GlobalSurg can be set up to maximise the ability of validators to accurately detect patients who have had an operation

Tom Drake [9:26 AM]

Not just for observational data! But for administrative data and trial data too

Katie Shaw [9:28 AM]

A beneficial side effect of validation is that we are doing a lot of the data cleaning as we go along; in preparation for releasing the data to validators we are undertaking a lot of checking of the primary data and where necessary asking teams to double check data points. For example, if data is missing or if the dates of operations fall outside of the stated 4-week data collection period

All of which is making the data better quality

And the systems that @Riinu has developed with live outputs to dashboards has really enabled this to work much more efficiently

Riinu Ots [9:29 AM]

One of the main rules of our validation process is that we do not change the data as a result of validation. The data is put through several internal checks before we release it to be validated by the independent validator, and in some cases it turns out the data is not of sufficient quality to be included in the study at all. But we do not use validation to remove data – more to audit our own quality checks after we have completed them

Stephen Knight [9:29 AM]

Including mini-team organisation, ORCiD, data cleaning and data quality rules would make a great flowchart of the methods

Cameron Fairfield [9:30 AM]

That’s a very important point. Helping the collaborators spot problems with their data such as operation dates outside of their collection period will be extremely important to helping validators identify patients and improves the validation process

Katie Shaw [9:31 AM]

Completely agree @cameron_fairfield. At the moment our National Leads have access to a lot of this information in a password controlled browser app but we’re already planning how a version of this could be made open access to allow all collaborators to easily spot where anything might be missing

Ewen Harrison [9:32 AM]

Can we see some screenshots of the National Leads App?!

Riinu Ots [9:34 AM]

Tom Drake [9:32 AM]

That’d be cool @katie_shaw. Would people be tempted to game the data? @cameron_fairfield’s rules are great for data quality!

Stephen Knight [9:34 AM]

Definitely could be tempted @tdrake, perhaps knowledge that their data could be validated reduces this? It may not?

Katie Shaw [9:36 AM]

I think there can be a temptation to game the data – not so much in entering incorrect data, but in trying to get everything to pass certain tests or checks without really appreciating how this might affect the rest of the validation process. For example, if you have an operation date outside of your stated period, there is a natural temptation to simply extend your data collection dates to include this operation. However, if you’ve not included every other case up to this date the validator will report the case ascertainment is incorrect

Riinu Ots [9:37 AM]

Yes, that has happened a couple of times – people “extending” their data collection period, rather than fixing the outlier’s date (probably entered as a typo). It’s important to try to communicate why we are so strict around the dates – yes, they are not important for statistical inference, but are important for identifying cases in the independent validation stage.

Tom Drake [9:36 AM]

Most people wouldn’t obviously! But needs some careful thinking! Validation might help, but what happens if you’re friends with your local validator?

Katie Shaw [9:37 AM]

We are trying to emphasize that the validator should be independent of the primary data collection team

Or at least perform the validation in an independent manner – rather than simply asking the primary data team ‘did you enter all the cases’, the validator should check themselves

Ewen Harrison [9:37 AM]

Trust and scientific rigour are important principles in any research. Validation helps provide a framework for that.

Katie Shaw [9:38 AM]

In summary, we are aiming in our methods paper to explain how the study has been set up and operated so that others interested in performing similar studies can share our experiences

And this doesn’t have to be only relevant to international patient cohort studies, it could be any type of study collecting any type of data at scale

Riinu Ots [9:42 AM]

Or not at scale – even a single hospital can use our methodology to organise an internal audit if their normal electronic health record doesn’t yet gather all the information automatically

Or non-electronic health record.

Katie Shaw [9:43 AM]

Thanks everyone, great discussion – draft of the manuscript in preparation!