Can we cope with too much testing?

251017-cattle-is-tested-for-tb-c-tim-scrivener-142014-11659

When all this coronavirus business kicked off, it was foot and mouth that came to mind. After all it was a case of coping with a virus. But now it’s our experience with bovine TB that seems to be most relevant.

All cattle farmers are used to regular TB testing, and I’m pretty sure we’ve all heard the terms Sensitivity and Specificity.

 

Sensitivity – The probability that if the animal has the disease then the test will be positive. The higher the sensitivity the fewer false negatives but at the cost of false positives.

 

Specificity – The probability that if the animal does not have the disease then the test will be negative. The higher the specificity the fewer the false positives but the more false negatives.

 

The terms “sensitivity” and “specificity” were introduced by American biostatistician Jacob Yerushalmy in 1947. So you can blame him for picking two such similar words which makes them easy to confuse.

 

The advantage of the current skin test is that it has a high specificity (thought to be around 99.98%). This means if the animal doesn’t have the disease, the test will show it to be clear. You don’t get many false positives. This is important on a herd test, because you’re doing a lot of cattle and if you get a lot of false positives, a herd will never be clear of TB. Test a 10,000 cows and you’ll probably get one false positive.

The problem is that the skin test has a lower sensitivity, between 81% and 85%. This means that you will miss cows with the disease. On a herd basis this isn’t too worrying because herds are big enough that if the disease is there, it’s probably in several animals and you only need to pick up one for the whole herd to be locked down and repeatedly retested. Mind you it’s a nightmare for the farmer trying to get rid of the disease in his livestock.

Then you have the gamma interferon test. It has a sensitivity of over 90% which means you miss fewer cattle who have the disease. You can clear the disease out of a herd more quickly.

Unfortunately the specificity is lower than the skin test. It’s about 96.6% so you have a lot higher chance of getting false positives.

To give an example, with a 200 cow herd with the skin test you’d be unlucky to have a false positive. With the gamma interferon test you’d expect two false positives.

So using gamma interferon you actually clear the disease out of a herd faster than you do with the skin test. But in reality you never stop testing because of the levels of false positives.

This is why the vets use them in tandem. You can use gamma interferon and be confident you’ve probably detected the diseased animals. But you stick with the skin test for determining whether the herd is finally clear because you don’t get false positives.

 

So as a rule of thumb, if there’s a lot of the disease about, you want a test with a high sensitivity, because then you don’t miss the disease carriers and false positives are a very small proportion of those you find.

But if there isn’t a lot of the disease about you want a test with a high specificity because otherwise your false positives can end up outnumbering your real positives.

 

Moving on to coronavirus (COVID-19) tests, I’ve seen an article in the BMJ

https://www.bmj.com/content/bmj/369/bmj.m1808.full.pdf

 

This stated “Further evidence and independent validation of covid-19 tests are needed. As current studies show marked variation and are likely to overestimate sensitivity, we will use the lower end of current estimates from systematic reviews, with the approximate numbers of 70% for sensitivity and 95% for specificity for illustrative purposes.”

 

Whilst these are approximate numbers, they aren’t entirely encouraging. A sensitivity of 70% leaves a lot of room for false negatives. It could miss up to 30% of the people tested who have the virus. These people walk away thinking they don’t have it and might spread it through the population

On the other hand, at 95% specificity, this means that you’ll get 5% false positives.

In a large population with a lot of the disease, this isn’t too much of a problem. But let’s assume that you have a large population with very little disease.

Let us assume we test 100,000 people, and only 0.1% have the virus. This is probably the stage we’re at now and it means there are 100 people with the virus.

With our test, we would expect to get 70% of them, so we find 70 people with the virus. 30 are missed.

But with 95% specificity we’ll have 5000 people marked down as false positives. So at the end of a day’s testing we have 5070 people test positive and are asked to isolate. But actually only 70 of them actually have it and we haven’t got a clue who they are.

The problem with fetishizing testing in populations with low levels of the disease is that whilst the disease might have died out, you’ll keep finding it for ever because of the nature of the test.

Not only that but the more you test, the more false positives you’ll get. It strikes me that they’re using the tests wrong. If you have three people in a factory who go down with the virus, then by all means go in and test everybody in the factory. But I see little use in testing entire populations, especially when the level of cases gets down to the level of expected false positives.

 

Actually we’re currently field testing the specificity of the test even as I write.

On the 18th August there were 150,174 tests and 1,089 new cases. This means that the system probably has at least 99% specificity because otherwise you’d get a lot more cases, even if they were all false positives. But of the 1,089 cases, it would be interesting to know how many of them were false positives? If has struck me that the high number of ‘asymptomatic’ cases, plus the fact that we’re not seeing people being admitted to hospital might give us a rough indication that a fair proportion of these ‘cases’ are people who don’t actually have the virus.

 

Now I’ve just heard the Minister on the radio this morning saying they’re going to roll out mass testing. Even assuming the test has a specificity of 99.99% that means that when they test us all, they’ll find 6,000 cases even if nobody has the disease. Second wave here we come.

♥♥♥♥

There again, if you’re going to be locked down for no obvious reason you might as well have a good book to read.

A collection of anecdotes, it’s the distillation of a lifetime’s experience of peasant agriculture in the North of England. I’d like to say ‘All human life is here,’ but frankly there’s more about Border Collies, Cattle and Sheep.

As a reviewer commented, “This is a delightful collection of gentle rants and witty reminiscences about life in a quiet corner of South Cumbria. Lots of sheep, cattle and collie dogs, but also wisdom, poetic insight, and humour. It was James Herriot who told us that ‘It Shouldn’t Happen to a Vet’ but Jim Webster beautifully demonstrates that it usually happened to the farmer too, but far less money changed hands.

I, for one, am hoping that this short collection of blogs finds a wide and generous audience – not least because I’m sure there’s more where this came from. And at 99p you can’t go wrong!”

Tagged: , , , ,

13 thoughts on “Can we cope with too much testing?

  1. cindy knoke August 20, 2020 at 5:20 am Reply

    I’ll pass this on to my husband, a PhD, decades published, medical researcher bio-statistician.

    • jwebster2 August 20, 2020 at 5:43 am Reply

      I’d be genuinely interested to see what he has to say, I suspect I may have over simplified

  2. Eddy Winko August 20, 2020 at 5:29 am Reply

    You should be working on the ‘More or Less’ team.
    So this really is the new normal 🙂

    • jwebster2 August 20, 2020 at 5:45 am Reply

      ‘Seek and ye shall find.’
      It has been pointed out that people have been tested positive because they have RNA fragments on them left over from when they had the virus some months ago

      With TB we have had a lot of years to get used to the system and to know how to cope with it. The problem with this virus is that we have the media howling for a silver bullet to ‘fix everything’ and the politicians are frantic to give them one.
      Testing is not going to solve all their problems

  3. Doug Jacquier August 20, 2020 at 6:52 am Reply

    Fascinating, Jim. Please keep us in the loop.

    • jwebster2 August 20, 2020 at 7:06 am Reply

      Hopefully Cindy’s husband will get back to us if I’m too far adrift

  4. Cathy Cade August 20, 2020 at 10:06 am Reply

    That’s told me a lot I didn’t know. Thanks for some context.

    • jwebster2 August 20, 2020 at 12:09 pm Reply

      hopefully we get some feedback from somebody who knows this stuff as well 🙂

  5. Latest News – Lockdown Sceptics August 20, 2020 at 6:35 pm Reply

    […] ‘Can we cope with too much testing?‘ – Intriguing analysis of the PCR test by a farmer. He’s concerned about specificity and, yes, you guessed it, false positives […]

  6. Cynthia Reyes August 22, 2020 at 3:27 pm Reply

    Jim, I think you should add ‘scientist’ or ‘agricultural researcher’ to your credentials. When I read your posts, I am reminded of how much skilled farmers have to know these days. (Maybe they always did.)

    • jwebster2 August 22, 2020 at 3:32 pm Reply

      I’m one of the people who likes to know how things happen and why. And as you say, farmers have to be multi-skilled. To understand our industry you’ve got to understand the science, the law and the politics. Hence I enjoy opening the bonnet and having a poke around inside that lot 🙂

      • Cynthia Reyes August 22, 2020 at 4:14 pm

        My childhood was spent on a farm, and our family more recently owned a farm, yet I never got to understand things the way you do. Bravo to you

      • jwebster2 August 22, 2020 at 4:18 pm

        I got lucky, I was asked to various stuff in a voluntary capacity (helping farmers cope with the system) so I learned a lot very quickly and part of what I had to do was to rescue people from being crushed by the machine

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: