23 and Me skepticism

By Eric Crampton 28/11/2013 24


I agree entirely with Alex Tabarrok’s post on the ridiculousness of the FDA’s shutting-down 23 and Me.

Further, I would love to subscribe to a service like 23 And Me. But I haven’t yet.

Let me reframe things just a bit.

We’ve recently found out that the NSA has basically been able to see everything that Google sees. And Google knows just about everything about me. I have zero worries about Google knowing everything about me – they just use it to better target ads, and I see that as a plus rather than a minus. They might turn ads from a nuisance to offers to sell me things I’d actually value at more than their selling price? Oh Nos! But having that big database of everything just sitting there is way too tempting for government. What they can’t subpoena from it, they’ll hack their way into and steal. Things that seemed kinda crazy conspiracy theory territory just a couple years ago… well, priors get updated.

23 and Me, run by Anne Wojcicki, Sergei Brin’s partner, is putting together a great big database of everybody’s DNA. Sure, for now, they’ve not run the full analysis along all base-pairs, just key markers. But we’ll get there as costs drop. Eventually they’ll have a great big searchable database of millions of people’s DNA. And it will be great, just like Google is great. We’ll find out all kinds of associations between genotype and phenotype we’d never otherwise have known about. Rare diseases will get spotted early, lives will be saved. Maybe we’ll even get some kind of merger with Google profiles to make it even more accurate. And the ads I get will be even better (again, I see as a plus).

But a database like that is awfully tempting, isn’t it? The first temptation will be searches for matches against samples from crime scenes. Even identifying the close kin of criminals can help pin them down. And it will be popular. Because finding and stopping criminals is popular. So warrants asking 23 and Me if they have any matches for particular samples will go down a treat.

Then we’ll start seeing 23 and Me elevated risk profiles for criminal activity. Parents will want to know if their kids are at higher risk for antisociality in their teenaged years, so we’ll see these start showing up in their profiles.

And my won’t those profiles be tempting. And we know that Google’s been way less vigilant against designing things to harden against government attacks. Where that’s normally irrelevant, it’s pretty hard to believe that we’d have that different a culture in a company run by Brin’s wife.

The odds of all this turning into pre-crime identification are pretty low. Maybe 5%.* But downside costs seem high.

If a service like 23 and Me started up in China, I’d likely sign up. I cannot imagine the Chinese government giving two whits about the genetic profile of individuals in New Zealand, or their being at all likely to respond positively to subpoenas from America or elsewhere asking to trawl through their databases. And they at least try to keep the NSA out of their stuff.

Transactions costs to New Zealand kept us from signing up to 23 and Me a few years ago. For once, I’m kinda glad about those kinds of costs.

* I suggested a bet with Bryan Caplan on this stuff on Twitter. He reckoned 1% chance; I’d thrown up a 10% ballpark. We’ve settled on 5%, but need to flesh out the details. I’d expect something like “If, by 2025, mainstream media reports that government agencies are using DNA database profiling to find groups more likely to commit crimes, and uses the information to target social assistance or criminal justice agencies for crime prevention, Bryan owes Eric $20. Otherwise, Eric owes Bryan $1. If the FDA mess effectively closes down 23 and Me (substantial drops in subscriber base), bet’s off: it’s a conditional bet.”

I’d be happy to go to one chance in four that 23 and Me is served a warrant asking them to match crime scene samples against their database by 2025 as well.


24 Responses to “23 and Me skepticism”

  • Seems less like reframing the issue and more like tackling a different issue. Unless I am much mistaken (which I would accept) none of this is the basis of the FDA’s problem with the 23&Me service.

    The issue seems to center around 23&Me showing that the tests are valid and reliable as well as the issues around what lay people might with the information if it is not properly related. This ties in (as covered here) with the general topic of screening tests and whether they are a good idea when you apply them to the general population. Screening tests work best when applied to the population that is most likely to benefit from them.

    More info is not better if it leads to worse outcomes over all – unnescassary (possibly invasive/dangerous) tests. Meddling with drug dosages, unclear risk ratios etc..

  • Darcy is correct. The FDA warning letter makes it very clear that 23andme has made claims that require FDA approval of their device.

    There are several issues, and the main current problem is 23andme’s refusal to provide information to the FDA on the accuracy, specificity, appropriateness, and precision of their measurements. The major concern is that such tests could lead victims of 23andme’s selfish marketing to make ill-informed decisions about future treatments.

    If 23andme had been a major multinational pharmaceutical company that ignored such warning letters, then fines of several hundred million dollars along with immediate plant closures would be imposed by the FDA, and the general public would cheer.

    The arrogance of 23andme is breathtaking, given how much work the FDA has already put into helping them. 23andme are already backtracking – as the days to closure count down. Here is a brief introduction to the real issues – just bypass any Forbes ads that appear….

    http://www.forbes.com/sites/matthewherper/2013/11/25/23andstupid-is-23andme-self-destructing/

  • I have been putting together a draft of an article on this over the past few days, but have been too busy to complete it just yet (as I noted in my latest post). Hopefully this will touch on the science & regulatory issues; I hope to also discuss the wider issues of regulating DTC personal genomics.

    The problems you describe are not the problems (several) that 23andMe are facing. The ‘pre-crime’ stuff hasn’t anything to do with 23andMe’s issues.

    23andMe are not “putting together a great big database of everybody’s DNA—unless they are doing whole genome DNA sequencing on-the-sly—they are measuring SNPs, single DNA base changes that may (or may not) be associated with different traits or diseases.

    Also – the problems with ‘predicting’ disease or traits from genome sequences are not with collecting millions of people’s DNA (per se), or some kind of a google search on them (we have already had that for many years), but with how to determine what, if any, effect the variations observed in a person might have.

    (Disclosure: I’m a computational biologist – a person in the field that analyses this sort of data.)

  • 1. Alex Tabarrok, at the link at the front, covers off those issues.
    2. I said, “Sure, for now, they’ve not run the full analysis along all base-pairs, just key markers. But we’ll get there as costs drop.”
    3. All I need is that government agencies come to think there is a link between particular types and enhanced risk. I’m sure I’ve seen papers linking MOA-A variants to various pro-sociality measures. I’m not saying those papers are all right, just that they could plausibly be looked at by a government agency as something that could predict crime. Again, I put a 5% chance.

  • Eric,

    I had already read his article, and he’s not addressing the FDA’s documented concerns in their letter.

    The link I provided has links to the FDA letter, which clearly explains that claiming any device produces diagnostic tests also requires validated scientific evidence of the accuracy and appropriateness of the underlying data, as well as the validity of any diagnostic interpretation provided.

    There is at least one report from Germany where the 23andme results were incorrect.

  • And all that would make sense if, say, surgeons were relying on 23andMe. But for a consumer product that can’t really lead to any treatment outcomes other than through physician intermediation, I don’t buy it.

  • Eric,

    Is your previous comment giving three numbered items in reply at me? (It would help you would address your comments.) I’m thinking not as point 3 doesn’t ‘connect’ with what I wrote. For clarity, though: I was writing about what you wrote, hence the quotes (not what someone else wrote) and point 2 wouldn’t help as that phrase clashes later wording you wrote as I pointed out.

    • Ah, but I wish that we here ran discus threaded comments, Grant. Yes, I was there replying to you.

      1. You noted I wasn’t hitting the “big issues”. I here noted that Alex Tabarrok had hit those “big issues” entirely to my satisfaction. As I had nothing to add to what he wrote about those, I just linked and agreed.
      2. I very well know that 23andMe isn’t putting together full geonome sequencing. I pointed you to the line where I said that. But as costs come down, it would be pretty surprising if their service didn’t move that way. Further, many of the markers they are finding could well provide correlates with things the government might reckon sufficiently correlated with criminal propensities that they get interested.
      3. You said that the main problem is in determining the effects of various variations. I argued that, instead, whether or not there is rock-solid basis for the government’s expecting that some set of genes correlates with criminal propensities, if they believe there is sufficient basis, they might well be interested in getting access to the big database that tells them which people have those markers.

  • ” And all that would make sense if, say, surgeons were relying on 23andMe. But for a consumer product that can’t really lead to any treatment outcomes other than through physician intermediation, I don’t buy it.”

    False negatives may be even more dangerous than false positives – as users may not seek further medical assistance or review of the data.

    Not all test results require surgical, or even medical, intervention. A possible scenario is that charlatans could offer cheap alternative “treatments” for people concerned about abnormal/unusual results, perhaps even offering free 23andme tests to prospective clients.

    23andme tried to avoid regulation by claiming that the tests were not diagnostic, but then marketed them as diagnostic, thus triggering the FDA overview. They then tried to ignore the FDA.

    Simple really.

  • Eric,

    Threaded comments may help, but so would addressing comments meant for one person 😉

    In your replies my original statements stand.

    Regards “You noted I wasn’t hitting the “big issues””, you agree with me that that your post does not address them but seem to be missing that I was offering a confirmation for others who appear to have read your post as if it meant to address the issues 23andMe have. (Note my first comment is open, i.e. addressed to any reader.)

    Regards, “You said that the main problem is…” + some stuff about ‘crime markers’. Not what I was saying; try reading back. (There’s more to making use of SNPs, and other sequence variants, than just collecting a bunch of sequences or markers; you’ve inaccurately paraphrased me BTW.)

    “ it would be pretty surprising if their service didn’t move that way” – but that’s not how your post reads & what I replied to.

  • I also read Alex’s post and remain unconvinced that the relavent issues are well enough understood to have been adiqualtey addressed.

    For one thing Alex says:

    “Moreover, the FDA wants to judge … the clinical validity, whether particular identified alleles are causal for conditions or disease. The latter requirement is the death-knell for the products because of the expense and time it takes to prove specific genes are causal for diseases.”

    What? So 23&Me can market tests for diseases before they have determined that the test actually correlates causally with said disease and that’s a good thing?

    The FDA is asking for the information that 23&Me must already have in order to honestly market these things in the first place. To argue otherwise means you must also agree that selling drugs to the public before they have been shown to be safe and effective is also a good thing. Clearly people like Stanislaw Burzynski are doing nothing wrong and the FDA should leave them alone as well.

    Either 23&Me has this information and is sitting on it, or it doesn’t and is dishonestly marketing the kits. In either case I don’t see 23&Me as doing anything virtuous.

  • Erics posts is freely proffering “bets” to an outcome.

    There is a time and place for “risk analysis” – especially in “social engineering”. Be in fluoridation, vaccination or general public health rather than gambling on an outcome.

    But to just throw out a bet for 2025 without analysis of the risk/benefits – of such database use by nefarious organisation, govt or private – seems to me to be purely a gamble.

    The consequences of (eg) insurance companies enabling them to “load the dice” (get that?) in who they accept premiums from leaves the vast and I mean the vast** – population at risk of no cover. Such an outcome will demand socially responsible support to the majority. But it’s OK, again, the 2% will make their profit.

    ** 85+% of use will utilise intervention in their lifetime. ( I bet $25 that figure is within 10% of the real number)

  • @Darcy: In that regime, FDA makes it impossible to run anything like 23andMe. If you need to go through the whole rigamarole for each new correlation found, it’s pretty much game over. Way too expensive to run. Maybe the whole thing moves to Singapore or someplace sane instead.

    I also have no problem with the marketing of drugs after they’re proven safe and before they’re proven effective. Delays by the FDA due to efficacy trials kill about 10 people for every person they save.

  • So if the test doesn’t measure what its been sold as measuring that’s ok? Alrighty then

    I also didn’t say safe but not effective I said safe and effective. But in that case companies should not claim they are effective if the evidence isn’t there to back them up. But this appears to be a clash of the role of evidence in public domain and how it should be used. a value judgement. In which case I now bow out.

  • @Darcy: FDA requires efficacy trials after safety trials, adding years to approval process. The whole thing is a nonsense though as a drug can be prescribed for off-label uses where efficacy hasn’t been proven once the drug is released. So prove efficacy for X (years of work), then prescribe for all kinds of stuff.

  • ” Delays by the FDA due to efficacy trials kill about 10 people for every person they save. ”

    Credible reference please ( ie medical, preferably with no economists involved ).

    I’m curious to learn how somebody can determine that trials ( which are not by the FDA, but the design and outcomes may be agreed in advance ) will kill 10 people for each person saved.

    If a drug fails Phase 1 or 2, there will be insufficient data to ascertain the number of deaths if it had progressed to market. Besides, death is only one adverse outcome, other side-effects often stop drug candidates.

    ” I also have no problem with the marketing of drugs after they’re proven safe and before they’re proven effective.”

    The current FDA requirement is that new drugs into a market must exhibit greater benefits ( improved efficacy and/or reduced side effects ) compared to current products. You can’t “prove safety” separately from effectiveness. First the drug has to pass preclinical animal trials, then the three phases of clinical trials.

    Phase 1 is for short term toxicity, and is usually on healthy people ( except for highly toxic drugs, eg cancer drugs – which may be evaluated on terminally ill volunteer patients ). The number of subjects typically ranges from 20 to 80.

    Phase 2 is for effectiveness, and is on a small subset of sufferers of the target ailment. It helps refine both the efficacy and dose, and also monitors short term side effects and longer term toxicity. The trials involve comparison with a placebo or an approved treatment, and usually involves 30 – 300 participants.

    Phase 3 studies gather more information about safety and effectiveness, studying different populations and different dosages and using the drug in combination with other drugs. The number of subjects is typical 300 – 3,000 people.

    Even after Phase 3, there will be post market requirements to monitor safety and effectiveness and to refine and define potential patients ( eg exclude pregnant ).

    From the above, there is no “safety OK” point that is separate to effectiveness, as the dose makes the poison.

    For drugs treating a serious ailment/disease with no current effective treatment, the FDA can expedite a drug programme, usually going to market after short trials, but with much-increased monitoring.

    Several recently-released drugs have been recalled because of serious side-effects ( premature death being one ) in the general population that weren’t detected in the clinical trials.

    The vast majority of drug candidates that reach Phase 1 never make it to market, often for unexpected safety reasons, as that may define maximum permitted dose. In NZ, usually around funding round times, we often see publicity about new drugs for cancer, perhaps after some preclinical testing in animals. Few even reach Phase 3.

    With regard to 23and me, the FDA is asking them to show their selected genotyping markers are appropriate for each diagnosis, and that 23and me can show data confirming their interpretation.

    Just because they have selected some markers with known associations to an ailment/disease, that doesn’t mean they
    have the causal markers, or even the majority of significant markers. The FDA wants to see the diagnostic evidence that 223andme should have, or at least be collecting.

  • Mind circling back around to 23andMe and explaining why no evidence is required to show the test is valid?

    To address drugs (which – don’t forget – is a side issue analogy) my understanding is that off-label prescriptions is not a lawless wasteland where anything goes; doctors still have to be able to point to studies that show efficacy or at least plausibility – not prescribe things willy-nilly.
    Whether this is adequately enforced is again another issue but I gather this is the ideal.

  • OK, I wrote…
    ” Credible reference please ( ie medical, preferably with no economists involved ). ”

    What I got in reply were links to opinion articles by…

    1. A Cato Institute lawyer.
    2. An Economist
    3. Abigail Alliance advocates, one a Geologist, one with an English language qualification.
    4. An Economist.

    I’m really sorry, but I don’t think my composition was that bad, so I have to assume your comprehension is substantially inferior to what I was expecting.

    Continuing this dialogue is clearly a waste of our resources, so bye…

    • I said those were pointers, Bruce, not that they’re the lit. They all cite a broader lit. The Tabarrok piece in particular cites the med lit. Further, it’s economists who you ought be looking to for the effects of different regulatory regimes.

      My specific source on the 10:1 figure is Gieringer, as cited by Tabarrok. Geiringer is available here. http://www.ncbi.nlm.nih.gov/pubmed/11616801

  • “no problem with the marketing of drugs after they’re proven safe and before they’re proven effective.”
    I suggest a close reading of Ben Goldacre’s excellent book “Bad Pharma”. Why on earth should people or their insurers pay for a drug that is safe but doesn’t actually do anything for them?

    On the ‘off-label’ comment – yes, & we are seeing that abused to the max by people like Stanislaw Burzynski, who puts together random cocktails of off-label drugs & with them claims to be able to cure cancer.

    Re 23&me – I’m with Darcy & Bruce on this one. Why is it OK to claim that a test is a useful indicator if there is no evidence demonstrating that said test actually does what’s claimed for it? That’s open to potential misuse by life & health insurers, for a start, not to mention the ethics around telling an individual they may/may not be predisposed to a particular disorder if the test is not accurate.