Getting full ECG through AliveCor?

A new clinical trial on the portable ECG (electrocardiogram) device AliveCor has been published. According to mobihealthnews, a study that aims to determine whether an ‘iphone ECG’ is capable of detecting heart attacks (specifically STEMI, which is a type of heart attack) with the same efficacy as the 12-lead ECG normally used in hospitals has been registered on clinicaltrials.gov, a website for listing clinical trials. The sponsoring institution for this study is Intermountain Health Care.

Clinicaltrials.gov is a website operated by the NIH where clinical trials are registered before they are started. The reason for this inconvenient process is to prevent ‘take and run’. In the case where clinical trials are conducted through the support of corporations like pharmaceutical companies, if the results are negative, it is likely that they will not be published. By registering clinical trials before they are conducted, one is able to keep track of the ongoing research, even if the results are not published in journals. Also, when the results are not so positive, it is possible that one might alter primary or secondary outcomes after the clinical study is terminated in order to produce positive results, and the website aims to prevent this.

It is stipulated in the Helsinki Declaration of 2008 that “Every research study involving human subjects must be registered in a publicly accessible database before recruitment of the first subject.” The International Committee of Medical Journal Editors has made it mandatory to register trials in the clinical trial registration system in order for researchers to publish studies in member journals. As a result, many countries have created clinical trial registration systems and these are often synchronized to the American website, clinicaltrials.gov. Thus, it can be said that clinicaltrials.gov is the most comprehensive clinical trial registration site in the world.

 

It is not clear whether the iphone ECG is AliveCor, but last year, Intermountain Health Care and AliveCor conducted a similar clinical study with 6 patients. The purpose of this study was ‘gaining experience with smartphone ECGs prior to conducting multi-institutional clinical trials comparing the smartphone ECG with the 12-lead ECG’, and one might guess that the current study is an extension of this previous study. The previous study was published, and since there is one more study prior to it, I will call this study “early study 2”.

I was very surprised to hear about the early study 2 and the study that is beginning now. This is because the original ECG shows 12 images, while AliveCor only shows a single image. Thus, the latter is not appropriate for detecting cardiac diseases other than abnormal heart rhythms (=arrhythmia) like atrial fibrillation.

To explain further, when taking a normal ECG, a total of 10 electrodes are placed: one on each hand and foot, and 6 on the chest. 12 images are obtained as a result, and each image corresponds to certain parts of the heart. As a result, if there is an abnormality at a specific part of the heart, it can be detected on one of the 12 images that corresponds to that part. (For instance, if a cardiac infarct occurs at the lower portion of the heart, an abnormality can be detected in the image that corresponds to leads II, III and aVF.) This is why it may be difficult to detect diseases that affects specific parts of the heart, like heart attacks, with a device that is only capable of acquiring a single image like AliveCor. Even the preexisting ECG only shows the anterior, base and lateral side of the heart, and does not detect well the abnormalities that occur in the back of the heart. (In this case, one must place 3 additional electrodes on the back to detect the infarct.)

If one is lucky and the myocardial infarct occurs in the front part of the heart and thus is detectable with AliveCor, or a medical personnel appropriately changes the location of the device, a diagnosis may be possible. An example of this would be when Dr. Eric Topol, the guru of digital healthcare, once saved a patient by diagnosing a heart attack (normally, this means myocardial infarct) with AliveCor on a plane.

However, the limit to using only a single lead is evident, and the original clinical studies on AliveCor focus on detecting abnormalities in cardiac rhythm like atrial fibrillation. For your reference, the portable ECG devices that can acquire up to 12 images are available in the market.

 

With this is mind, I was surprised to discover that the clinical studies from last year and this year (pilot studies to be more accurate) target the ECG’s of patients diagnosed not with arrhythmia but with myocardial infarct, and compare those obtained with AliveCor with those obtained with 12-lead ECG’s. ’What sort of trick is this,’ I thought.

There is not much information on the new clinical trial that is registered on clinicaltrials.gov, so I searched publications from last year. I discovered that there was one more study prior to the study mentioned above. I will call this ‘early study 1’. Regrettably, both articles are available to readers for a fee. The abstract is free, but does not contain detailed information. For those who are unwilling to pay but would like access to more detailed content, I recommend reading an article from Fierce medical device.

 

The title of early study 1 is ‘Surface 12 lead electrocardiogram recordings using smart phone technology). In the study, 12 ECG’s were taken of 5 subjects who were either normal or had cardiac abnormalities (LVH, LBBB, RBB) using AliveCor. These were compared to the results obtained with the standard 12-lead ECG.

The title of early study 2 is ‘Smart ECG for evaluation of STEMI: Results of the ST LEUIS Pilot Study’. In the paper, 6 STEMI patients were evaluated with the standard 12-lead ECG as well as AliveCor, and the results from the standard ECG were compared to those from the 12 different images taken with AliveCor.

Obviously, the key was how to obtain the 12 images with AliveCor. Early studies 1 and 2 both seem to have used the same method. The secret is contained in the figure published in early study 2.

AliveCor was not used exactly as it was sold, but was modified. In this manner, the leads were attached to different positions on the limbs and the chest, and after 12 times, 12 images were acquired.

Specifically how the images were obtained can be understood by looking at the picture in early study 1.

 

I am not familiar with the precise mechanism of electrocardiograms even though I am a board-certified physician. However, it is important to point out that although the method of obtaining aVR, aVL and aVF is different from that which applies to standard ECG’s, the images obtained through the novel method proved to be quite sufficient, to say the least.

In what sort of situation would using AliveCor to obtain 12 different ECG’s be efficient? This may be useful in nearly every situation where it might be difficult to use standard ECG’s. The authors of the paper point out in particular that this will be useful in rural areas that lack healthcare facilities, in developing countries and on cruise ships and buses.

It may be possible for the average person to utilize this at home, but realistically, it does not seem likely that a layman without proper education will be able to correctly take 12 images one at a time. In this case, one may consider having educated personnel use the device in certain areas (such as in health centers in rural areas).

If the device is hard to use, what if one was instructed on how to take ECG’s in real time using the video function on smartphones? This is of course possible but one must take into account that ECG’s do not provide a complete solution for diagnosing cardiac diseases. To repeat a story from my book, imagine a situation where there is chest pain that suggests myocardial infarct:

  1. Even when there is no abnormality on the ECG, one cannot rule out the possibility of a myocardial infarct
  2. Although chest pain in a patient may be suggestive of myocardial infarct, it may be due to conditions like aortic dissection or pulmonary embolism that are no less dangerous than the myocardial infarct and are difficult to diagnose via ECG. Therefore, in the case of severe chest pain, if it is not impossible to go to a hospital, the best thing to do would be to race to the hospital, rather than try to obtain 12 images using AliveCor.

AliveCor’s recent research efforts to obtain 12 images like the standard ECG seem to be in order to overcome the limit that acquiring only a single image places on the original product, which can only be used to diagnose cardiac arrhythmia. It is likely that AliveCor may already be developing a product that operates on the same principle as the device shown in the first image, but looks better and is more convenient to use. If so, the target market will be rural areas that are distant from hospitals or developing countries, However, in these cases, especially in developing countries, funding for healthcare will be limited and will need to be distributed efficiently. Practically speaking, cardiac diseases (especially diseases like myocardial infarct) are secondary in importance to infectious diseases, and purchasing AliveCor may not be the most cost-efficient decision. Whatever the case, as I have presented in an earlier post, AliveCor is presumed to have sold fewer devices than expected, and this may be an opportunity to overcome this problem.

Looking into the industry of telemedicine: the listing documents for Teladoc

In addition to Fitbit, which I mentioned in a previous posting (the link can be found here), Teladoc, the first and largest telemedicine company in the U.S., was listed the NYSE. This means that just as Fitbit had done in the past, Teladoc must now reveal a significant amount of internal data.

Teladoc submitted its S-1 for its listing in the U.S. stock exchange at 4:53PM, Friday May 29th, in U.S. time. Those who work in relevant industries in the U.S. diligently studied these documents over the weekend and submitted their analyses on Sunday, May 31st (the relevant link can be found here). These analyses are acceptable but their content appears rather insufficient, so I made a study of the S-1 of Teladoc. The contents mentioned here may strike those familiar with the industry as obvious, but some points may be of interest to others.

 

What is the business model of Teladoc?

The main business model for Teladoc consists of making a contract with employers or insurance companies so that the employees or members can receive remote medical care. Therefore, it is a type of B2B2C model.

I thought that direct-to-consumer (B2C) model would be a significant part of their business but this does not seem to be the case. Of the companies included in the Fortune top 1000 list, 160 are customers of Teladoc and the number of members is estimated at 11,000,000 (this is the number of members affiliated with client companies).

Major clients include

Companies such as Accenture, Pepsi, Shell, T-mobile, Bank of America, General Mills

Insurance companies such as Aetna, Amerigroup, Blue Shield of California, Centene, Highmark, Universal American

Medical institutions such as Mount Sinai, Health Partners, Henry Ford, Memorial Hermann

Teladoc makes a contract with these client companies so that in exchange for providing remote medical treatment, it can bill clients for a regular subscription fee (a fee for the right to use the service for a certain period of time) and a separate fee every time a member actually uses the service. The company pays for the regular subscription fee while the additional fee per usage may be covered by the company or the individual member. The annual sales for 2014 was $43.5M, of which 85% consisted of regular subscription fees, while 15% consisted of fees per usage. The total number of remote medical treatment usage in 2014 was 300,000.

Also, Teladoc claims that the number of current (eligible) members is 11M but this supposedly includes only a portion of those affiliated with companies that made a contract with Teladoc. Therefore, this means that client companies are providing Teladoc services to only a part of their employees or members. Teladoc claims that the total number of members affiliated with client companies is 50M and that it will be able to continue growing just by maintaining its current clientele and ensuring satisfaction, which will lead to the extension of contracts.

 

What is the structure of sales and costs?

Teladoc-financial-statement

Teladoc-financial-statement

Above is the part of the Teladoc’s income statement. Sales increased from $19.9M in 2013 to $43.5M in 2014, by 119%. Recently, sales increased from $9.4M in the 1st quarter of 2014 to $16.5M in the 1st quarter of 2015, by 75%. This shows that the company is growing rather quickly. However, it continues to record significant losses.

What is interesting is that in 2014, approximately 300,000 cases of remote medical treatment were reported. In another section of the document, the cost per session of remote medical treatment was stated as $40. If you multiply 300,000 by $40, you get $12M. However, it was stated above that of the total sales from 2014, which is $43.5M, of which 15% comes from fees for remote medical treatment, but 15% of $43.5M only amounts to $6.5M. Therefore, the cost actually paid for by the users of remote medical treatment is estimated at just over $20 per case.

Of the expenses, the cost of revenue needs to be examined. This can be considered as the cost of goods sold (COGS) and includes:

#1 the amount paid to doctor

#2 the cost associated with operating the network, when a contract is made with a doctor via physician networks.

#3 the cost of operating a call cente

#4 insurance premium for medical malpractice insurance

The proportion of the COGS out of total sales was approximately 23% in 2014. Previously, it was mentioned that 15% of total sales consisted of fees from remote medical treatment. Thus, I may assume that the amount paid to doctors stand somewhere between 15 and 23 percent of the total revenue. This means that doctors receive at least all the fees from remote medical treatment and maybe a bit more.

Suppose that the doctors take all of the fees from the remote medical treatment. The total amount in 2014 would be $6.5M (15% of $43.5M). Since the number of medical personnel that made a contract with Teladoc (doctors + behavioral health professionals) is 1,100, if you divide the total amount by 1,100, you are left with approximately $6,000 for income earned per individual per year.

According to what Teladoc reported in its S-1, medical personnel who participate in Teladoc is capable of earning up to $150 per hour. According to the 2013 edition of Becker’s Healthcare Report, if you consider the fact that the hourly income of a doctor working full-time is $99, the amount earned via Teladoc exceeds this by more than 50%. Furthermore, some doctors working for Teladoc supposedly report an annual income of over $100,000 (in order to earn over $100,000, I guess that a doctor would have to be working almost full-time for Teladoc).

Another point that needs to be emphasized is that of the income earned by Teladoc, the amount taken away by doctors is smaller than expected. Approximately 15-20% is taken by doctors, and considering the fact that Teladoc only provides outpatient services without complicated lab tests, procedures and operations, one cannot help but remark that Teladoc pays doctors less than what I expected.

 

What about medical specialties?

Telemedicine companies including Teladoc usually deal with rather simple diseases that require immediate symptomatic treatment, like the common cold. In the S-1 document, Teladoc claims that it mainly treats upper respiratory tract infections, urinary tract infections, sinusitis and dermatologic diseases. It also provides treatment for anxiety and quitting smoking through behavioral health professionals.

However, mentioning future growth strategies, Teladoc states that it will expand into novel clinical areas. For example, these include standalone dermatology services, secondary medical opinions and treatment for chronic diseases like diabetes. Telemedicine companies like Teladoc used to avoid treating chronic diseases like diabetes and hypertension while claiming that these needed to be treated in a long-term relationship with a doctor. However, they are now showing a change in position.

 

Are customers satisfied with the services provided by Teladoc?

According to a self-conducted survey by Teladoc, 95% of users reported being satisfied, but it is hard to tell from this result alone. Thus, Teladoc has set forth another index called Annual Net Dollar Retention Rate. This is a type of growth rate in the number of long-term contract customers, and calculation is as follows:

You divide the total regular subscription charges of the customers who maintained a contract of over 12 months and divide this by the total regular subscription charges of all customers from the previous year. Teladoc claims that this value is 104%.

I can understand that Teladoc wants to secure the trust of the investor, but honestly, it is hard to imagine what sort of meaning is contained in this index. It appears as weak as the PAUs (Paid Active Users) set forth by Fitbit.

 

Points related to quality assurance of outpatient services and treatment

In the S-1, parts related to the remote medical treatment provided by doctors practicing telemedicine are also mentioned. Before every session, a doctor reviews the content of the EMR (belonging to Teladoc) for each patient and goes through a checklist. During and after treatment, the doctor utilizes over 100 clinical guidelines that are proprietary to Teladoc (these are called proprietary Evidence-based clinical guidelines, and how it differs from other guidelines eludes me). After treatment, the doctor shows personalized notes and educational materials to the patient, who is then able to pose questions to the clinical team through the Teladoc message center. The quality assurance team reviews 10% of all remote medical treatment and checks if the treatment and prescription were appropriate. Up until now, Teladoc alleges that it has not received a single medical malpractice claim.

 

Points related to reduction in medical costs

Teladoc provides services by making a contract with employers or insurance companies, and it is thus important to offer value to these clients. Teladoc emphasizes the fact that these companies can reduce medical costs through telemedicine. It has conducted studies on this through external organizations and the results are discussed in the S-1.

Two types of studies were conducted

 

1. Episode-based Analysis Methodology

: This is a study that compares first-time users of Teladoc to patients who visited outpatient services or the emergency room for similar diagnoses and conditions. The study targeted two client companies. Teladoc screened patients in both groups based on 16 characteristics, and claims that an objective comparison was thus possible. The results (as predicted) is that Teladoc is capable of reducing medical costs. For one client company, a Teladoc customer saved $1,157 per outpatient visit compared to the outpatient clinic or the emergency room (ER). In addition, Teladoc claimed that it yielded a $9.1 ROI (return on investment) for every dollar invested by the client company.

Another client company claimed that by using Teladoc, it saved $284 compared to using outpatient services and $2,419 compared to using the ER.

This result has the potential for generating controversy. One must take into account that currently, those who use Teladoc has mild diseases such as the common cold and allergies. It is possible that those primarily using Teladoc may not have had to visit the ER or the outpatient clinic and would have instead bought over-the-counter medication for the common cold or antihistamines, but this is not taken into account. If so, it is not that Teladoc allowed users to reduce medical costs by saving them a trip to the ER, but actually incurred further medical costs when in fact no costs could have been incurred, just by allowing access to remote medical treatment.

 

2. Per-Member-Per-Month Analysis Methodology

: This study targets a single client company and compares medical costs prior to May 2012, before it started using Teladoc, to costs after May 2012, after it started using Teladoc, and analyzes how they changed. To be more specific, based on medical expenditures and healthcare usage behavior over the course of 16 months prior to May 2012, it generated a prediction model and compared this to actual medical costs and healthcare usage behavior from May 2012 to December 2013.

In a study that targeted a single client company, a member (not someone who used Teladoc but who received the right to use Teladoc from the employer) saved an estimated monthly average medical fee of $21.30. When the entire company was considered, the monthly average healthcare cost was reduced by 9.8%.

The second study is difficult to evaluate because the exact research methods are unknown. Cost reduction up to 10% appears to be very attractive. Moreover, assuming that this result is true, the flaw pointed out from the results of the first study does not appear to be very problematic.

 

To sum up, up until the present, Teladoc has maintained a high growth rate and it seems that it will continue to do so for a while. However, contrary to expectations, the proportion of the revenue paid to doctors is rather low, so it is debatable whether Teladoc will be able to continue to secure excellent doctors. Furthermore, up until now, Teladoc has dealt with simple and easily treatable diseases such as the common cold and allergies. However, in the future, Teladoc claims that it will expand its target to chronic diseases like diabetes and hypertension. What kind of influence this will exert on the medical community in the days to come is of particular interest.

How much has IBM’s Watson improved? Abstracts at 2015 ASCO

Every year around end of May, the American Society of Clinical Oncology (ASCO) holds an annual meeting. It is the most representative of oncologic societies, which eminent scholars in the field from all across the world participate. Although my specialty is not in oncology, I have been waiting for the meeting, during which abstracts on IBM’s Watson have been presented regularly since 2013. The reason I looked forward to the meeting so much is that it is difficult to obtain information on exactly what stage of development the product has reached aside from the rosy reports released by the company, as is the case in other products in digital healthcare. ASCO presents an opportunity to obtain relatively objective information on Watson’s level of development.

This time, without fail, four abstracts on research that applied IBM’s Watson to the field of oncology were presented. Through these, I attempted to obtain clues into the current stage of development for IBM’s Watson.

First, I introduce abstracts related to Watson that were presented in the previous ASCO meetings.

In the 2013 ASCO meeting, Memorial Sloan Kettering Cancer Center (MSKCC), which has been IBM’s partner since early on, presented an abstract titled, ‘Beyond Jeopardy!: Harnessing IBM’s Watson to improve oncology decision making.

Focusing on lung cancer, the authors evaluated natural language processing (NLP) and machine learning (ML). Watson was instructed with 525 actual lung cancer patient cases and 420 virtual patient cases.

The result is summarized in the table below.

Watson-at-ASCO-2013

Watson-at-ASCO-2013

The abstract claims that NLP, which is the ability to extract key pieces of information from patient cases, and ML, which is the ability to recommend adequate treatment plans, were both enhanced after repeated tests.

On examining the table above, it is hard to tell whether NLP is actually improving based solely on the results from Batch 1~7. However, as one can discern from Batch 8~16, if 300 cases are repeatedtly tested, the ability to recommend the correct treatment plan increases from 40% to 77%.

One may wonder whether it is appropriate to determine the ‘correct treatment plan’ based on the judgment of experts at MSKCC. If this is a result that numerous specialists from eminent cancer centers such as MSKCC have agreed upon, I think it is safe to do so. However, it may be hard to trust the diagnosis if it only has an accuracy of 77%. Still, given that the accuracy increased rapidly with repeated instruction, it is feasible that the reliability will increase even further with sufficient programming.

 

Even more abstracts were presented in the 2014 ASCO meeting.

First, MSKCC presented research that further expanded from that of 2013. It developed a similar model on colon cancer, rectal cancer, bladder cancer, pancreatic cancer, renal cancer, ovarian cancer, cervical cancer, and endometrial cancer in a study titled ‘Next steps for IBM Watson Oncology: Scalability to additional malignancies. Watson was repeatedly instructed with this model and the proportion of correct treatment plans that were recommended was investigated. .

It was shown that with repeated instruction, Watson increased in accuracy with respect to all types of cancers. However, the caveat was that the same patient cases were repeatedly tested, just as in the 2013 study. In order to have clinical significance, Watson needs to show outstanding diagnosing ability on new patient cases after it has been instructed with the patient cases. However, it has not yet reached this stage.

The MSKCC has produced another set of study results called Piloting IBM Watson Oncology within Memorial Sloan Kettering’s regional network. Oncologists in hospitals affiliated with the MSKCC network evaluated Watson’s ability to recommend correct treatment plans for breast, colon and rectal cancer and gave feedback on their experiences using Watson. Only 6 people were included in this study, so there may be limits to this study. Nevertheless, the users evaluated Watson to be helpful in their decision-making process. However, they claimed that too much time was spent on entering in patient data, as they were required to input over 20 unnecessary components. They pointed out that Watson should be able to collect data directly from preexisting materials, and that it will have to provide more evidence for its decision to rank treatment plans in a certain order. Thus we can say that although Watson is said to have natural processing power, it still has to recognize data that is entered in by the doctor.

Meanwhile, MD Anderson, another world-class cancer center, presented research on leukemia. According to an abstract titled MD Anderson’s Oncology Expert Advisor powered by IBM Watson: A Web-based cognitive clinical decision support tool, Watson was instructed with 400 leukemia patient cases and the treatment plans based on the treatment decisions of oncologists at MD Anderson. Overall, Watson’s accuracy reached 82.6%. It was not indicated whether Watson evaluated the same patient cases that it was instructed with or whether it was given a completely novel patient case following instruction. To me it seems like it was instructed in the same way as in the MSKCC study.

Based on research presented from 2013 to 2014, we may conclude that Watson’s natural processing ability does not yet meet expectations, and that although it has an excellent learning ability, its ability to apply its knowledge to novel patient cases has not been sufficiently verified.

 

Now, we will examine abstracts that were presented this year.

MSKCC, a long-time partner of IBM, once again presented an abstract this year. The abstract, which is titled Assessing the performance of Watson for oncology, a decision support system, using actual contemporary clinical cases, evaluated Watson’s ability to apply its knowledge to new patient cases (just as we have awaited). 20 thoracic cancer cases (the abstract stated that thoracic medical oncologists chose the patient cases, but considering the fact that the category within which this abstract was included was Lung cancer-Non-Small Cell Metastatic, these cases may be on non-small cell lung cancer)  who underwent treatment for the first time were used and each patient case was provided with sufficient diagnostic materials, including molecular pathology lab results. These cases were entered into Watson as structured attributes. Watson and MSKCC medical personnel were told to classify possible treatment options for each patient case into categories of ‘Recommended’, ‘For Consideration’ or ‘Not Recommended’. Subsequently, it was determined whether or not the decisions made by Watson and MSKCC medical personnel corresponded to one another.

50% of the options that were ‘Recommended’ by MSKCC medical personnel were also ‘Recommended’ by Watson. Of those ‘Recommended’ by MSKCC, 25% were ‘For Consideration’ and 25% were ‘Not Recommended’ by Watson. 16 cases consisted of metastatic lung cancer, and of the chemotherapy agents actually used by the MSKCC medical personnel, 88% were classified as either ‘Recommended’ or ‘For Consideration’ by Watson. There were cases of which the treatment option was “Recommended’ by MSKCC medical staff but ‘Not Recommended’ by Watson and these were the cases of elderly patients with co-morbidities that Watson had not yet learned.

The authors thus concluded that that Watson’s choice of options came within the boundaries of evidence-based medicine. They also claimed that Watson will be able to increase its accuracy by repeated training with medical personnel and further development. (Still, given the values above, it is questionable whether Watson truly comes ‘within the boundaries of evidence-based medicine.’) However, they pointed out that in the case of elderly patients with comorbidities that have heterogeneous treatment options, Watson still faces obstacles that it needs to overcome.

The MSKCC produced yet another abstract, which dealt with metastatic breast cancer. The research was titledSteps in developing Watson for Oncology, a decision support system to assist physicians choosing first-line metastatic breast cancer (MBC) therapies: Improved performance with machine learning. This abstract is rather difficult to understand, so I will attempt to restructure the information in it . For those who are unsure of my explanation, I recommend reading the original document.

Even when they have many characteristics in common (such as age, activity level, expression of receptors, and baseline treatment), metastatic breast cancer (MBC) patients tend to undergo very different types of treatment. This difference may be found in the choice of chemotherapy agent or in the selection of hormonal therapy agents. Doctors assign weightings to individual factors related to breast cancer (such as the location, extent, size of the tumor, and the severity of symptoms) and by combining these choose the best treatment plan. Researches have been conducted on how to best assign weightings as to help predict a patient’s prognosis and come up with treatment decisions.

By showing Watson on how MSKCC specialists made decisions specific for each case of breast cancer, this research aimed to instruct Watson how specialists assign weightings and thereby improve its ability to recommend treatment options. 101 manufactured MBC cases were used for this process of instruction.

As a result, when all 101 cases were evaluated, the accuracy improved from a pre-instructional value of 73.6% to a post-instructional value of 82.1%, which represented an increase of 11.5%. When the cases were analyzed based on HR and HER2 status, an increase of 28.8% for HR+HER2+, 9.6% for HR+HER2-, and 2.8% for HR-HER2+ were reported, while a decrease of 1.4% was reported for HR-HER2-. (For those of you who are not familiar with breast cancer, HR and HER2 mean hormone receptors crucial to breast cancer physiology and treatment)

Ultimately, Watson’s mechanical learning model was able to make decisions that more closely resembled those of MSKCC specialists when instructed with manufactured cases and decision-making logic than when instructed with just algorithms (originating from preexisting guidelines).

The results of this study are very surprising. Many people, including myself, thought that Watson was only capable of  handling clinical situation based on preexisting research results and guidelines, and that it would not be able to recommend treatment plans for complex cases not yet sufficiently addressed in published papers or textbooks. Thus, it was unfathomable that tacit knowledge learned through experience by the best of experts could also be acquired by Watson. However, it now seems likely that with appropriate instruction, Watson will be able to reproduce exactly that kind of knowledge. If Watson’s ability to assign weightings in clinical decision-making is improved, this means that it will ultimately reach a stage where it can generate medical evidence and guidelines on its own.

The MSKCC produced three sets of abstracts, the last of which is on early breast cancer. Its title is Integration of multi-modality treatment planning for early stage breast cancer (BC) into Watson for Oncology, a Decision Support System: Seeing the forest and the trees.

While metastatic breast cancer is only treatable with chemotherapy and hormone therapy, early breast cancer requires surgery and may additionally necessitate axillary lymph node dissection or radiotherapy. In the cases where the cancer is of genetic origin, one may need to seek a consult for genetic counseling. Depending on treatment options, one may also need to seek a consult for fertility preservation, which could be potentially harmed or altered by chemotherapy agents. Given that multidisciplinary team intervention, in which doctors of diverse specialties partake in treatment, is becoming even more widespread, one will need to determine whether Watson is capable of playing the role of the primary oncologist.

MSKCC’s breast cancer specialists evaluated how well Watson can seek consults for lymph node dissection (BS), radiotherapy (RT), clinical genetic counseling (CG) and fertility preservation (FP) after instruction.

When compared to expert opinion, Watson’s ability to seek consults for RT matched 98% of the time; for CG, 94% of the time; and for FP, 91% of the time. Furthermore, in terms of BS, it recommended surgery for all 8 of the cases where an expert determined that surgery was necessary. Of the 12 cases where surgery was not recommended, Watson recommended surgery in 7 cases. The authors concluded that Watson’s performance was quite excellent.

The last abstract was presented not by the MSKCC but by the BC Cancer Agency (BCCA) Genomic Sciences Centre of Vancouver, Canada. It has the title, Implementation of Watson Genomic Analytics processing to improve the efficiency of interpreting whole genome sequencing data on patients with advanced cancers.’ The ability to provide information and therapeutic modalities based on the results of genetic analyses to assist with treatment was evaluated. While it took a human being over 10 days to complete this task, Watson finished its analysis within a matter of minutes (the abstract includes many anecdotes on genetic analyses that I have not discussed here because I am neither very knowledgeable nor interested in them). When it states that Watson is capable of analyzing big data quickly, this abstract presents nothing new, so I will not discuss it further.

I would like to share my thoughts on the first three abstracts presented at 2015 ASCO meeting. It is unclear whether the same cases were consistently tested as in the abstracts presented at previous meetings, or whether a novel case was applied after repeated instruction. Since no additional comments were made, the former appears to be the case. In that case, it may be premature to apply Watson to the clinical setting.

According to reports by IBM in October, 2014 (the link can be found here), a 5-year contract was made to use Watson in oncology in Thailand’s Bumrungrad International Hospital. If Watson is still at the stage where it is working on improving accuracy with repetitive instruction, it does not make sense to apply Watson to the clinical setting in Bumrungrad Hospital. This made me curious more about Watson’s current level of ability and its purpose in Bumrungrad. (Because I was curious, I even studied Bumrungrad’s annual report for 2014, but it only mentions the implementation of Watson, and nothing more specific. For your information, Bumrungrad Hospital is listed in the stock market.)

 

I was most surprised by the content of the second abstract. As I mentioned previously, it implied that Watson is not only capable of organizing existing information that is scattered, but can also combine this to synthesize new medical knowledge.

Overall, it seems that we are still far from implementing Watson in the clinical setting of cancer treatment (based on only the content of the presented abstracts). Underestimating the power of technology often leaves us sorry, but there currently seem to be limits to using Watson, even to simply assist doctors.  Still, there is no doubt that Watson will continue to improve and will eventually end up playing the role of the doctor in a majority of cases.

Witnessing the bare truth of the industry of activity trackers: the listing documents for Fitbit

 

As companies grow up and get ready to be listed on the stock market, they are required to reveal internal data to protect investors. In order to be listed in the U.S. stock market, one is required to submit a document called the S-1, and to meddlers like myself, the information contained within is, while insufficient, interesting and difficult to obtain otherwise.

Fitbit, which has attained a solid No. 1 position in the market for activity trackers, submitted its S-1 that was released by the U.S. stock commission (to view the original document, click here).

 

In fact, the listing of Fitbit in the stock market and its disclosure of the S-1 is a well-known fact, but media coverage of this issue deals mostly with the revenue, net profit, and sales of the company. As an aficionado of data, I was aware of the fact that careful examination of the S-1 could reveal more interesting information. I began to read the document that amounts to nearly 200 pages, but only managed to read a few pages due to a lack of time.

However, I came across the fact that Rock Health, well-known digital healthcare incubator, had already analyzed and presented the S-1 of Fitbit (the related link can be found here). Because the material is great, I would like to start with it and put my own analysis on it.

The key point of this document concerns an issue that those interested in wearables are most curious about, which is “how many users continue to be users.”

 

Before going over the document, I would like to show how the estimates were made in the first place through consumer surveys. The best known survey result comes from a report called Inside Wearables, which was presented by Endeavour Partners, a consulting firm (the original link is here).

According to the data, which was presented by the rather provocative title, ‘Dirty little secret of Wearables,’ one-third of activity meter users cease to use the product in 6 months. This company has been conducting consumer surveys every year, and the result is as follows:

Abandonment rates

Figure 1. Abandonment rates

This content will be included in the new report that is soon to be made public, and you can see that the user dropout rate increased by a significant amount after 2013. This result content is not limited to Fitbit but is constantly used when dealing with the engagement issue of wearable devices.

 

Now, let us move on to the Rock Health analysis.

I will only deal with the parts related to engagement issue, but the original article containing other material that may be interesting to the viewer depending on his or her viewpoint, so I recommend that those who are very interested in wearables read the original document at least once.

Fitbit users

Figure 2. Fitbit users

I have redrawn the figure from the analysis by Rock Health using the S-1 data (the unit is in millions).

Regarding the number of users, the S-1 reveals three important pieces of information. These are the total number of devices sold, the total number of registered users and the total number of paid active users (PAUs).

The concept of PAUs was developed as a marker for those who actively use the device. Instead of Monthly Active Users (MAU), which is frequently used in the app industry, Fitbit has set forth, in a sense, a marker of its own.

The definition of PAU is a person who has satisfied at least one of the following criteria over the past three months.

  1. Someone who has held an active account of Fitbit premium or Fitstar.
  2. Someone who synchronized a meter or weight scale to the Fitbit account
  3. Someone who measured over 100 steps of activity or weight

What do you think? Do these criteria sufficiently define someone who actively uses Fitbit? For now, I will just move on to the next part.

 

What is remarkable from the figure above is the proportion of PAUs out of the total number of registered users. In 2014, this number was 46%; in 2015, it is approximately 50%. Aside from how accurate the definition of PAUs is, since approximately 50% are actively using Fitbit, could I say that Fitbit’s engagement is better than expected?

However, upon careful examination of the figure, you can see that the number of Fitbit users has recently increased by a significant amount. This is good for corporate performance, but from the perspective of PAUs, this means that the number of recent subscribers far exceeds that of past subscribers. Given the fact that those who have held a subscription for a longer period of time are more likely to terminate their usage, the percentage above falls short of accurately reflecting the actual proportion of active users.

 

The clever analyst of Rock Health was also aware of this point and decided to further refine the data.

Devices sold, change in PAUs from previous year, decrease in PAUs

Figure 3. Devices sold, change in PAUs from previous year, decrease in PAUs

If the previous was cumulative data, this figure is organized by incidence per year or per quarter (the unit is in millions.) For each period, three types of data are given, and the one to the far left is the total number of devices sold. For example, 10,900,000 devices were sold throughout the year of 2014. The red bar in the middle represents the change in PAUs from the previous year to the given year. For instance, at the end of 2014, compared to the end of 2013, the number of PAUs increased by 4,100,000 people.

Isn’t something a bit off?

10,900,000 more devices were sold but the number of PAUs increased by only 4,100,000. This makes sense if each user purchased 2 or 3 devices of Fitbit. However, in the first figure that was shown (figure 2), if you compare the cumulative number of devices sold and the total number of users registered, there is not much of a difference. Thus, this means that Fitbit users have not purchased multiple devices (we will discuss this later in more detail.)

If so, throughout the course of a year, 10,800,000-4,100,000=6,800,000 people have stopped using Fitbit. This is the number shown in the far right in Figure 3.

Furthermore, if you return to the figure shown at the very top (figure 2), the number of Fitbit PAUs at the end of 2013 is 2,600,000. If we make the most conservative assumption by claiming that all of these users dropped out during 2014, those who purchased Fitbit in 2014 who did not become PAUs or became dropouts amount to 6,800,000 – 2,600,000 = 4,200,000 people. Certainly, there must be those who purchased Fitbit prior to 2013 who continued to be users, so this is an extreme assumption.

As the definition of PAUs is based on activity over the course of 3 months, those who purchased Fitbit in the 4th quarter of 2014 cannot be included here. Thus, in the 4,200,000 people calculated above, only those who purchased Fitbit in the first 3 quarters of 2014 are included. Since the number of purchasers for the first 3 quarters of 2014 was not otherwise given, if we conduct a simple calculation by multiplying the number of purchasers during the year of 2014, which was 10,900,000, by 3/4, we obtain 8,180,000 people. Furthermore, if we assume that the number of purchasers in the 4th quarter of 2014 is the same as the number of purchasers in the 1st quarter of 2015, this results in to 10,900,000-3,900,000=7,000,000 people. Since the number of Fitbit purchasers is rapidly increasing, we can assume that approximately 7,000,000~8,000,000 people have purchased it. Of this number, if the previously calculated 4,200,000 people have dropped out, we may conclude that approximately 53-60% have stopped using Fitbit.

However, since we assumed that all who purchased Fitbit before 2013 have ceased to be users, the proportion of users who terminated their service in 2014 must be higher than this. In the analysis by Rock Health, it is shown that over 70% of users terminated their usage (it seems that Rock Health estimated the number of users in the first three quarters of 2014 based on Fitbit earnings per quarter). More people than expected are ceasing to use activity meters in a shorter period of time.

 

Also, we cannot overlook the fact that the definition of PAUs is very broad. From the point of view of healthcare, it is important to encourage the users to continue to use the device, but from a business point of view, if you can continue to sell as many devices as possible, user termination may not be a very significant consideration (I certainly believe that it is).

In order for Fitbit to continue to produce meaningful results, the generation of novel markets and repeated sales to its established customer base are both important. For a product whose consumer usage decreases by a significant amount within a year, both are not easy to achieve.

 

Now I would like to deal with the problem of making repeated sales to preexisting consumers.

Even without considering the data, ㅛㅐㅕ can imagine that there are not many people who will repeatedly purchase activity trackers like Fitbit. This is because it is highly unlikely that the sensor will be improved dramatically or a function beyond one’s imagination will be included. I will calculate a parameter based on the Fitbit S-1 data that will help us better visualize this.

If you divide the cumulative number of devices sold by the total number of registered users, the following is obtained.

 

2012 2013 2014 2015 1Q
Cumulative number of devices sold (in millions) 1.3 5.8 16.7 20.6
Total number of registered users (in millions) 1.1 4.5 14.6 19
Cumulative number of devices sold / total number of registered users 118% 129% 114% 108%

 

If you just consider the recent data, the average number of devices purchased per user is 1.08. Rock Health claims that this number is actually 1.09, but this seems to be derived from data that includes the number of devices sold prior to 2012. In my calculation, this data was omitted.

A phenomenon like purchasing a new smartphone every 2 years is not observed for Fitbit. Of course, we need to wait and see because the product has only recently been launched and the number of users has recently undergone a rapid increase. However, given the nature of the product, it will be of interest to see to which extent repeated sales of the product are possible.

 

There may still be limits to the data from Rock Health as it is based on PAUs, a concept developed by Fitbit.

In relation to this, another piece of data was reported. Recently, a certain media outlet mentioned the content of the Fitbit S-1. It claimed that it is difficult to estimate the extent of the continued user base of Fitbit simply based on the content of the document, and alluded to data from an outside company called Evidation Health (the link can be found here).

 

Evidation Health operates an activity information platform that enables users of over 100 fitness apps and devices to synchronize their personal activity information and provides points that can be converted to cash or prizes for those who have managed to maintain healthy lifestyles.

 

According to officials at Evidation Health, the company has managed the data of 20,000 Fitbit users since the latter half of 2010, and Fitbit users are actually very highly engaged and “those who purchased Fitbit tend to be very attached to their devices.”

Based on the data from this company, if we apply the definition of Paid Active Users from the S-1 of Fitbit, approximately half of PAUs have updated their activity information in 70 out 90 days. In addition, 5% of Fitbit users terminate their usage within a week of purchase, and 12.5% terminate their usage within one month of purchase.

 

If we define active users as those who have updated their activity information in 70 out of 90 days in accordance with the definition from the analysis by Evidation Health (this number is approximately half of PAIs), the dropout rate of Fitbit users increases dramatically. Based on this, I will reconstruct the second figure from above (figure 3). The number of PAUs was halved for the modification.

Devices sold, change in PAUs from previous year, decrease in PAUs , modified version

Devices sold, change in PAUs from previous year, decrease in PAUs , modified version

 

The method of calculation is as follows.

The number of active users at the end of 2013 was 13,000,000, and if we assume that all of these users ceased to use Fitbit, the number of dropouts in the first three quarters of 2014 is 75,000,000. Given the fact that the number of new users in the first three quarters of 2014 was 7,000,000~8,000,000, nearly all of them have chosen to terminate their usage. Since the data from 20,000 users was applied to the data from the entire company, there is certainly a gap between this calculation and reality, but the results are nonetheless surprising.

More surprising is the comment from Evidation Health that Fitbit users are more active compared to users of other devices. Aside from the data from Evidation Health, even by looking at the internal data from Fitbit, we can see that the user dropout rate is quite high. If so, then how much higher is the dropout rate for Jawbone or Misfit?

 

If Fitbit is listed, more data will be revealed via conference calls or corporate inquiries. Through the disclosure of such data, I hope that we will be able to gain better insight into the workings of the activity meter industry. The future of the activity meter industry is becoming the object of even greater curiosity.

How many of the portable EKG device, AliveCor, have been sold?

A diverse range of market research organizations and consulting firms have made projections on the size of the digital healthcare market and its future growth.

The work in estimating the market size cannot help but have its limits,                            and it is especially difficult in the case of digital healthcare,                                               for the market is still in its initial stages and most companies tend to be startups,            and therefore data for each company is hard to find.

Thus, instead of data on the size of the market, in many cases, we determine how ‘hot’ this industry is based on the scale of VC (venture capital) investment that can be objectively tracked. Companies like Rock Health and Startup + Health regularly present their data on the extent of funding.

In the process of collecting information on FDA regulations, I came across an interesting piece of data. I am sharing it because I think it reflects the current state of the digital healthcare market.

The data is related to AliveCor, known as the portable EKG device. To those who are interested in digital healthcare, AliveCor is considered to be a leader in the industry and is mentioned in related books, articles and lectures repeatedly. Currently, the 3rd generation of products is on the market.

How many devices of this well-known product have been sold? My guess was over 100,000 devices, and my acquaintances, when inquired, gave a similar reply.

Last February, there was an incident in which the app that runs with the AliveCor EKG failed to function properly, and since this app received FDA approval as a Mobile Medical App, the FDA ordered a recall. Of course, AliveCor fixed the problem without difficulty and there were no major problems. However, on the recall order, the number of actual users of the AliveCor EKG app on iOS was revealed.

FDA-recall-AliveCor

FDA-recall-AliveCor

The content of the order is the same as above (the link is here) and the actual number of users of the app is shown to be 5,600. Since it is impossible to use the AliveCor portable EKG without the app, the number of users of the app is considered equivalent to the number of current users of the portable EKG device.

This is the number of users using the most recent version at the time (version 2.1.2), and there may be more people who are using the previous version. However, in September 2014, a new app embedded with the function for diagnosing atrial fibrillation was launched, and afterwards, another that was embedded with the function for differentiating between normal states and uninterpretable states. From the latter half of 2014 to the beginning of 2015, the app for AliveCor had gone through revolutionary change. Given that this function was probably useful to AliveCor users, it is not a far claim to assume that the majority of users upgraded to the most recent version.

In addition, while the 1st and 2nd generation of AliveCor products are available only for iOS, the 3rd generation of products is availableintended for both iOS and android. Thus, one must take into account the fact that this number excludes android users of the 3rd generation of AliveCor products. Also, there may be those who purchased AliveCor but stopped using it after a few months.

However, even if we assume that only 10-30% of all those who purchased AliveCor are current users of the device. The total units sold of the AliveCor EKG device amount to 18,000~60,000 devices. If we apply the unit price of $150-200 per device (consider the fact that the unit price of the 2nd generation of AliveCor products is $200 and for the 3rd generation of products is $75), the total revenue for AliveCor up to the present is estimated at $2,700,000~12,000,000. If we consider the fact that AliveCor launched its products in March of 2013, we may estimate its sales over the past 2 years at  $1.3~6Mil.

Of course, we must consider the fact that the market is still in its early stages, but that one of the most promising products in the world’s largest digital healthcare market that is the U.S. has sales of this level reveals a great deal on the digital healthcare market. In particular, AliveCor is a B2C product that the consumer needs to purchase without insurance coverage, and this shows that doing business in digital healthcare targeting general consumers not an easy feat.