What Makes a Sensitivity Analysis?

Mindless & Modern Sensitivity Analyses


This post will be split into two sections, in the first part, I will discuss the nature and difficulty of conducting sensitivity analyses and the quality of many published sensitivity analyses in various scientific fields, including medicine. I will then discuss a general framework that has existed for a long time but seen a major resurgence by regulatory agencies around the world that may help researchers conduct these analyses and other analyses in general.

In the second part of this post, I will use similar data from a historical parallel-arm RCT and analyze it from start to finish, however, the caveat is that there are extensive missing data, therefore I will have to utilize a number of non ad-hoc SA techniques to better understand these collected data and how informative they are. Much of the code in this second part will be in R and Stata, but references will also be given to specific techniques and theory.


Cargo-Cult Sensitivity Analyses


Sensitivity analyses are an important part of statistical science and many other disciplines when conducted in a principle and systematic manner. However, in the published sensitivity analyses literature, there are many inconsistencies, misconceptions, and highly misleading findings from these analyses and what information they bring to the table. An obstacle that prevents these issues from occurring and also more researchers from conducting them, is that they (sensitivity analysis techniques) are quite difficult to learn and often advanced statistical methods.

Yet, the appearance of difficulty does not dissuade all researchers away from adopting them for their own work, (so that they may give their colleagues and the stakeholders they report to the impression of rigor and confidence), and when this is done carelessly, researchers will often skip over learning the details and theory, and like many statistical procedures, they will rely on default settings in statistical software – much of these procedures masquerade as thorough sensitivity analyses and are often a formality to appease stakeholders and give researchers a false sense of confidence about what they are doing. Thus, like many statistical procedures new and old, they too will inevitably be abused as they become more popular and as they are required in research reports.


As Stark & Saltelli (1), along with many others such as Gigerenzer (2) and Greenland (3) have written in the past,


In our experience, many applications of statistics are cargo-cult statistics: practitioners go through the motions of fitting models, computing p-values or confidence intervals, or simulating posterior distributions. They invoke statistical terms and procedures as incantations, with scant understanding of the assumptions or relevance of the calculations, or even the meaning of the terminology. This demotes statistics from a way of thinking about evidence and avoiding self-deception to a formal “blessing” of claims. The effectiveness of cargo-cult statistics is predictably uneven. But it is effective at getting weak work published - and is even required by some journals…



Statistical software does not help you know what to compute, nor how to interpret the result. It does not offer to explain the assumptions behind methods, nor does it flag delicate or dubious assumptions. It does not warn you about multiplicity or p-hacking. It does not check whether you picked the hypothesis or analysis after looking at the data, nor track the number of analyses you tried before arriving at the one you sought to publish - another form of multiplicity. The more “powerful” and “user-friendly” the software is, the more it invites cargo-cult statistics. (1)


Greenland (3) focuses on how traditional statistical education is inadequate to deal with the messy and chaotic nature of the world and the missing data that is produced by it. Greenland writes that traditional statistical practice opens the room for several cognitive biases, making many reports and conclusions highly misleading. He provides some hope however, by pointing to the utility of sensitivity analyses for modeling bias and uncertainty.


I argue that current training in statistics and analytical methods is inadequate for addressing major sources of inference distortion, and that it should be expanded to cover the biased perceptual and thinking processes (cognitive biases) that plague research reports. As commonly misused, null-hypothesis significance testing (NHST) combines several cognitive problems to create highly distorted interpretations of study results. Interval estimation has proven highly vulnerable to the same problems. Sensitivity and bias analyses address model uncertainties by varying and relaxing assumptions, but (like Bayesian analyses) they are difficult to perform with proper accounting for prior information and are easily manipulated because they depend on specification of many models and parameters.

Surprisingly, P values can be reframed to lessen cognitive problems by 1) presenting them without reference to a cutoff, 2) providing them for relevant alternatives to the null hypothesis, and 3) interpreting them with reference to all assumptions used in their computation rather than just the parameter they are tailored to test. P values, however, are poorly scaled for measuring evidence, a problem which could be addressed by transforming them into the information they supply against the model used to compute them. (3)


However, he then follows up to discuss how difficult sensitivity and bias analyses can be due to the plethora of decisions that the analyst must make when specifying parameters in the analysis. These judgements will differ from researcher to researcher, even in the same group, and such analyses have not been studied as widely as conventional statistical practice.


Even with realistic choices, the sensitivity of sensitivity and bias analyses must be evaluated (51). The plausibility of an estimated bias function is determined by intuitions, prejudices, and understanding of the applied context; those can vary dramatically across researchers, in turn leading to very different specifications and inferences even if they are anchored to the same conventional analysis. Adding to this problem, sensitivity and bias analyses are more difficult to perform correctly and more easily massaged toward preferred conclusions, because they require specification of many more equations and their parameters.

And unlike NHST, abuse of sensitivity and bias analysis is as yet barely studied because the pool of such analyses remains small and highly selective. It thus seems implausible that these analyses will increase replicability of inferences, although they can reveal how assumptions affect those inferences. (Here “replicability” is used according to recommendations of the American Statistical Association (52) to denote independent checks of reported results with new data; “reproducibility” then denotes checks of reported results using the original data and computer code.)


Indeed, it seems Greenland’s predictions and concerns were good approximations; in a large systematic review of published sensitivity analyses led by the mathematician Andrea Saltelli’s group, they found that highly cited papers in top impact factor journals rarely contained sensitivity analyses, and that many of these were poorly done and one-dimensional, in the sense of only varying one parameter at a time, which is hardly informative as a multi-dimensional analysis in which multiple parameters are varied at the same time.


Here we will explore one example of how mindless sensitivity analyses are conducted within the context of clinical trials, although we will not explore multi-dimensional sensitivity analyses here. I make this distinction because the environment to conduct sensitivity/bias analyses differs substantially in trials where regulators are involved versus the environment in which many observational studies are analyzed and presented.


In medicine, sensitivity analyses (SA) are conducted in some trials and observational studies, but again, often little thought is given to what they should entail. (1) None of this is surprising given that they are not usually taught in traditional settings, although they have been discussed extensively within the statistical and epidemiological literature (410) Yet, the other barrier that remains is that they are incredibly technical and will require the assistance of an analyst familiar with both the theory and the applications. It also does not help that the topic is vast and there are multiple new developments every day.

Indeed, the Handbook of Missing Data Methodology, (11) one of the most authoritative and comprehensive books to date on statistical methods to handle missing data, has nearly six chapters devoted to the topic of sensitivity analyses and different approaches to use. Unfortunately, at the rate that this field is advancing, this book may already be outdated, despite only coming out six years ago.

This article will focus on what a sensitivity analysis is often assumed to be based on practices in the literature, some differing points of view regarding sensitivity analyses (with an example of a highly promoted), measure for conducting sensitivity analyses that has been very controversial, and a practical example of conducting principled(by principled, I mean non ad-hoc techniques, as James Carpenter would say) sensitivity analyses in the context of missing data in a randomized clinical trial.


The Vagueness of ‘Sensitivity’


Before we move forward, we must consider what a sensitivity analysis means. The first word of the phrase is clearly an obvious tell, it suggests an analysis that examines how sensitive or robust a result or assumption is. But a natural question is, sensitive specifically to what? Can I vary anything in the analysis to test the sensitivity of the result or assumption? For example, if I obtained an odds ratio of 1.7 from the primary analysis in my experiment and I conducted an extra analysis to see whether the odds ratio would change by a resampling method, is that a sensitivity analysis?

What if I did a completely different type of analysis, for example, suppose the primary analysis was a classical statistical test with an adjustment for multiple comparisons and I decided to run a hierarchical Bayesian regression with a concentrated prior on the null, would that be a sensitivity analysis? The possibilities of what can be varied are endless when a loose definition of sensitivity analysis is assumed. This is unique to every discipline and their culture. In epidemiology, sensitivity analyses and bias analyses which attempt to extensively explore assumptions and the robustness of the result are used synonymous, yet in the clinical trial world, this is not the case.


Frequent Misconceptions


Take for example in in clinical trials and observational studies, in particular, it is common to see primary analyses often being intent-to-treat (ITT) analyses and sensitivity analyses being per-protocol analyses (PP).


I will not define these terms here and would encourage readers to consult other sources.


The following is a similar example from a high-profile trial published in the The Lancet in 2002, (12)


The Multicentre Aneurysm Screening Study group randomised 67,800 men to receive an invitation to an abdominal ultrasound scan or not [6]. Of those invited to receive an abdominal scan, 20% did not accept. The primary analysis was by intention to treat, thus estimating the effect of being randomised to abdominal ultrasound. Another analysis investigated the complier average causal effect, which considers what the (average) effect of treatment was in patients who would have adhered to protocol however they were randomised [7].


To many, this may seem perfectly fine, even great. The authors used the question they were primarily interested in as the main analysis, and then conducted an additional analysis to see if these results are consistent. Unfortunately, this is highly problematic as Morris et al. (2014) (13) describes


These questions are different, and observing different results should not shake our confidence in either. The CACE analysis was a secondary analysis, not a sensitivity analysis.

It is common for authors to compare the results of intention-to-treat with per-protocol analysis; see for example [8, 9]. While it is hard to pin down the precise question of per-protocol analysis [10], this is clearly different to the question intention-to-treat addresses. Per-protocol analysis should not therefore be considered as a sensitivity analysis for intention-to-treat but as a secondary analysis, if at all.


Randomness, Uncertainty, & Sensitivity


Portable Sensitivity Analyses


These sorts of misunderstandings are so common and so prevalent throughout the literature, so it should come as no surprise that sensitivity analyses are rarely done, or they are done incorrectly, or using questionable methods and assumptions. One particular controversial example has been the promotion and use of the \(E\)-value (the ‘E’ apparently stands for ‘evidence’, but I am not sure.) within epidemiology to assess the amount of confounding necessary within an observational study to practically reduce the effect estimate to something that is practically null.

The method and value has been adopted with open arms by many epidemiologists and health researchers given that it has simplified a task that is often arduous and requires extensive and careful thought in comparison to the traditional classical analysis. Yet, others have also been highly critical of it for a number of reasons, and although they see the value in promoting sensitivity analyses to more researchers around the world, many are concerned that this measure will also eventually go down the road of P-values, but for observational research.


Regulators & Sensitivity Analyses


As mentioned above, there are many possible ways to conduct a sensitivity analysis, especially if the phrase is used in a loose/vague way. Whether or not these are valid and principled approaches to conducting sensitivity analyses is another question. Luckily, many of us do not have to ponder day and night about the semantics about this because both The Handbook of Missing Data Methodology and the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) have given this topic much thought, and for the latter it is reflected by the fact that they recently created an entire new addition to their E9 guidance document (Statistical Principles for Clinical Trials) which has served as a reference for clinical trial statisticians for decades. In the addendum, titled Addendum on Estimands and Sensitivity Analysis In Clinical Trials, which has now been legally adopted by regulatory agencies around the world including the FDA and EMA, they elaborate on the concept of estimands and the role sensitivity analyses play.

To be clear, estimands are not new and have been discussed in the statistical literature since Tukey (14) but the ICH working group’s addendum formalized its adoption in the clinical trial world and the importance of sensitivity analyses. (1517)


Estimands & Sensitivity


A.5.2.1. Role of Sensitivity Analysis

Inferences based on a particular estimand should be robust to limitations in the data and deviations from the assumptions used in the statisticral model for the main estimator. This robustness is evaluated through a sensitivity analysis. Sensitivity analysis should be planned for the main estimators of all estimands that will be important for regulatory decision making and labelling in the product information. This can be a topic for discussion and agreement between sponsor and regulator.

The statistical assumptions that underpin the main estimator should be documented. One or more analyses, focused on the same estimand, should then be pre-specified to investigate these assumptions with the objective of verifying whether or not the estimate derived from the main estimator is robust to departures from its assumptions. This might be characterised as the extent of departures from assumptions that change the interpretation of the results in terms of their statistical or clinical significance (e.g. tipping point analysis).

Distinct from sensitivity analysis, where investigations are conducted with the intent of exploring robustness of departures from assumptions, other analyses that are conducted in order to more fully investigate and understand the trial data can be termed “supplementary analysis” (see Glossary; A.5.3.). Where the primary estimand(s) of interest is agreed between sponsor and regulator, the main estimator is pre-specified unambiguously, and the sensitivity analysis verifies that the estimate derived is reliable for interpretation, supplementary analyses should generally be given lower priority in assessment.

A.5.2.2. Choice of Sensitivity Analysis

When planning and conducting a sensitivity analysis, altering multiple aspects of the main analysis simultaneously can make it challenging to identify which assumptions, if any, are responsible for any potential differences seen. It is therefore desirable to adopt a structured approach, specifying the changes in assumptions that underlie the alternative analyses, rather than simply comparing the results of different analyses based on different sets of assumptions. The need for analyses varying multiple assumptions simultaneously should then be considered on a case by case basis. A distinction between testable and untestable assumptions may be useful when assessing the interpretation and relevance of different analyses.

ICH E9 (R1) Guideline

The need for sensitivity analysis in respect of missing data is established and retains its importance in this framework. Missing data should be defined and considered in respect of a particular estimand (see A.4.). The distinction between data that are missing in respect of a specific estimand and data that are not directly relevant to a specific estimand gives rise to separate sets of assumptions to be examined in sensitivity analysis.


they explicitly define a sensitivity analysis as being an analysis which realistically varies the assumptions from the primary analysis, still targets the same estimand, examines the robustness of the results to assumption violations or departures, and can possibly change the results/conclusions drawn. They contrast this with more extensive analyses that investigate these violations/departures of assumptions, and characterize them as being supplementary rather than sensitivity. The latter is typically what is seen in rigorous epidemiological studies that employ quantitative bias analyses using bounded methods or probabilistic ones.

The addendum contains similar views that are echoed by the National Research Council’s advice on clinical trials (National Research Council 2010) (18)


Recommendation 15: Sensitivity analysis should be part of the primary reporting of findings from clinical trials. Examining sensitivity to the assumptions about the missing data mechanism should be a mandatory component of reporting.



Now, back to the NEJM paper. The reason why an ITT analysis and a PP analysis cannot serve as primary and sensitivity analyses, respectively, is because they are targeting entirely different estimands. (13) Thus, because they are answering completely different questions, they’re just two different analyses. And the additional PP analysis, although important to certain stakeholders, can be classified as a supplementary analysis or a non-sensitivity analysis.

Indeed, the following flowchart from Morris et al. (2014) (13) is particularly useful as a guide to differentiate sensitivity analyses from non-sensitivity analyses.


Adapted flowchart from Morris et al. (2014)


Utility of Subject-Matter Experts


So what does a sensitivity analysis actually look like within the context of a clinical trial or experiment? Below, I walk through the analysis of a trial to give some examples.


Sensitivity in Clinical Trials


I’ll use a sample clinical trial dataset, which is described here: (19)


“This dataset is typical of diastolic blood pressure data measured in small clinical trials in hypertension from the mid-to-late 1960s and for approximately a decade thereafter.

During this time, hypertension was more severe, the number of effective treatments was relatively small, and the definition (DBP > 95 mmHg) of essential hypertension was not as stringent as it is now (DBP > 80 mmHg) - as seen in the 1967 report from the Veterans Administration Cooperative Study Group on Antihypertensive Agents VA Study Group (1967).

In Table 3.1, diastolic blood pressure (DBP) was measured (mmHg) in the supine position at baseline (i.e., DBP) before randomization… Patients’ age and sex were recorded at baseline and represent potential covariates. The primary objective in this chapter in the analysis of this dataset is to test whether treatment A (new drug) may be effective in lowering DBP as compared to B (placebo) and to describe changes in DBP across the times at which it was measured.


So to recap, the primary investigators of this trial were interested to see whether treatment A was more effective in lowering DBP when compared to placebo.

Although we will not be using some of the scripts and project setup that I discussed in my statistical workflow post for the sake of efficiency and ease, I would urge and recommend others do so that it remains easy to catch errors. In particular, I would especially recommend using the following packages:





I will now simulate some of the data from this historical clinical trial.


# Simulate Trial Data  -------------------------------------

set.seed(1031, kind = "L'Ecuyer-CMRG", 
         normal.kind = "Inversion", 
         sample.kind = "Rejection")
BaselineDBP <- rnorm(500, 117, 2)
Group <- rbinom(500, 1, 0.5)
Age <- rnorm(500, 50, 7)
R1 <- rnorm(500, 0, 1)
R2 <- rnorm(500, 1, 0.5)
errors <- rmvnorm(500, mean = c(0, 0),
                  sigma = matrix(c(1, 0.5, 0.5, 1),
                                 nrow = 2, byrow = TRUE))
PostDBP <- BaselineDBP + (Age * 0.33) + 
           (Group * 9.7) + errors[, 1]

I have made some minor adjustments to this dataset and also generated a new variable (Z) based on the existing ones in the dataset that is strongly correlated with the outcome of interest (PostDBP).



# Explore Data  -------------------------------------

str(BP_full)
#> 'data.frame':    500 obs. of  7 variables:
#>  $ PostDBP    : num  142 140 129 148 135 ...
#>  $ BaselineDBP: num  119 116 115 119 119 ...
#>  $ Group      : int  1 1 0 1 0 0 1 0 1 1 ...
#>  $ Age        : num  41.8 44.4 41.1 56.6 50.2 ...
#>  $ Z          : num  0.0141 -0.0629 -0.1101 0.0313 0.0495 ...
#>  $ R1         : num  0.523 1.021 -0.839 0.413 -0.175 ...
#>  $ R2         : num  0.451 0.35 0.897 1.227 1.166 ...

summary(BP_full)
#>     PostDBP       BaselineDBP        Group            Age              Z                    R1                  R2         
#>  Min.   :123.8   Min.   :109.9   Min.   :0.000   Min.   :27.65   Min.   :-1.845e-01   Min.   :-3.205036   Min.   :-0.6456  
#>  1st Qu.:133.4   1st Qu.:115.6   1st Qu.:0.000   1st Qu.:45.34   1st Qu.:-4.440e-02   1st Qu.:-0.732386   1st Qu.: 0.6447  
#>  Median :138.5   Median :116.9   Median :1.000   Median :49.37   Median : 9.454e-05   Median :-0.000058   Median : 0.9601  
#>  Mean   :138.1   Mean   :116.9   Mean   :0.516   Mean   :49.73   Mean   : 0.000e+00   Mean   :-0.042800   Mean   : 0.9620  
#>  3rd Qu.:142.8   3rd Qu.:118.2   3rd Qu.:1.000   3rd Qu.:54.69   3rd Qu.: 4.562e-02   3rd Qu.: 0.552206   3rd Qu.: 1.2832  
#>  Max.   :150.5   Max.   :123.1   Max.   :1.000   Max.   :68.45   Max.   : 2.025e-01   Max.   : 3.377526   Max.   : 2.4805

ztable(head(BP_full, 10))
PostDBP BaselineDBP Group Age Z R1 R2
141.67 119.47 1 41.82 0.01 0.52 0.45
139.98 115.56 1 44.40 -0.06 1.02 0.35
128.64 115.40 0 41.14 -0.11 -0.84 0.90
147.91 118.75 1 56.60 0.03 0.41 1.23
135.24 118.76 0 50.21 0.05 -0.17 1.17
135.14 118.27 0 55.33 0.01 -0.89 0.77
143.32 117.08 1 54.34 0.00 1.07 0.34
132.75 116.04 0 55.76 -0.05 -1.91 0.96
143.61 118.21 1 46.93 0.08 1.03 1.75
149.89 118.85 1 62.01 0.01 -0.94 2.48


Now, we can quickly explore the other characteristics of the dataset, and first we look at the rank correlations between the predictors and the response variable.


# Data Attributes -------------------------------------

study_formula <- as.formula(PostDBP ~ .)

plot(spearman2(study_formula, BP_full), cex = 1, 
     pch = 18, col = alpha("darkred", 0.75))

abline(v = 0, col = alpha(zred, 0.75), lty = 3)


Along with a correlation matrix between the variables of interest.



We will not be looking at the estimates from a fitted model yet. We are mainly looking at a few characteristics of this dataset before we move on to the next step.


Now that we have familiarized ourselves with this dataset, we suddenly “notice” that there are missing data (which I actually generated below using a custom function) and suppose we didn’t actually know how the missing data were generated, we might suspect or start with the assumption that the data are missing at random (MAR).


I do not plan on touching what the different missing data mechanisms are (MCAR, MAR, MNAR) as described by Rubin (1978). Although it is necessary to be familiar with them to better understand the approaches used throughout this post. Some individuals have remarked that the following diagram below in Statistical Rethinking helped them grasp these concepts, so I am pasting that here as a reference.


Adapted flowchart from McElreath (2020)


Of course, here, we do have an idea of what the missing data mechanism is, but we shall assume the role of a data analyst who has just been given a dataset with missing values in the outcome from their clinical colleague.


# Generate Missing Data  -------------------------------------

set.seed(1031, kind = "L'Ecuyer-CMRG", 
         normal.kind = "Inversion", 
         sample.kind = "Rejection")
Ry <- ifelse(lesslikely() > 0, 1, 0)
PostDBP[Ry == 0] <- NA
BP_miss <- data.frame(PostDBP, BaselineDBP,
                      Group, Age, Z, R1, R2)
str(BP_miss)
#> 'data.frame':    500 obs. of  7 variables:
#>  $ PostDBP    : num  142 140 129 148 135 ...
#>  $ BaselineDBP: num  119 116 115 119 119 ...
#>  $ Group      : int  1 1 0 1 0 0 1 0 1 1 ...
#>  $ Age        : num  41.8 44.4 41.1 56.6 50.2 ...
#>  $ Z          : num  0.0141 -0.0629 -0.1101 0.0313 0.0495 ...
#>  $ R1         : num  0.523 1.021 -0.839 0.413 -0.175 ...
#>  $ R2         : num  0.451 0.35 0.897 1.227 1.166 ...

sum(is.na(BP_miss))
#> [1] 245

We will now attempt to visualize our dataset and examine what proportion of these data are missing.


missing_col <- c(zblue, zred)
aggr(BP_miss, col = alpha(missing_col, 0.60),
     plot = TRUE, prop = F, numbers = T)



So now, we must examine this missing dataset and its characteristics and look for any systematic differences. We also examine the proportion of missingness and the influx and outflux patterns. This is important to assess how well connectected the missing data are to the observed data, explained by van Buuren here


The influx of a variable quantifies how well its missing data connect to the observed data on other variables. The outflux of a variable quantifies how well its observed data connect to the missing data on other variables. In general, higher influx and outflux values are preferred…

Influx and outflux are summaries of the missing data pattern intended to aid in the construction of imputation models. Keeping everything else constant, variables with high influx and outflux are preferred. Realize that outflux indicates the potential (and not actual) contribution to impute other variables. A variable with high \(O_{j}\) may turn out to be useless for imputation if it is unrelated to the incomplete variables. On the other hand, the usefulness of a highly predictive variable is severely limited by a low \(O_{j}\).

More refined measures of usefulness are conceivable, e.g., multiplying \(O_{j}\) by the average proportion of explained variance. Also, we could specialize to one or a few key variables to impute. Alternatively, analogous measures for \(I_{j}\) could be useful. The further development of diagnostic summaries for the missing data pattern is a promising area for further investigation.


# Examine Flux Patterns  -------------------------------------

round(fico(BP_miss), 1)[2:6]
#> BaselineDBP       Group         Age           Z          R1 
#>         0.5         0.5         0.5         0.5         0.5
ztable(flux_temp)
POBS Influx Outflux AINB AOUT FICO
PostDBP 0.51 0.45 0 1 0.00 0.00
BaselineDBP 1.00 0.00 1 0 0.08 0.49
Group 1.00 0.00 1 0 0.08 0.49
Age 1.00 0.00 1 0 0.08 0.49
Z 1.00 0.00 1 0 0.08 0.49
R1 1.00 0.00 1 0 0.08 0.49
R2 1.00 0.00 1 0 0.08 0.49

Exploratory Analyses


Next, We start off doing some preliminary analyses to eyeball differences between the observed data and the missing data.


#> [1] 245
pobs influx outflux ainb aout fico
PostDBP 0.51 0.45 0 1 0.00 0.00
BaselineDBP 1.00 0.00 1 0 0.08 0.49
Group 1.00 0.00 1 0 0.08 0.49
Age 1.00 0.00 1 0 0.08 0.49
Z 1.00 0.00 1 0 0.08 0.49
R1 1.00 0.00 1 0 0.08 0.49
R2 1.00 0.00 1 0 0.08 0.49

Having explored these data, we fit a preliminary model quickly to these missing data using the rlm() function from the MASS package and glance at what our estimates look like.


# Exploratory Model Estimates  -------------------------------------

eda_1 <- rlm(PostDBP ~ Group + BaselineDBP +
             Age + Z + R1 + R2, data = BP_miss, 
             method = "MM", psi = psi.huber)

ztable(summary(eda_1)[["coefficients"]])
Value Std. Error t value
(Intercept) 2.10 5.78 0.36
Group 9.58 0.13 71.57
BaselineDBP 0.98 0.05 19.86
Age 0.34 0.01 34.33
Z -0.93 1.48 -0.62
R1 -0.03 0.07 -0.43
R2 -0.13 0.13 -0.94
ztable(performance(eda_1))
AIC BIC RMSE Sigma
749.22 777.55 1.02 1.03


Because the MASS::rlm() function will automatically delete missing observations and do a complete-case analysis, we can gauge what our estimates would look like if we assume that the data are missing completely at random (MCAR), which is almost always unrealistic.

It is almost always more realistic to assume that the data are missing at random (MAR), \(\operatorname{Pr}\left(R=0 \mid Y_{\mathrm{obs}}, Y_{\mathrm{mis}}, \psi\right)=\operatorname{Pr}\left(R=0 \mid Y_{\mathrm{obs}}, \psi\right)\). This can often be a reasonable assumption to start off with for the primary analysis. (20)


The Analysis Pipeline


For our main analysis, we will build an imputation model to impute our missing data. We include all possible information in our imputation model, meaning all the variables in our dataset, and impute the missing data using the predictive mean matching method, and then analyze each dataset and pool the results. Our basic workflow will look like this:


Adapted flowchart from van Buuren (2018)


We do this with a fully-adjusted model, with all relevant covariates and a reduced model with just the treatment group. This is because we want both the adjusted and unadjusted estimates. We will then compare these models using the likelihood-ratio test (21) to see whether there are any substantial differences between the fully-adjusted and reduced model.


Monte Carlo Error Analysis


However, before we are able to move onto this step, we must determine how many imputations we will construct. To many, this may seem like a trivial and even nonsensical task when any individual can impute a large number of datasets. Indeed, it may seem intuitive that more is better. This is why we must conduct an MI power analysis.


I refer to this here as MI (multiple imputation) power analysis, however, it has little to do with statistical power and statistical significance, although the concepts of interest are related to power. Indeed, it may be more appropriate to refer to it as MI Information Analysis.


The issue with this is that without giving careful thought to how many imputations may be needed for a particular study, one may run the risk of imputing too few datasets and losing information, or they may waste resources imputing a nonsensical amount that may not be necessary. Indeed, the latter seems especially trivial, however, it may often yield no advantages whatsoever, and only incur costs as many authors have argued.


The argument is that “the additional resources that would be required to create and store more than a few imputations would not be well spent” (Schafer 1997, 107), and “in most situations there is simply little advantage to producing and analyzing more than a few imputed datasets” (Schafer and Olsen 1998, 549).


Therefore, we will examine certain characteristics of our dataset with missing values and use them in an analysis to determine how many datasets we should impute to efficiently achieve our goals. We will base this off a number of characteristics such as the fraction of missing information, the proportion of missing observations, and the losses that we are willing to incur.

We first start by running a “dry” or naive imputation of our data to quickly examine some of these characteristics. This imputed dataset will not be used for our primary analysis, it is simply serving to guide us in our main imputation model. This is also why we have set the number of iterations to 0.

Once again, we will include all the variables in our dataset, and use the predictive mean matching method with approximately 5 donors.


# Dry Imputation -------------------------------------

form <- list(PostDBP ~ Group + BaselineDBP + Age + Z + R1 + R2)
pred <- make.predictorMatrix(BP_miss)

parlmice(BP_miss, method = "pmm", m = 20, 
     maxit = 0, cores = cores, donors = 5,
     cluster.seed = 1031, formulas = form, ridge = 0, 
     predictorMatrix = pred, remove.collinear = TRUE, 
     remove.constant = TRUE, allow.na = TRUE,
     printFlag = FALSE, visitSequence = "monotone") -> init_imp

head(init_imp$loggedEvents, 10) 
#> NULL

Now that we have imputed our dataset, we may examine it for any issues before moving onto the next step of analyzing it.



Although it is only a dry run, there seems to be no issues with it. We now pool our estimates using Rubin’s rules.


init_res <- mice::pool(with(init_imp,
                      rlm(PostDBP ~ Group + 
                          BaselineDBP + Age + 
                          Z + R1 + R2, 
             method = "MM", psi = psi.huber)))

init_res_sum <- summary(init_res)

colnames(init_res_sum) <- c("Term", "Estimate", "SE",
                            "Statistic", "df", "P-val")
ztable(init_res_sum)
Term Estimate SE Statistic df P-val
(Intercept) 25.24 16.22 1.56 64.87 0.12
Group 8.25 0.39 21.08 67.22 0.00
BaselineDBP 0.81 0.14 5.90 66.98 0.00
Age 0.28 0.03 8.57 41.85 0.00
Z 0.69 4.11 0.17 89.16 0.87
R1 0.01 0.18 0.08 160.06 0.94
R2 -0.01 0.38 -0.02 70.53 0.98

Now that we have our vector, we can quickly examine certain characteristics such as the fraction of missing information (FIMO), the between-imputation variance and the within-imputation variance. The reason that we focus on these particular attributes is because they are crucial for several tasks in any statistical analysis plan, such as having enough power/precision, minimum long-run coverage, etc.

We use the FIMO from the dry imputation and a number of other criteria to determine how many imputations we need:


  • Some have suggested taking the FIMO and multiplying by 100 to obtain the number of imputed datasets needed

\[m \approx 100 \lambda\]


  • For our goals, we wish to ensure that the monte-carlo standard error from the imputations are less than 10% of the between-imputation standard error

\[mcse < B^{\hat{-}}_{SE} * 0.10\]


  • We wish to minimize the monte-carlo error that is derived from dividing the chosen number of datasets to impute from the fraction of missing information so that the monte-carlo standard error is less than 0.01

\[\frac{FMI}{m} ≈ 0.01\]


  • We also wish to minimize the total variance so that the square-root of it is no greater than the ideal variance and the corresponding confidence interval width.

\[T_{m}=\left(1+\frac{\gamma_{0}}{m}\right) T_{\infty}\]


  • Furthermore, it is commonly advocated by missing data experts to impute the same or similar number of datasets as fractions of missing observations \[\gamma_{0}*100\]

We start by extracting the FMIs from our dry imputation and calculating all these numbers.


# Calculate Fraction of Missing Information 

fmi <- max(summary(init_res, type = c("all"), 
                   conf.int = TRUE, conf.level = 0.95)["fmi"])  

m <- fmi / 0.01  # FMI Imputations

mcse <- sqrt(max(init_res$pooled$b) / m) # MCSE
se <- max((init_res$pooled$ubar) / sqrt(500)) # SE

What we find is that the fraction of missing information is nearly 50% which is very high and that in order to get a monte-carlo error of less than 0.01, we would need a minimum of at least 50 imputed datasets. We also calculate both the monte-carlo standard error and the standard error from the dry run and find that they are practically equivalent.

However, we must also explore other possibilities in terms of number of imputed datasets to reduce the monte-carlo errors so we run a quick function to do that, checking the effects of a number of imputed datasets on the width of the confidence interval, which is directly tied to the monte-carlo error.

Indeed, Von Hippel (2018) proposed a relationship between the fraction of missing information and the number of imputations needed to achieve a targeted CI length or monte-carlo error using a quadratic formula,


\[m=1+\frac{1}{2}\left(\frac{\gamma_{0}}{\operatorname{SD}\left(\sqrt{U_{\ell}}\right) \mathrm{E}\left(\sqrt{U_{\ell}}\right)}\right)^{2}\]


where \(\mathrm{E}\left(\sqrt{U_{\ell}}\right)\) and \(\mathrm{SD}\left(\sqrt{U_{\ell}}\right)\) are the coefficient of variation (CV), summarizing the imputation variation in the SE estimates. We can construct a function to depict this relationship and calculate how many imputations we will need based on some desired error and further display the distribution of this relationship by varying these parameters.

In the function below, adopted from von Hippel, and van Buuren, (20) I vary both the coefficient of variation and the \(\alpha\)-level.



With this function, we can now graphically examine the effects of different parameter values on the monte-carlo errors from the imputations.



We can see above that as we aim for smaller monte-carlo errors, we need many more imputations to achieve our desired target.

Now that I have constructed and graphed three differing relationships between monte-carlo errors and number of imputations needed to achieve those errors, and varied the \(\alpha\)-levels, we can now move onto choosing a specific number of imputations to construct.

If I choose an \(\alpha\) level of 5%, which the maximum tolerable type-I error rate, and a coefficient of variation of 5%


# Estimate Number of Imputations -------------------------------------

powerimp(model = init_res, 
         cv = 0.05, alpha = 0.05)
#> [1] 114

I need to construct approximately 90 imputations to achieve this coefficient of variation, which is what I will use for all the analyses from here on.


The Primary Analysis


For the primary analysis, I will use the predictive-mean matching method and specify 5 donors (based on a number of simulation studies that have examined the effect of various donors specified) (22), and fit a robust regression that includes all the variables within the dataset. Our imputation model will be


\[Y^{\operatorname{Post}} = \beta_{0} + \beta_{1}^{\operatorname{Group}} + \beta_{2}^{\operatorname{Base}} + \beta_{3}^{\operatorname{Age}} + \beta_{4}^{\operatorname{Z}} + \beta_{5}^{\operatorname{R1}} + \beta_{6}^{\operatorname{R2}} + \epsilon\]


So we will not impute our datasets.


# Impute Missing Data  -------------------------------------

form <- list(PostDBP ~ Group + BaselineDBP + Age + Z + R1 + R2)
pred <- make.predictorMatrix(BP_miss)

parlmice(BP_miss, method = "pmm", 
     predictorMatrix = pred,  
     m = 10, maxit = 5, cluster.seed = 1031, 
     n.core = cores, n.imp.core = (12 / cores), 
     formulas = form, ridge = 0, cl.type = "PSOCK",
     donors = 5, remove.collinear = TRUE,
     remove.constant = TRUE, allow.na = TRUE, 
     printFlag = FALSE, visitSequence = "monotone") -> imp1

head(imp1$loggedEvents, 10) 
#> NULL

Now that we have imputed our datasets, we may examine them for any issues before moving onto the next step of analyzing it.



Our imputed datasets seem to look fine so far. We now fit our models to each of these datasets and combine them using Rubin’s rules.


# Primary Analysis -------------------------------------

analysis1 <- with(imp1, rlm(PostDBP ~ Group +
                          BaselineDBP + Age + Z + R1 + R2, 
             method = "MM", psi = psi.huber))
analysis2 <- with(imp1, rlm(PostDBP ~ Group, 
             method = "MM", psi = psi.huber))
print(anova(analysis1, analysis2, method = "D3", use = "LR"))
#>    test statistic df1     df2 dfcom p.value      riv
#>  1 ~~ 2         0   5 55.3891   493       1 253.0947

result1 <- mice::pool(analysis1)

results1_sum <- summary(result1)

colnames(results1_sum) <- c("Term", "Estimate", "SE",
                            "Statistic", "df", "P-val")
ztable(results1_sum)
Term Estimate SE Statistic df P-val
(Intercept) 3.70 5.06 0.73 63.83 0.47
Group 9.52 0.14 68.18 35.90 0.00
BaselineDBP 0.97 0.04 22.33 62.02 0.00
Age 0.33 0.01 36.92 64.42 0.00
Z -0.21 1.25 -0.17 129.69 0.87
R1 -0.02 0.08 -0.22 25.19 0.82
R2 -0.16 0.14 -1.13 31.13 0.27

While we have our primary analysis estimates, we also continue to examine our imputations for any anomalies.



We’ve done our primary analysis, in which we imputed the missing data under the assumption of MAR, which is often a reasonable assumption to start with. Unfortunately, we cannot verify whether or not this assumption is true, but we can vary this missing data assumption and assume that the data are missing not at random (MNAR) \(\operatorname{Pr}\left(R=0 \mid Y_{\mathrm{obs}}, Y_{\mathrm{mis}}, \psi\right)\), and that in the individuals with missing data, we can assume different numbers than those with complete data. Our likelihood-ratio test also suggests that there’s very little difference between the full model and the reduced model, so we generally will go with the full model.



SA I: Controlled MI


Now suppose we wish to handle these missing data under the assumption of MNAR, we would now be conducting a sensitivity analysis because we are still targeting the same estimand, but only varying the assumptions. The question / target remain the same. To handle missing data that we assume are MNAR, there are a number of different and complex approaches.

We start off with the \(\delta\)-adjustment, controlled multiple imputation method, (23) in which we assume that the group with missing observations differs systematically from the group with complete observations by a certain quantity. We produce a range of these quantities and add or subtract them from the imputed values and then conduct our analysis on this adjusted, imputed dataset. Here I will make the assumption that the average PostDBP estimate from the missing value group differs from at least 0 mmHg to at most 30 mmHg from the group with no missing observations.


# Controlled MI  -------------------------------------

ini <- mice(BP_miss, method = "pmm", maxit = 0,
            predictorMatrix = pred)

ini$method["BaselineDBP"] <- ""
ini$method["PostDBP"] <- "pmm"
ini$method["Age"] <- ""
ini$method["R1"] <- ""
ini$method["Group"] <- ""
ini$method["R2"] <- ""
ini$method["Z"] <- ""
meth <- ini$method

delta <- seq(-30, 30, 5)

imp.all <- vector("list", length(delta))
post <- ini$post
for (i in 1:length(delta)) {
  d <- delta[i]
  cmd <- paste("imp[[j]][, i] <- imp[[j]][, i] +", d)
  post["PostDBP"] <- cmd
  imp <- parlmice(BP_miss, method = meth,
              predictorMatrix = pred,
              m = 5, maxit = 5, donors = 5,
              n.core = cores, ridge = 0, formulas = form,
              remove.collinear = TRUE, 
              remove.constant = TRUE, cl.type = "PSOCK",
              allow.na = TRUE, post = post,
              cluster.seed = i * 1031, printFlag = FALSE)
  imp.all[[i]] <- imp
}

Now that we have our imputed datasets that are modified by \(\delta\)-adjustments, we can fit our models to each imputed dataset and combine the estimates using Rubin’s rules.


# Sensitivity Analysis 1  -------------------------------------

output <- sapply(imp.all,
                 function(x) mice::pool(with(x, 
                 rlm(PostDBP ~ Group +
                     BaselineDBP + Age + Z + R1 + R2, 
                     method = "MM", psi = psi.huber))),
                 simplify = "array", USE.NAMES = TRUE)

a <- (as.data.frame(t(output)))$pooled
r <- array(rep(NA, length(delta) * 7),
           dim = c(length(delta), 7), 
           dimnames = list(c(delta),
                           c("Intercept", "Group",
                             "BaselineDBP", "Age",
                              "Z", "R1", "R2")))
for (i in 1:length(delta)) {
   r[i, ] <- cbind(a[[i]][["estimate"]])
}

r <- as.data.frame(r)
r <- data.frame(Delta = row.names(r), r, row.names = NULL)
ztable(r)
Delta Intercept Group BaselineDBP Age Z R1 R2
-30 -31.08 39.03 1.01 0.33 -1.24 -0.05 -0.06
-25 -249.07 17.07 3.02 0.33 -9.40 -0.16 -2.39
-20 -117.53 11.72 1.93 0.30 -3.52 0.10 -1.09
-15 -64.45 10.73 1.49 0.31 -2.36 0.05 -0.61
-10 -35.41 10.19 1.26 0.32 -1.76 0.03 -0.40
-5 -15.32 9.84 1.11 0.32 -1.61 0.01 -0.30
0 1.13 9.54 0.99 0.33 -1.30 -0.03 -0.14
5 22.28 9.20 0.83 0.33 -0.65 -0.04 -0.05
10 42.46 8.83 0.68 0.33 -0.49 -0.06 0.07
15 71.21 8.29 0.45 0.34 0.32 -0.10 0.24
20 126.62 7.26 -0.01 0.35 1.51 -0.16 0.71
25 242.77 2.27 -0.98 0.34 1.86 -0.39 2.65
30 56.79 -18.37 0.75 0.34 -1.07 -0.10 0.32

The table above gives the various \(\delta\)’s (the first column with values ranging from -30 - 30 in increments of 5) that were added to the multiply imputed datasets, and the corresponding estimates from the models fit to those datasets. So now we’ve conducted our first sensitivity analysis using the \(\delta\)-adjustment, controlled multiple imputation method. This form of sensitivity analysis is also typically what is used and preferred by regulators. (23)


Regulators prefer simple methods that impute the missing outcomes under MAR, and then add an adjustment δ to the imputes, while varying δ over a plausible range and independently for each treatment group (Permutt 2016).

The most interesting scenarios will be those where the difference between the δ’s correspond to the size of the treatment effect in the completers. Contours of the p-values may be plotted on a graph as a function of the δ’s to assist in a tipping-point analysis (Liublinska and Rubin 2014).


And as expected, lower \(\delta\) values gave us results that were consistent with the primary analysis, while the upper end of the extreme \(\delta\) values gave us estimates that were substantially off the mark.


Next, we quickly examine the state of a random sample of these imputations to make sure they’re reliable.



Now we move onto our next and final sensitivity analysis.



SA II: Selection Models


To further explore the MNAR assumption, we could also use a Heckman selection model \(P(Y,R)=P(Y)P(R|Y)\), as described by van Buuren here (20)


The selection model multiplies the marginal distribution \(P(Y)\) in the population with the response weights \(P(R|Y)\). Both \(P(Y)\) and \(P(R|Y)\) are unknown, and must be specified by the user. The model where \(P(Y)\) is normal and where \(P(R|Y)\) is a probit model is known as the Heckman model. This model is widely used in economics to correct for selection bias.


We can implement this in R using the miceMNAR package, which is an extension to the mice R package.


# Selection Models  -------------------------------------

JointModelEq <- generate_JointModelEq(data = BP_miss,
                                      varMNAR = "PostDBP")

JointModelEq[,"PostDBP_var_sel"] <- c(0, 1, 1, 1, 1, 1, 1)
JointModelEq[,"PostDBP_var_out"] <- c(0, 1, 1, 1, 0, 1, 1)

arg <- MNARargument(data = BP_miss, varMNAR = "PostDBP",
                    JointModelEq = JointModelEq)

imp2 <- mice(data = arg$data_mod,
             method = "ri", seed = 1031,
             predictorMatrix = arg$predictorMatrix,
             JointModelEq = arg$JointModelEq,
             control = arg$control, 
 remove.collinear = TRUE, 
             remove.constant = FALSE, allow.na = FALSE,
             printFlag = FALSE, visitSequence =  "monotone")

head(imp2$loggedEvents, 10)
#> NULL

imp2$data$Group <- as.factor(imp2$data$Group)


Now we fit our models to these new datasets and pool them using Rubin’s rules.


# Sensitivity Analysis 2  -------------------------------------

analysis3 <- with(imp2, rlm(PostDBP ~ Group +
                            BaselineDBP + Age + Z + R1 + R2, 
             method = "MM", psi = psi.huber))
result3 <- mice::pool(analysis3)

results3_sum <- summary(result3)
colnames(results3_sum) <- c("Term", "Estimate", "SE",
                            "Statistic", "df", "P-val")

ztable(results3_sum)
Term Estimate SE Statistic df P-val
(Intercept) 1.53 7.18 0.21 8.52 0.84
Group1 9.64 0.15 64.11 12.50 0.00
BaselineDBP 0.98 0.06 16.83 9.43 0.00
Age 0.34 0.01 30.28 11.72 0.00
Z -1.77 1.62 -1.10 14.20 0.29
R1 -0.05 0.07 -0.67 20.42 0.51
R2 -0.13 0.12 -1.13 45.57 0.27

Now we examine our imputed values.



so now we have completed our second sensitivity analysis.



The results from both the primary analysis, the first sensitivity analysis, and the second seem somewhat consistent. However, with the first sensitivity analysis where we applied \(\delta\)-adjustments, as the adjustments became larger, the coefficients changed drastically. However, such large \(\delta\)-adjustments are not realistic and those that were smaller led to coefficients that were closer to the other two analyses.


Supplementary Analyses


The following analyses are what I mostly consider to be supplementary analyses that may further investigate some violations of assumptions but often go far beyond what would be necessary for a principled sensitivity analysis that would typically accompany a primary analysis.


First, I like to check that the results from my analyses are consistent across different statistical software So I may check to see that the results from my primary analysis from R are also similar to the results from another statistical software suite like Stata.

So in Stata, I would likely run something similar to the following in order to mimic the primary analysis in R.


import delimited "C:\datasets\temp.csv", numericcols(2 3 5 6 7 8)

summarize

mi set mlong

mi register imputed baselinedbp postdbp age z group r1 r2

mi impute chained (pmm, knn(5)) postdbp = baselinedbp age z r1 r2 group, burnin(20) add(100) rseed (1031) savetrace(trace1, replace) nomonotone 

mi estimate, mcerror: rreg postdbp i.group baselinedbp age z r1 r2

quietly mi passive: generate byte imputed = _mi_miss

replace imputed = 1 if imputed==.

generate float postdbp_Delta_A1=postdbp

replace postdbp_Delta_A1=postdbp + 5 if imputed==1

mi estimate, mcerror: rreg postdbp_Delta_A1 i.group baselinedbp age z r1 r2
#> . import delimited "C:\datasets\temp.csv", numericcols(2 3 5 6 7(encoding automatically selected: ISO-8859-2)
#> (8 vars, 500 obs)
#> 
#> . 
#> . summarize
#> 
#>     Variable |        Obs        Mean    Std. dev.       Min        Max
#> -------------+---------------------------------------------------------
#>           v1 |        500       250.5    144.4818          1        500
#>      postdbp |        209    135.0179    4.550848   125.9651     149.47
#>  baselinedbp |        500    116.9717    1.961848    110.615   123.7551
#>        group |        500        .508    .5004367          0          1
#>          age |        500     49.4683     7.08616   26.96167   70.72639
#> -------------+---------------------------------------------------------
#>            z |        500   -9.29e-11    .0626851   -.209414   .2471141
#>           r1 |        500   -.0715254    .9988079  -3.568279   3.036855
#>           r2 |        500    .9799438    .4913294  -.6787649   2.356054
#> 
#> . 
#> . mi set mlong
#> 
#> . 
#> . mi register imputed baselinedbp postdbp age z group r1 r2
#> (291 m=0 obs now marked as incomplete)
#> 
#> . 
#> . mi impute chained (pmm, knn(5)) postdbp = baselinedbp age z r1 r2 group, burn
#> > in(20) add(100) rseed (1031) savetrace(trace1, replace) nomonotone 
#> note: missing-value pattern is monotone.
#> 
#> Conditional models:
#>            postdbp: pmm postdbp baselinedbp age z r1 r2 group , knn(5)
#> 
#> Performing chained iterations ...
#> 
#> Multivariate imputation                     Imputations =      100
#> Chained equations                                 added =      100
#> Imputed: m=1 through m=100                      updated =        0
#> 
#> Initialization: monotone                     Iterations =     2000
#>                                                 burn-in =       20
#> 
#>            postdbp: predictive mean matching
#> 
#> ------------------------------------------------------------------
#>                    |               Observations per m             
#>                    |----------------------------------------------
#>           Variable |   Complete   Incomplete   Imputed |     Total
#> -------------------+-----------------------------------+----------
#>            postdbp |        209          291       291 |       500
#> ------------------------------------------------------------------
#> (Complete + Incomplete = Total; Imputed is the minimum across m
#>  of the number of filled-in observations.)
#> 
#> . 
#> . mi estimate, mcerror: rreg postdbp i.group baselinedbp age z r1 r2
#> 
#> Multiple-imputation estimates                   Imputations       =        100
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     1.6877
#>                                                 Largest FMI       =     0.7159
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =      82.46
#>                                                         avg       =     127.69
#>                                                         max       =     191.63
#> Model F test:       Equal FMI                   F(   6,  374.8)   =    1171.55
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#>      postdbp | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |      10.46   .1504111    69.54   0.000     10.16183    10.75818
#>              |   .0118239   .0053087     2.46   0.000     .0144955    .0173849
#>              |
#>  baselinedbp |   .9542017   .0509793    18.72   0.000     .8532671    1.055136
#>              |   .0038745    .002412     0.91   0.000     .0063144    .0061611
#>              |
#>          age |   .3229139   .0085025    37.98   0.000     .3061434    .3396845
#>              |   .0005449    .000226     1.02   0.000     .0007102    .0007091
#>              |
#>            z |  -2.622068   1.511717    -1.73   0.085    -5.611361    .3672254
#>              |   .1101344    .054734     0.09   0.016     .1620533    .1500265
#>              |
#>           r1 |   -.360883   .0856236    -4.21   0.000    -.5312012   -.1905648
#>              |   .0071805   .0051837     0.29   0.000      .011663    .0138291
#>              |
#>           r2 |  -.1834333   .1392627    -1.32   0.190    -.4588711    .0920045
#>              |   .0102243   .0053406     0.09   0.030     .0145569    .0151887
#>              |
#>        _cons |   6.511102    5.96798     1.09   0.277     -5.30351    18.32571
#>              |   .4518723   .2785653     0.09   0.040     .7162938    .7298489
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.
#> 
#> . 
#> . quietly mi passive: generate byte imputed = _mi_miss
#> 
#> . 
#> . replace imputed = 1 if imputed==.
#> (0 real changes made)
#> 
#> . 
#> . generate float postdbp_Delta_A1=postdbp
#> (291 missing values generated)
#> 
#> . 
#> . replace postdbp_Delta_A1=postdbp + 5 if imputed==1
#> (29,100 real changes made)
#> 
#> . 
#> . mi estimate, mcerror: rreg postdbp_Delta_A1 i.group baselinedbp age z r1 r2
#> 
#> Multiple-imputation estimates                   Imputations       =        100
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     0.4841
#>                                                 Largest FMI       =     0.4147
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =     193.60
#>                                                         avg       =     279.66
#>                                                         max       =     333.66
#> Model F test:       Equal FMI                   F(   6,  458.8)   =     915.91
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#> postdbp_De~1 | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |   13.66522    .206023    66.33   0.000      13.2596    14.07084
#>              |   .0110344   .0036253     1.17   0.000     .0125281    .0138535
#>              |
#>  baselinedbp |   1.001625   .0727854    13.76   0.000      .858348    1.144902
#>              |   .0038007   .0015713     0.31   0.000     .0051118     .004746
#>              |
#>          age |   .3123472   .0137482    22.72   0.000     .2853031    .3393913
#>              |   .0006216   .0001824     0.30   0.000     .0006903    .0007489
#>              |
#>            z |  -2.568087   2.228407    -1.15   0.250    -6.953131    1.816957
#>              |   .1093152   .0351552     0.05   0.021     .1344406    .1252701
#>              |
#>           r1 |  -1.453223   .1130773   -12.85   0.000    -1.676245   -1.230202
#>              |   .0072142   .0038154     0.45   0.000     .0096133    .0113823
#>              |
#>           r2 |  -.6723698   .2049326    -3.28   0.001    -1.075681   -.2690583
#>              |   .0102784   .0037834     0.08   0.000     .0131094    .0124015
#>              |
#>        _cons |   3.106196   8.548041     0.36   0.717    -13.71994    19.93233
#>              |   .4439893   .1809842     0.05   0.039     .5565776    .5887493
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.

Trace plot of imputed datasets


As you can see from above, I imported the missing data I generated in R into Stata, and then used it to multiply impute the datasets using the predictive mean matching method with 5 donors. I chose the exact same number of variables to include in the imputation model as I did with the primary and sensitivity analyses. Just like the primary analysis, I also used robust regression, rreg in Stata, to analyze the datasets. The numbers seem to be fairly consistent with those from the primary analysis and the some of the results from the sensitivity analyses.

I will now conduct another supplementary analysis, in which I fit a Bayesian regression model using a t-distribution so that it is analogous to the other robust regression models I have utilized so far, and I will specify a weakly informative prior. I will fit one model using the brm_multiple() function which will fit the model to each of the imputed datasets from before, while I fit another Bayesian model which will impute values during the model fitting process and compare the results.


# Bayesian Regression  -------------------------------------

brm_form1 <- bf(PostDBP ~ Group + BaselineDBP + 
               Age + Z + R1 + R2)

brm_form2 <- bf(PostDBP | mi() ~ Group + BaselineDBP + 
               Age + Z + R1 + R2)

prior1 <- c(prior(normal(0, 100), class = Intercept),
            prior(normal(0, 100), class = b),
            prior(normal(0, 100), class = sigma))

brm_multiple(brm_form1, data = imp1, prior = prior1, 
             silent = 2, family = student(link = "identity"), 
             iter = 2000, refresh = 0,
             warmup = 100, chains = 1, inits = "0",
             thin = 2, open_progress = FALSE, cores = cores,
             combine = TRUE, sample_prior = "no", 
             seed = 1031, algorithm = "sampling", backend = "rstan",
             control = list(max_treedepth = 10, adapt_delta = 0.8),
             file = NULL, file_refit = "never") -> b_1

b_2 <- brm(brm_form2, data = BP_miss, prior = prior1, cores = cores,
           silent = 2, family = student(link = "identity"), 
           iter = 2000, refresh = 0, backend = "rstan",
           warmup = 100, chains = 2, inits = "0",
           thin = 2, open_progress = FALSE,
           seed = 1031, algorithm = "sampling")

preds <- b_1[["fit"]]@sim[["fnames_oi"]][2:7] 
preds <- b_2[["fit"]]@sim[["fnames_oi"]][2:7]  

The accompanying Stan (extracted via the brms stancode() function)code for our first model is the following,


/* Stan code for analyzing multiply imputed datasets */
functions {
  /* compute the logm1 link 
   * Args: 
   *   p: a positive scalar
   * Returns: 
   *   a scalar in (-Inf, Inf)
   */ 
   real logm1(real y) { 
     return log(y - 1);
   }
  /* compute the inverse of the logm1 link 
   * Args: 
   *   y: a scalar in (-Inf, Inf)
   * Returns: 
   *   a positive scalar
   */ 
   real expp1(real y) { 
     return exp(y) + 1;
   }
}
data {
  int<lower=1> N;  // total number of observations
  vector[N] Y;  // response variable
  int<lower=1> K;  // number of population-level effects
  matrix[N, K] X;  // population-level design matrix
  int prior_only;  // should the likelihood be ignored?
}
transformed data {
  int Kc = K - 1;
  matrix[N, Kc] Xc;  // centered version of X without an intercept
  vector[Kc] means_X;  // column means of X before centering
  for (i in 2:K) {
    means_X[i - 1] = mean(X[, i]);
    Xc[, i - 1] = X[, i] - means_X[i - 1];
  }
}
parameters {
  vector[Kc] b;  // population-level effects
  real Intercept;  // temporary intercept for centered predictors
  real<lower=0> sigma;  // residual SD
  real<lower=1> nu;  // degrees of freedom or shape
}
transformed parameters {
}
model {
  // likelihood including constants
  if (!prior_only) {
    // initialize linear predictor term
    vector[N] mu = Intercept + Xc * b;
    target += student_t_lpdf(Y | nu, mu, sigma);
  }
  // priors including constants
  target += normal_lpdf(b | 0, 100);
  target += normal_lpdf(Intercept | 0, 100);
  target += normal_lpdf(sigma | 0, 100)
    - 1 * normal_lccdf(0 | 0, 100);
  target += gamma_lpdf(nu | 2, 0.1)
    - 1 * gamma_lccdf(1 | 2, 0.1);
}
generated quantities {
  // actual population-level intercept
  real b_Intercept = Intercept - dot_product(means_X, b);
}



Next, we examine the results from the Bayesian model in which values were imputed during model fitting.


ztable(posterior_summary(b_2))[1:8]
#> [1] "<div style=\"border: 1px solid #ddd; padding: 5px; overflow-x: scroll; width:100%; \"><table class=\"striped lightable-classic\" style=\"font-size: 16px; font-family: Cambria; margin-left: auto; margin-right: auto;\">\n <thead>\n  <tr>\n   <th style=\"text-align:left;color: #777 !important;\">   </th>\n   <th style=\"text-align:center;color: #777 !important;\"> Estimate </th>\n   <th style=\"text-align:center;color: #777 !important;\"> Est.Error </th>\n   <th style=\"text-align:center;color: #777 !important;\"> Q2.5 </th>\n   <th style=\"text-align:center;color: #777 !important;\"> Q97.5 </th>\n  </tr>\n </thead>\n<tbody>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> b_Intercept </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -12.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 12.73 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> b_Group </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.52 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.84 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> b_BaselineDBP </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.99 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.09 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> b_Age </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.01 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.33 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.37 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> b_Z </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -1.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -4.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.67 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> b_R1 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -0.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -0.25 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.06 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> b_R2 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -0.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.13 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -0.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.18 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> sigma </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.79 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.02 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> nu </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.01 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 0.02 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.07 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[6] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.13 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[9] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.47 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.47 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[10] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.79 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.21 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.06 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 159.38 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[11] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.18 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.45 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[12] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.42 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.68 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 151.16 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[13] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 11.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 80.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.89 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[14] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.75 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 122.76 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.97 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[15] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.77 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 102.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.21 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[16] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 33.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -1.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.26 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[17] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.54 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.79 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.07 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[18] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 36.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -19.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.89 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[22] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 16.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 50.94 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.54 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[23] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 56.30 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 75.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -48.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.11 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[26] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 100.02 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 44.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 33.99 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.96 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[29] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.46 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 117.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.99 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[33] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.68 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.47 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[38] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.91 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[39] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 20.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 107.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -145.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.05 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[43] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 59.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 68.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -46.47 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.16 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[44] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 12.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 101.65 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.53 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[46] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.90 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.29 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[47] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 92.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 51.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 1.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 160.81 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[50] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.07 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 163.04 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[51] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.68 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.24 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 151.79 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[52] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 157.72 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 32.18 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 96.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 212.24 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[53] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 117.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 37.49 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 15.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.02 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[54] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 96.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 54.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -11.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.99 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[55] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.81 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[57] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.15 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.86 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[58] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 156.46 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 18.42 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 189.73 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[61] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 121.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.48 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[63] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 52.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 96.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -131.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.67 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[65] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.48 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 25.90 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 35.55 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.68 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[68] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.36 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 20.65 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 205.99 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[70] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.90 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.54 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 155.58 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[71] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.93 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.64 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.96 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[73] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 19.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 209.70 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[74] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.75 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.18 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[75] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.75 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.40 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[79] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.52 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 115.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.48 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[81] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.48 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.36 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.21 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[84] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.53 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.80 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[86] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.30 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.71 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[87] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 41.18 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 73.28 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -73.53 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.33 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[88] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.40 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[89] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 113.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 48.25 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -9.23 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.69 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[91] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.06 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.25 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.69 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[92] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.64 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.46 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[94] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.90 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.07 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[96] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.47 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[98] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.79 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 23.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 78.90 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 208.72 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[99] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.36 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 164.58 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[101] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.98 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 14.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.46 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 186.91 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[103] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.49 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.98 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.58 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[104] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.69 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[105] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.76 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.36 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 152.34 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[106] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.64 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 10.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 119.79 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.20 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[107] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.94 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[108] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.52 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 175.74 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[110] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.46 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.12 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.53 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.28 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[114] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.48 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 122.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.05 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[115] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 99.67 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 42.51 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 18.73 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.95 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[120] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.67 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.69 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -155.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.70 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[123] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.31 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[124] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.42 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.28 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 120.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 161.95 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[125] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.61 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.27 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.19 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[128] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.89 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[130] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 167.18 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[131] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.74 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[132] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 15.47 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 80.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.60 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[134] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 122.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.78 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[135] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 24.48 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 57.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.97 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[136] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 50.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 91.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -97.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.85 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[139] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 105.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.52 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[142] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.72 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.12 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 151.41 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[143] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.54 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.68 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 169.61 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[144] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.76 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.75 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.01 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.83 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[145] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -113.25 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 85.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -243.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -6.63 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[148] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 112.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 40.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 19.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.19 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[149] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.94 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.70 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[150] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -2.64 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -177.94 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.47 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[154] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -60.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 220.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -328.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 220.04 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[155] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 12.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 118.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.13 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[156] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 84.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 58.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -16.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.35 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[159] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.59 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[160] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 97.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 54.94 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -48.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.86 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[165] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 151.47 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[166] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.49 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 122.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.85 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[168] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 110.18 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 46.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -35.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.78 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[169] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -28.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 159.55 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -208.47 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.89 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[173] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.21 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.88 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[176] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 70.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 76.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -34.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 181.59 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[177] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 83.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 56.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -61.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.82 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[182] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.51 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.54 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 153.16 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[183] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.07 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.57 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[186] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.71 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[187] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 64.79 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 82.61 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -113.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.23 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[188] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.36 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[191] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 69.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 71.98 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -39.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.55 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[192] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 63.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 49.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -79.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.49 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[194] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 8.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 117.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.48 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[195] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 51.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 94.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -116.53 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.09 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[203] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 106.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 47.02 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -14.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.26 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[206] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 117.54 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.71 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[212] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 158.64 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 31.15 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 263.66 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[215] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.53 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.27 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.29 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[222] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.01 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 33.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -11.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.75 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[224] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 26.53 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 23.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.00 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[225] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -59.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 194.51 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -281.65 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.77 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[226] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -10.27 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -179.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.25 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[230] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.09 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[231] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 10.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 100.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 151.84 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[233] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -6.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -180.67 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.08 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[235] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 120.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.09 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[239] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -15.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 162.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -263.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.95 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[240] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.13 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -214.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.75 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[242] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 7.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.36 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.34 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[248] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.47 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.98 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 155.63 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[256] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.24 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.21 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[260] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.04 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[261] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.36 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[262] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 16.61 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 111.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -112.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.15 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[264] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.97 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[265] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 26.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 116.42 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -125.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.46 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[267] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 53.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 65.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -63.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.55 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[270] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 109.67 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 42.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -18.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.25 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[271] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 39.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -26.48 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.94 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[272] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 50.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 90.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -85.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.44 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[274] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.43 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.02 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[276] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 64.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 76.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -50.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.17 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[277] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.43 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[278] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 58.24 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 83.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -88.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.70 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[279] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 43.94 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 90.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -88.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.97 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[281] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 15.51 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 103.03 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 173.53 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[282] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 12.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 121.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.58 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[283] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -14.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 157.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -191.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.29 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[285] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 42.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -18.49 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.17 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[286] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 96.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 60.55 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -75.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.81 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[287] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 113.73 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 42.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 15.69 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 151.51 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[288] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.35 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[292] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 53.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 83.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -52.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.70 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[294] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.26 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[297] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.36 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[298] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.21 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.92 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 117.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.81 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[299] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -14.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -191.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.16 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[303] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.51 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 11.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.02 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.60 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[304] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.30 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.96 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[305] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 108.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 67.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -135.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.80 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[307] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -14.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.13 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -196.23 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.52 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[308] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.23 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 151.55 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[309] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 15.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 75.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.67 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[312] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.72 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[314] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 13.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.05 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[315] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.52 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[317] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 7.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 165.88 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[318] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.12 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 165.92 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[320] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.01 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 118.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.61 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[323] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 86.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 58.72 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -18.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.58 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[326] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.68 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.85 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[327] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.42 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.22 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[329] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 28.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 26.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 160.60 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[331] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.51 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 122.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.53 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[335] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.83 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[337] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.65 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.13 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.89 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[339] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 22.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 76.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.26 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[341] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 27.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 33.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 157.93 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[345] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.13 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.24 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[346] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.42 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.25 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[348] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.77 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.41 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[350] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.63 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[351] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.06 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 16.25 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 77.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.99 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[352] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 12.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 111.57 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.46 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[353] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 20.76 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 77.06 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 155.76 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[354] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 95.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 71.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -91.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.29 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[360] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.64 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[361] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -5.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -162.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.77 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[363] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 131.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.28 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 126.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.52 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[364] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.02 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.67 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[365] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.61 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 11.43 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 113.76 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.45 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[368] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 115.97 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 28.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 33.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.82 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[370] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -9.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.30 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -166.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.75 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[371] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 121.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 17.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 47.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.02 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[378] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.38 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.93 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[380] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.20 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.51 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[382] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 121.16 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.56 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[383] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -3.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.21 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -215.67 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.89 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[385] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.01 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 21.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 68.28 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.57 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[391] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 109.54 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 44.99 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 24.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.48 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[392] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.50 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[395] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 63.95 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 82.26 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -91.75 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.52 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[396] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 14.12 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 97.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.16 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[397] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.65 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.89 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[398] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 103.19 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 38.67 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 43.27 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.22 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[401] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 107.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 35.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 55.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.69 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[402] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.35 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.49 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.69 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 159.58 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[403] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.70 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.53 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.96 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[406] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.77 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 125.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.06 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[407] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.30 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 17.07 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 121.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.11 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[408] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.74 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.17 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[411] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 122.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 22.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 63.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.43 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[412] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 57.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 87.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -58.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 149.30 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[413] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.39 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.52 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 157.63 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[414] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.43 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 147.84 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[416] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 36.89 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 97.33 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -99.62 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.59 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[418] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.60 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 13.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 81.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.27 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[420] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.15 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 154.39 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[427] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 89.98 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 64.37 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -97.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.43 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[428] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.33 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.09 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.42 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.06 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[429] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.58 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 22.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 64.27 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.62 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[430] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 86.66 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 46.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 22.15 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 135.94 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[434] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 36.48 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 106.34 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -143.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.18 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[436] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.55 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 129.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.52 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[440] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -72.86 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 211.61 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -310.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.12 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[441] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.03 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.33 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.14 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[443] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -76.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 32.96 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -122.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.21 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[444] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.94 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 159.10 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[446] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.46 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.18 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.50 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[447] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 19.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 62.52 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.14 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[451] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 31.10 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 12.48 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 153.46 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[453] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 64.58 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 79.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -34.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 195.37 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[454] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 6.51 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 128.03 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.82 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[457] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 139.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 2.76 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 132.83 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.15 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[460] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 45.81 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 82.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -63.00 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.50 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[461] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 10.88 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 177.18 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[462] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 56.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 74.14 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -50.90 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.47 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[466] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 21.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 53.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 152.74 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[468] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.24 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 39.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -6.71 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.86 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[469] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 51.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 92.40 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -97.93 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.16 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[472] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.52 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.69 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -158.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 143.87 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[473] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.13 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 115.56 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 142.57 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[480] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 130.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 5.85 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.79 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.25 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[481] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 112.33 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 27.30 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 38.64 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 133.04 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[482] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 108.82 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 45.77 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -18.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.92 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[483] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 52.28 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 90.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -92.36 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 145.10 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[485] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 59.78 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 76.59 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -87.07 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.72 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[486] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 76.68 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 59.50 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -46.04 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 137.97 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[488] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 96.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 73.24 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -60.27 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 146.32 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[489] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.31 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 9.87 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 127.08 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.64 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[491] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 123.84 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 28.32 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 51.76 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.96 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[494] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 39.46 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 92.45 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -71.52 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.67 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[496] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 144.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.67 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.22 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 150.81 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[497] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -4.63 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.64 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -185.11 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 136.57 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[498] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 134.33 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 3.29 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 124.80 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 140.39 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[499] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 148.44 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 4.17 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 138.41 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 155.41 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> Ymi[500] </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 156.23 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 21.58 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 141.72 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 240.20 </td>\n  </tr>\n  <tr>\n   <td style=\"text-align:left;color: #777 !important;\"> lp__ </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -1390.91 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> 161.27 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -1700.05 </td>\n   <td style=\"text-align:center;color: #777 !important;\"> -1149.83 </td>\n  </tr>\n</tbody>\n</table></div>"
#> [2] NA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
#> [3] NA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
#> [4] NA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
#> [5] NA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
#> [6] NA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
#> [7] NA                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
#> [8] NA
post <- posterior_samples(b_2)
#> Warning: Method 'posterior_samples' is deprecated. Please see ?as_draws for recommended alternatives.

They seem to be similar to the results we’ve been seeing so far.


Indeed, it is no surprise that the Bayesian model with a weakly dispersed prior would give similar results to the base rlm() function. We now look at robust methods and resampling and the results they give.


Although this is currently under the supplementary analysis section, I would very much consider these next two analyses to be sensitivity analyses. The first method utilized here is known as the random indicator method, and it is an experimental method that essentially combines a selection model and a pattern-mixture model and is designed to handle data that are missing not at random. As van Buuren writes (20)


The random indicator method (Jolani 2012) (24) is an experimental iterative method that redraws the missing data indicator under a selection model, and imputes the missing data under a pattern-mixture model, with the objective of estimating δ from the data under relaxed assumptions. Initial simulation results look promising. The algorithm is available as the ri method in mice.


The script below utilizes this method (random indicator), but bootstraps the missing dataset before the imputation procedure, and then after multiply imputing the datasets, obtains estimates using maximum likelihood multiple (MLMI) imputation, an alternative and more efficient method to the more popular posterior draw multiple imputation (PDMI). More about MLMI can be found here. (25)


# Bootstrap Inference -------------------------------------

bootMice(BP_miss, method = "ri", maxit = 5, 
         predictorMatrix = pred, nBoot = 216, nImp = 2,
         seed = 1031, formulas = form, nCores = cores,
         ridge = 0, remove.collinear = TRUE,
         remove.constant = TRUE, allow.na = TRUE,
         printFlag = FALSE, visitSequence = "monotone") -> imps

analyseImp <- function(inputData) {
  return(coef(rlm(PostDBP ~ Group + BaselineDBP + Age + Z + R1 + R2,
                    data = inputData, 
             method = "MM", psi = psi.huber)))
}

ests <- bootImputeAnalyse(imps, analyseImp, quiet = TRUE)

ests <- as.data.frame(ests)
colnames(ests) <- c("Estimate", "Var",
                            "CI.ll", "CI.ul", "df")
rownames(ests) <- c("Intercept", "Group", "BaselineDBP",
                    "Age", "Z", "R1", "R2")
ztable(ests)
Estimate Var CI.ll CI.ul df
Intercept 1.89 25.90 -8.21 11.99 97.27
Group 9.56 0.02 9.30 9.82 122.76
BaselineDBP 0.98 0.00 0.89 1.07 101.14
Age 0.34 0.00 0.32 0.36 124.30
Z -0.92 2.15 -3.82 1.98 127.26
R1 -0.02 0.01 -0.18 0.15 132.46
R2 -0.13 0.02 -0.39 0.14 116.33

For our second supplementary analysis, We will utilize the GAMLSS method, which is an extension of the generalized linear model and the generalized additive model. For the vast majority of the imputations we have conducted, we have drawn values from an assumed normal distribution. Thus, GAMLSS allows us to be far more flexible with our assumptions. However, it is also worth noting that general normal imputations are robust against violations of the assumption of normality


In general, normal imputations appear to be robust against violations of normality. Demirtas, Freels, and Yucel (2008) found that flatness of the density, heavy tails, non-zero peakedness, skewness and multimodality do not appear to hamper the good performance of multiple imputation for the mean structure in samples n > 400, even for high percentages (75%) of missing data in one variable. The variance parameter is more critical though, and could be off-target in smaller samples.


Regardless, we will be using the GAMLSS method and also imputing from a t-distribution, which is much more robust to outliers.


gamt <- mice(BP_miss, method = "gamlssTF",
             predictorMatrix = pred, 
             m = 5, maxit = 1, seed = 1031,
             cores = cores, formulas = form, ridge = 0,
             remove.collinear = TRUE, remove.constant = TRUE,
             allow.na = TRUE, printFlag = FALSE) 

gt1 <- with(gamt, rlm(PostDBP ~ Group + 
                    BaselineDBP + Age + 
                    Z + R1 + R2))

resultsgt <- mice::pool(gt1) 
ztable(summary(resultsgt))
term estimate std.error statistic df p.value
(Intercept) 1.02 8.27 0.12 5.77 0.91
Group 9.56 0.13 74.29 13.59 0.00
BaselineDBP 0.99 0.07 14.11 5.81 0.00
Age 0.33 0.01 32.31 10.11 0.00
Z -1.06 1.75 -0.60 8.22 0.56
R1 0.00 0.05 0.00 48.38 1.00
R2 -0.13 0.13 -0.98 12.43 0.35

Having done that, I shall try one more supplementary analysis which piqued my curiosity. I recently came across a few papers that concluded that random forests outperformed other imputation algorithms in recovering the true parameter values.

Generally, imputation algorithms are evaluated based on a general simulation design which often resembles the following:


  1. Sampling mechanism only. The basic simulation steps are: choose \(Q\), take samples \(Y^{(s)}\), fit the complete-data model, estimate \(\hat{Q}^{(s)}\) and \(U^{(s)}\) and calculate the outcomes aggregated over \(s\).

  1. Sampling and missing data mechanisms combined. The basic simulation steps are: choose \(Q\), take samples \(Y^{(s)}\), generate incomplete data \(Y_{\mathrm{obs}}^{(s, t)}\), impute, estimate \(\hat{Q}^{(s, t)}\) and \(T^{(s, t)}\) and calculate outcomes aggregated over \(s\) and \(t\).

  1. Missing data mechanism only. The basic simulation steps are: choose \((\hat{Q}, U)\), generate incomplete data \(Y_{o b s}^{(t)}\), impute, estimate \((Q, U)^{(t)}\) and \(B^{(t)}\) and calculate outcomes aggregated over \(t\).

These designs generally lead to evaluating a number of metrics which include the following:


  • Raw Bias: where \(RB\) is bias and \(Q\) is the difference between the estimate and the truth \(RB=E(Q)-Q\), which may also be expressed as a percent \(100*\frac{RB=E(Q)-Q}{Q}\)

  • Coverage Rate: which is simply the proportion of times the intervals actually contain the true parameter value

  • Average Width: the width of the interval estimates, with longer widths indicating less information

  • Root mean squared error (RMSE) \(\operatorname{RMSE}=\sqrt{\frac{1}{n_{\text {mis }}} \sum_{i=1}^{n_{\text {mits }}}\left(y_{i}^{\text {mis }}-\dot{y}_{i}\right)^{2}}\)

The authors primarily based their conclusions on the root mean squared error (RMSE). However, this is problematic for a number of reasons; imputation is not the same as prediction, and so the same loss functions are not as useful; indeed, the RMSE is highly misleading as a measure of performance.

van Buuren explains here:


It is well known that the minimum RMSE is attained by predicting the missing \(\dot{{y}}_{i}\) by the linear model with the regression weights set to their least squares estimates. According to this reasoning the “best” method replaces each missing value by its most likely value under the model. However, this will find the same values over and over, and is single imputation.

This method ignores the inherent uncertainty of the missing values (and acts as if they were known after all), resulting in biased estimates and invalid statistical inferences. Hence, the method yielding the lowest RMSE is bad for imputation. More generally, measures based on similarity between the true and imputed values do not separate valid from invalid imputation methods.


Having set up our simulation design, we can now test the performances between regression imputation norm.predict and and regular regression imputation norm and we will be examining the metrics above to evaluate performance. Regression imputation generally gives very biased results given that it uses its predictions to impute the missing values. Thus, I will show here how RMSE is highly misleading as a metric to evaluate the performance of imputation algorithms.



Our simulation above will generate a dataset, delete values from the dataset and then impute the missing values for that dataset using regression imputation and stochastic regression imputation. We will use two metrics to evaluate the performance of these two algorithms, RMSE and MAE. Once again, it is worth keeping in mind that regression imputation generally performs very poorly when it comes to imputation and that stochastic regression imputation performs better.



What this simulation clearly shows is that despite regression imputation norm.predict giving highly biased results as usual relative to the truth, it still has a much lower RMSE than stochastic regression imputation, which both had a higher RMSE and a closer distance to the correct parameter values. Once again, this is not surprising given that imputation is not the same thing as prediction and therefore the loss functions cannot be used synonymously. However, when it came to raw bias and mean absolute error, stochastic regression imputation outperformed regression imputation which had a much larger mean absolute error and raw bias.

This is why the most informative metrics to focus on when evaluating an imputation algorithm include information/width, sufficient long-run coverage, and less misleading metrics such as mean absolute error and mean absolute percentage error.


The authors specifically used the missForest package. I tried using the package, but encountered too many difficulties so I simply specified the rf method in mice, and worked from there.


# MI Using Random Forest  -------------------------------------

form <- list(PostDBP ~ Group + BaselineDBP + Age + Z + R1 + R2)

rf <- parlmice(BP_miss, method = "rf", ntree = 20, 
               predictorMatrix = pred, 
               m = 10, maxit = 5, cluster.seed = 1031,
               cores = cores, formulas = form, ridge = 0,
               remove.collinear = TRUE, remove.constant = TRUE,
               allow.na = TRUE, printFlag = FALSE,
               visitSequence = "monotone") 

rf1 <- with(rf, rlm(PostDBP ~ Group + 
                    BaselineDBP + Age + 
                    Z + R1 + R2))

results5 <- mice::pool(rf1)

results5_sum <- summary(results5)

colnames(results5_sum) <- c("Term", "Estimate", "SE",
                            "Statistic", "df", "P-val")
ztable(results5_sum)
Term Estimate SE Statistic df P-val
(Intercept) 29.47 7.63 3.86 54.58 0.00
Group 9.34 0.19 49.62 50.28 0.00
BaselineDBP 0.77 0.06 11.79 55.03 0.00
Age 0.29 0.02 18.78 30.31 0.00
Z 1.82 1.93 0.94 92.30 0.35
R1 0.01 0.12 0.12 22.43 0.91
R2 -0.03 0.21 -0.13 31.54 0.90

Full Analysis Set


After we run all those analyses, we can finally check our numbers from the full dataset with no missing observations to see how our approaches to handle the missing data have performed. We first look at the fully adjusted model fit with rlm().


# Full Analysis Set  -------------------------------------

final_full_mod <- rlm(PostDBP ~ Group + BaselineDBP +
                      Age + Z + R1 + R2, data = BP_full, 
                      method = "MM", psi = psi.huber)

ztable(summary(final_full_mod)[["coefficients"]])
Value Std. Error t value
(Intercept) -0.48 3.72 -0.13
Group 9.66 0.09 106.85
BaselineDBP 1.00 0.03 31.68
Age 0.33 0.01 50.17
Z -2.02 1.01 -2.00
R1 0.01 0.05 0.21
R2 -0.01 0.09 -0.09
ztable(check_collinearity(final_full_mod))
Term VIF SE_factor
Group 1.01 1.00
BaselineDBP 1.97 1.40
Age 1.02 1.01
Z 1.98 1.41
R1 1.01 1.01
R2 1.01 1.00

We can also graphically inspect our assumptions.



Then we look at the unadjusted model, where the difference is the group that the participants were randomized to.


# Unadjusted Model  -------------------------------------

final_reduced_mod <- rlm(PostDBP ~ Group, data = BP_full, 
                         method = "MM", psi = psi.huber)

final_reduced_mod_sum <- summary(final_reduced_mod)[["coefficients"]]

colnames(final_reduced_mod_sum) <- c("Value", 
                                     "Std. Error",
                                     "t-value")

ztable(final_reduced_mod_sum)
Value Std. Error t-value
(Intercept) 133.36 0.20 660.28
Group 9.20 0.28 32.73

We can also compare the two to see if there are any substantial differences.


# Compare Models -------------------------------------

ztable(compare_performance(final_full_mod, final_reduced_mod))
Name Model AIC AIC_wt BIC BIC_wt RMSE Sigma
final_full_mod rlm 1419.76 1 1453.47 1 0.98 0.99
final_reduced_mod rlm 2556.47 0 2569.12 0 3.10 3.11

While the results have mostly been consistent, I also wanted to run a few additional analyses of my own to see whether these effect estimates were consistent I first started by constructing the confidence distribution using the concurve package to see the range of parameter values consistent with the data given the background model assumptions. (26-27)



# Compatibility of Values -------------------------------------

fit1 <- lm(PostDBP ~ Group + BaselineDBP +
           Age + Z + R1 + R2, data = BP_full)

curve1 <- curve_lm(fit1, var = "Group", method = "rlm", 
                    log = FALSE, penalty = NULL,
                    m = NULL, steps = 1000, 
                    table = TRUE)


Indeed, our confidence distribution ends up being very similar to our Bayesian posterior distribution for the treatment effect. Under certain situations, the maximum likelihood estimate ends up also being equivalent to the median unbiased estimate. In this case, it happens to be very similar to the maximum a posteriori estimate.


\[C V_{n}(\theta)=1-2\left|H_{n}(\theta)-0.5\right|=2 \min \left\{H_{n}(\theta), 1-H_{n}(\theta)\right\}\]



We can also check to see that our confidence distribution and posterior distribution are consistent with the profile-likelihood function, which is free of nuisance parameters. The advantage of profiling is allowing us to recover more accurate parameter estimates when there are several nuisance parameters or messy data. Below are implementations in both R and Stata, except this time we do not specify errors from the t-distribution.


\(L_{t_{n}}(\theta)=f_{n}\left(F_{n}^{-1}\left(H_{p i v}(\theta)\right)\right)\left|\frac{\partial}{\partial t} \psi\left(t_{n}, \theta\right)\right|=h_{p i v}(\theta)\left|\frac{\partial}{\partial t} \psi(t, \theta)\right| /\left.\left|\frac{\partial}{\partial \theta} \psi(t, \theta)\right|\right|_{t=t_{n}}\)


xx <- glm(PostDBP ~ Group + BaselineDBP +
          Age + Z + R1 + R2,
          data = BP_full, family = gaussian())

ztable(summary(xx))
#> Error in 1:nrow(x): argument of length 0

ctrl <- heavy.control(maxIter = 200, 
                      tol = 1e-04, ndraws = 9000)

fit <- heavyLm(PostDBP ~ Group + BaselineDBP +
          Age + Z + R1 + R2, data = BP_full, 
          family = Student(df = 4), control = ctrl)

ztable(summary(fit))
#> Error in 1:nrow(x): argument of length 0

We now focus on profiling on the treatment effect and calculating the profile deviance function. We then multiply our deviance by -0.5 to obtain the profile log-likelihood function.


# Profile Likelihood -------------------------------------

prof <- profile(xx)

disp <- attr(prof, "summary")$dispersion
mindev <- attr(prof, "original.fit")$deviance

dev1 <- prof[[1]]$tau^2
dev2 <- dev1 * disp + mindev

tmpf <- function(x, n) {
data.frame(par = n, tau = x$tau,
           deviance = x$tau^2 * disp + mindev,
           x$par.vals, check.names = FALSE)
          }

pp <- do.call(rbind, mapply(tmpf, prof, names(prof),
              SIMPLIFY = FALSE))

pp2 <- melt(pp, id.var = 1:3)
pp3 <- subset(pp2, par == variable, select = -variable)
pp4 <- pp3[-c(1:13), ]


clear
#> . cl. import delimited "C:\datasets\temp.csv", numericcols(2 3 5 6 7 8)
#> (encoding automatically selected: ISO-8859-2)
#> (8 vars, 500 obs)
#> 
#> . foreach var of varlist  group baselinedbp age z r1 r2 {
#>   2. pllf glm postdbp group baselinedbp age z r1 r2, profile(`var') 
#>   3. graph copy Graph `var'
#>   4. }
#> command pllf is unrecognized
#> r(199);
#> 
#> end of do-file
#> r(199);

Profile Log-Likelihood


Lower Limit Upper Limit CI Width Level (%) CDF P-value S-value
9.617 9.674 0.057 25.0 0.625 0.750 0.415
9.585 9.706 0.120 50.0 0.750 0.500 1.000
9.543 9.748 0.205 75.0 0.875 0.250 2.000
9.531 9.760 0.229 80.0 0.900 0.200 2.322
9.517 9.774 0.257 85.0 0.925 0.150 2.737
9.499 9.792 0.294 90.0 0.950 0.100 3.322
9.471 9.821 0.350 95.0 0.975 0.050 4.322
9.445 9.846 0.400 97.5 0.988 0.025 5.322
9.415 9.876 0.461 99.0 0.995 0.010 6.644
Table of Statistics for Various Interval Estimate Percentiles

I then compared the results of the full analysis set to the results from the primary analysis and sensitivity analyses.



Examining the Analyses


Table 1: Evaluation of Imputation Algorithms
PMM Heckman Stata Bayes GAMLSS RI RF True
(Intercept) 3.70 1.53 6.84 -7.21 1.02 1.89 29.47 -0.48
Group 9.52 9.64 10.44 9.72 9.56 9.56 9.34 9.66
BaselineDBP 0.97 0.98 0.95 1.05 0.99 0.98 0.77 1.00
Age 0.33 0.34 0.32 0.35 0.33 0.34 0.29 0.33
Z -0.21 -1.77 -2.64 -1.94 -1.06 -0.92 1.82 -2.02
R1 -0.02 -0.05 -0.36 -0.16 0.00 -0.02 0.01 0.01
R2 -0.16 -0.13 -0.17 -0.19 -0.13 -0.13 -0.03 -0.01
MPE -109.47 -88.32 427.76 -300.41 -155.78 -104.09 889.51 0.00
Huber Loss 0.71 0.22 1.06 0.90 0.21 0.36 4.70 0.00
MAE 0.91 0.36 1.33 1.04 0.39 0.54 4.92 0.00
MSD -0.80 -0.29 -0.98 0.98 -0.32 -0.46 -4.74 0.00
Metric Abbreviations:
MPE = Mean Percentage Error; MAE = Mean Absolute Error; MSD = Mean Signed Deviation
MI Abbreviations:
PMM = Predictive Mean Matching; Heckman = Heckman Selection Model; RI = Random Indicator; RF = Random Forest; Bayes = Imputation During Model Fitting

We can see that that the overall treatment effect is approximately 9.7-9.9, and for most of the principled methods we used to handle the missing data, we did not obtain similar estimates. For example, both the primary analysis and the controlled multiple imputation methods did not yield the correct treatment effect estimate in the adjusted model.

In fact, nearly all of the methods used except for the Heckman selection model, yielded overestimates of the treatment effect. Even though such analyses can be substantially improved with more specification and more information, the reality is that data that are assumed to be MNAR are difficult to handle and that no method can guarantee the correct answer. Indeed, we knew the missing data model / mechanism, via our custom function


# Missing Data Mechanism -------------------------------------

lesslikely <- function() {
set.seed(1031, kind = "L'Ecuyer-CMRG", 
         normal.kind = "Inversion", 
         sample.kind = "Rejection")
  logistic <- function(x) exp(x) / (1 + exp(x))
  1 - rbinom(n, 1, logistic(errors[, 2]))
}

Ry <- ifelse(lesslikely() > 0, 1, 0)
PostDBP[Ry == 0] <- NA

\[\begin{equation} \Pr(R_1=0)=\psi_0+\frac{e^{Y_1}}{1+e^{Y_1}}\psi_1 \end{equation}\]


for MNAR we set \(\psi_\mathrm{MNAR}=(0,1)\)


Thus, we obtain the following model:


\[\begin{align} \mathrm{MNAR}&:\mathrm{logit}(\Pr(R_1=0)) = Y_1 \\ \end{align}\]


Even the random indicator method, which has shown much promise in simulation studies (24) and offers the advantage of not having to specify a pattern-mixture or selection model could not recover the correct treatment effect estimate either, showing the difficulty of handling missing data and the limitations of automated approaches.


I hope it is clear from the examples above that it is quite easy to do any analyses with missing data and use a variety of ad-hoc approaches or even principled approaches to explore whether or not the results you’ve obtained are trustworthy or at least even give that impression, but doing a careful and principled one is quite difficult and requires much thought that will take into account real departures from assumptions and whether or not those will lead to substantial changes in results and conclusions/decisions.


Acknowledgements


I’d like to thank Dr. Tim Morris and Dr. Andrew Althouse for helpful comments on an earlier version of this piece. My acknowledgment of their help does not imply endorsement of my views by these colleagues or vice versa, and I remain solely responsible for the views expressed herein. All errors are my own. To point out such errors, please leave a comment below or use the contact form to get in touch.


Statistical Software


R Environment


#>  setting  value
#>  version  R version 4.1.2 (2021-11-01)
#>  os       Windows 10 x64 (build 19043)
#>  system   x86_64, mingw32
#>  ui       RTerm
#>  language (EN)
#>  collate  English_United States.1252
#>  ctype    English_United States.1252
#>  tz       America/New_York
#>  date     2022-01-18
#>  pandoc   2.14.0.3 @ C:/Program Files/RStudio/bin/pandoc/ (via rmarkdown)
#> R version 4.1.2 (2021-11-01)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 19043)
#> 
#> Matrix products: default
#> 
#> Random number generation:
#>  RNG:     L'Ecuyer-CMRG 
#>  Normal:  Inversion 
#>  Sample:  Rejection 
#>  
#> locale:
#> [1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
#> [4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    
#> 
#> attached base packages:
#>  [1] splines   grid      stats4    parallel  stats     graphics  grDevices datasets  utils     methods   base     
#> 
#> other attached packages:
#>  [1] texPreview_1.5        tinytex_0.36          rmarkdown_2.11        brms_2.16.3           bootImpute_1.2.0     
#>  [6] miceMNAR_1.0.2        knitr_1.37            boot_1.3-28           reshape2_1.4.4        ProfileLikelihood_1.1
#> [11] ImputeRobust_1.3-1    gamlss_5.3-4          nlme_3.1-155          gamlss.dist_6.0-1     MASS_7.3-55          
#> [16] gamlss.data_6.0-2     mvtnorm_1.1-3         performance_0.8.0     summarytools_1.0.0    tidybayes_3.0.2      
#> [21] htmltools_0.5.2       missForest_1.4        itertools_0.1-3       randomForest_4.6-14   rstan_2.21.3         
#> [26] StanHeaders_2.21.0-7  Statamarkdown_0.7.0   car_3.0-12            carData_3.0-5         qqplotr_0.0.5        
#> [31] ggcorrplot_0.1.3      Amelia_1.8.0          Rcpp_1.0.8            blogdown_1.7.1        doParallel_1.0.16    
#> [36] iterators_1.0.13      foreach_1.5.1         bayesplot_1.8.1       wesanderson_0.3.6     VIM_6.1.1            
#> [41] colorspace_2.0-2      here_1.0.1            mi_1.0                Matrix_1.4-0          broom_0.7.11         
#> [46] yardstick_0.0.9       svglite_2.0.0         Cairo_1.5-14          cowplot_1.1.1         kableExtra_1.3.4     
#> [51] heavy_0.38.196        forcats_0.5.1         stringr_1.4.0         dplyr_1.0.7           purrr_0.3.4          
#> [56] readr_2.1.1           tibble_3.1.6          tidyverse_1.3.1       ggtext_0.1.1          concurve_2.7.7       
#> [61] showtext_0.9-4        showtextdb_3.0        sysfonts_0.8.5        future_1.23.0         tidyr_1.1.4          
#> [66] magrittr_2.0.1        mice_3.14.0           rms_6.2-0             SparseM_1.81          Hmisc_4.6-0          
#> [71] ggplot2_3.3.5         Formula_1.2-4         survival_3.2-13       lattice_0.20-45      
#> 
#> loaded via a namespace (and not attached):
#>   [1] mitools_2.4            pander_0.6.4           haven_2.4.3            tcltk_4.1.2            vctrs_0.3.8           
#>   [6] usethis_2.1.5          mgcv_1.8-38            gmp_0.6-2.1            prodlim_2019.11.13     later_1.3.0           
#>  [11] nloptr_1.2.2.3         DBI_1.1.2              distributional_0.3.0   jpeg_0.1-9             MatrixModels_0.5-0    
#>  [16] pspline_1.0-18         htmlwidgets_1.5.4      miscTools_0.6-26       inline_0.3.19          laeken_0.5.2          
#>  [21] markdown_1.1           DEoptimR_1.0-10        DT_0.20                promises_1.2.0.1       pkgload_1.2.4         
#>  [26] magick_2.7.3           RcppParallel_5.1.5     startupmsg_0.9.6       fs_1.5.2               mnormt_2.0.2          
#>  [31] ranger_0.13.1          digest_0.6.29          png_0.1-7              polspline_1.1.19       bookdown_0.24         
#>  [36] pkgconfig_2.0.3        dygraphs_1.1.1.6       gower_0.2.2            minqa_1.2.4            xfun_0.29             
#>  [41] bslib_0.3.1            zoo_1.8-9              tidyselect_1.1.1       pcaPP_1.9-74           viridisLite_0.4.0     
#>  [46] gamm4_0.2-6            pkgbuild_1.3.1         rlang_0.4.12           jquerylib_0.1.4        Rmpfr_0.8-7           
#>  [51] glue_1.6.0             gdtools_0.2.3          RColorBrewer_1.1-2     pryr_0.1.5             distrEx_2.8.0         
#>  [56] modelr_0.1.8           matrixStats_0.61.0     lava_1.6.10            ggsignif_0.6.3         recipes_0.1.17        
#>  [61] threejs_0.3.3          labeling_0.4.2         httpuv_1.6.5           class_7.3-20           pbmcapply_1.5.0       
#>  [66] TH.data_1.1-0          clipr_0.7.1            shinystan_2.5.0        webshot_0.5.2          jsonlite_1.7.3        
#>  [71] tmvnsim_1.0-2          mime_0.12              systemfonts_1.0.3      gridExtra_2.3          easypackages_0.1.0    
#>  [76] stringi_1.7.6          insight_0.15.0         processx_3.5.2         distr_2.8.0            gsl_2.1-7.1           
#>  [81] survey_4.1-1           ggdist_3.0.1           cli_3.1.0              data.table_1.14.2      rsconnect_0.8.25      
#>  [86] survminer_0.4.9        officer_0.4.1          rstudioapi_0.13        sfsmisc_1.1-12         listenv_0.8.0         
#>  [91] miniUI_0.1.1.1         survMisc_0.5.5         dbplyr_2.1.1           sessioninfo_1.2.2      readxl_1.3.1          
#>  [96] lifecycle_1.0.1        timeDate_3043.102      posterior_1.2.0        munsell_0.5.0          cellranger_1.1.0      
#> [101] visNetwork_2.1.0       codetools_0.2-18       coda_0.19-4            systemfit_1.1-24       magic_1.5-9           
#> [106] lmtest_0.9-39          sys_3.4                htmlTable_2.4.0        rstantools_2.1.1       gWidgets2_1.0-9       
#> [111] xtable_1.8-4           scam_1.2-12            abind_1.4-5            farver_2.1.0           parallelly_1.30.0     
#> [116] km.ci_0.5-2            rapportools_1.0        credentials_1.3.2      askpass_1.1            svUnit_1.0.6          
#> [121] rcartocolor_2.0.0      shinythemes_1.2.0      cluster_2.1.2          future.apply_1.8.1     ellipsis_0.3.2        
#> [126] prettyunits_1.1.1      lubridate_1.8.0        ggridges_0.5.3         reprex_2.0.1           details_0.2.1         
#> [131] flextable_0.6.10       igraph_1.2.11          ismev_1.42             shinyjs_2.1.0          remotes_2.4.2         
#> [136] testthat_3.1.1         yaml_2.2.1             loo_2.4.1              utf8_1.2.2             svgPanZoom_0.3.4      
#> [141] ModelMetrics_1.2.2.2   e1071_1.7-9            ggpubr_0.4.0           foreign_0.8-82         withr_2.4.3           
#> [146] VineCopula_2.4.3       trust_0.1-8            multcomp_1.4-18        robustbase_0.93-9      gWidgets2tcltk_1.0-7  
#> [151] devtools_2.4.3         memoise_2.0.1          evaluate_0.14          VGAM_1.1-5             tzdb_0.2.0            
#> [156] callr_3.7.0            caret_6.0-90           ps_1.6.0               DiagrammeR_1.0.7       metafor_3.0-2         
#> [161] evd_2.3-3              fansi_1.0.2            highr_0.9              xts_0.12.1             conquer_1.2.1         
#> [166] checkmate_2.0.0        renv_0.15.1            cachem_1.0.6           desc_1.4.0             bcaboot_0.2-3         
#> [171] maxLik_1.5-2           tensorA_0.36.2         rstatix_0.7.0          GJRM_0.2-5.1           rprojroot_2.0.2       
#> [176] tools_4.1.2            stabledist_0.7-1       sass_0.4.0             sandwich_3.0-1         proxy_0.4-26          
#> [181] pbivnorm_0.6.0         xml2_1.3.3             httr_1.4.2             assertthat_0.2.1       globals_0.14.0        
#> [186] R6_2.5.1               nnet_7.3-17            Brobdingnag_1.2-6      gtools_3.9.2           sampleSelection_1.2-12
#> [191] rematch2_2.1.2         projpred_2.0.2         generics_0.1.1         base64enc_0.1-3        gridtext_0.1.4        
#> [196] pillar_1.6.4           sp_1.4-6               uuid_1.0-3             plyr_1.8.6             gtable_0.3.0          
#> [201] rvest_1.0.2            zip_2.2.0              psych_2.1.9            colourpicker_1.1.1     latticeExtra_0.6-29   
#> [206] fastmap_1.1.0          crosstalk_1.2.0        ADGofTest_0.3          copula_1.0-1           quantreg_5.86         
#> [211] extremevalues_2.3.3    vcd_1.4-9              arrayhelpers_1.1-0     openssl_1.4.6          scales_1.1.1          
#> [216] arm_1.12-2             backports_1.4.1        ipred_0.9-12           lme4_1.1-27.1          hms_1.1.1             
#> [221] shiny_1.7.1            KMsurv_0.1-5           numDeriv_2016.8-1.1    mathjaxr_1.4-0         whisker_0.4           
#> [226] crayon_1.4.2           pROC_1.18.0            bridgesampling_1.1-2   rpart_4.1-15           compiler_4.1.2

Stata Environment


#> . versversion 17.0

Bibliographic References


1. Stark PB, & Saltelli A. (2018). ‘Cargo-cult statistics and scientific crisis’. Significance. 15:40–43. doi: 10.1111/j.1740-9713.2018.01174.x.
2. Gigerenzer G. (2018). ‘Statistical Rituals: The Replication Delusion and How We Got There. Advances in Methods and Practices in Psychological Science. 1:198–218. doi: 10.1177/2515245918771329.
3. Greenland S. (2017). ‘Invited commentary: The need for cognitive science in methodology’. American Journal of Epidemiology. 186:639–645. doi: 10.1093/aje/kwx259.
4. Greenland S. (2005). ‘Multiple-bias modelling for analysis of observational data’. Journal of the Royal Statistical Society Series A (Statistics in Society). 168:267–306. doi: 10.1111/j.1467-985X.2004.00349.x.
5. Greenland S, & Lash TL. (2008). ‘Bias analysis’. In: Rothman KJ, Greenland S, & Lash TL, editors. Modern Epidemiology. 3rd edition. Lippincott Williams & Wilkins. p. 345–380
6. Lash TL, Fox MP, & Fink AK. (2009). ‘Applying Quantitative Bias Analysis to Epidemiologic Data. Springer New York
7. Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, & Greenland S. (2014). ‘Good practices for quantitative bias analysis’. International Journal of Epidemiology. 43:1969–1985. doi: 10.1093/ije/dyu149.
8. Lash TL, Ahern TP, Collin LJ, Fox MP, & MacLehose RF. (2021). ‘Bias Analysis Gone Bad. American Journal of Epidemiology. doi: 10.1093/aje/kwab072.
9. Gustafson P. (2021). ‘Invited Commentary: Toward Better Bias Analysis. American Journal of Epidemiology. doi: 10.1093/aje/kwab068.
10. Greenland S. (2021). ‘Dealing with the Inevitable Deficiencies of Bias Analysis and All Analyses. American Journal of Epidemiology. doi: 10.1093/aje/kwab069.
11. Molenberghs G, Fitzmaurice G, Kenward MG, Tsiatis A, & Verbeke G. (2014). ‘Handbook of Missing Data Methodology. CRC Press
12. Scott RaP. (2002). ‘The Multicentre Aneurysm Screening Study (MASS) into the effect of abdominal aortic aneurysm screening on mortality in men: A randomised controlled trial’. The Lancet. 360:1531–1539. doi: 10.1016/S0140-6736(02)11522-4.
13. Morris TP, Kahan BC, & White IR. (2014). ‘Choosing sensitivity analyses for randomised trials: principles’. BMC Medical Research Methodology. 14:11. doi: 10.1186/1471-2288-14-11.
14. Mosteller F, & Tukey JW. (1987). ‘Data analysis including statistics’. In: The Collected Works of John W. Tukey: Philosophy and Principles of Data Analysis 1965-1986. CRC Press
15. Akacha M, Bretz F, Ohlssen D, Rosenkranz G, & Schmidli H. (2017). ‘Estimands and Their Role in Clinical Trials. Statistics in Biopharmaceutical Research. 9:268–271. doi: 10.1080/19466315.2017.1302358.
16. Permutt T. (2020). ‘Do Covariates Change the Estimand?’. Statistics in Biopharmaceutical Research. 12:45–53. doi: 10.1080/19466315.2019.1647874.
17. Mitroiu M, Oude Rengerink K, Teerenstra S, Pétavy F, & Roes KCB. (2020). ‘A narrative review of estimands in drug development and regulatory evaluation: Old wine in new barrels?’. Trials. 21:671. doi: 10.1186/s13063-020-04546-1.
18. Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. (2012). ‘The Prevention and Treatment of Missing Data in Clinical Trials. New England Journal of Medicine. 367:1355–1360. doi: 10.1056/NEJMsr1203730.
19. Chen D-G(Din), Peace KE, & Zhang P. (2017). ‘Clinical trial data analysis using r and SAS’. 2nd edition. Chapman; Hall/CRC. doi: 10.1201/9781315155104.
20. Buuren S van. (2018). ‘Flexible Imputation of Missing Data, Second Edition. CRC Press
21. Meng X-L, & Rubin DB. (1992). ‘Performing Likelihood Ratio Tests with Multiply-Imputed Data Sets. Biometrika. 79:103–111. doi: 10.2307/2337151.
22. Schenker N, & Taylor J. (1996). ‘Partially parametric techniques for multiple imputation’. Computational Statistics & Data Analysis. 22:425–446. doi: 10.1016/0167-9473(95)00057-7.
23. Permutt T. (2016). ‘Sensitivity analysis for missing data in regulatory submissions’. Statistics in Medicine. 35:2876–2879. doi: 10.1002/sim.6753.
24. Jolani S. (2012). ‘Dual Imputation Strategies for Analyzing Incomplete Data
25. von Hippel PT, & Bartlett J. (2019). ‘Maximum likelihood multiple imputation: Faster imputations and consistent standard errors without posterior draws’. arXiv:12100870 [stat]. https://arxiv.org/abs/1210.0870
26. Rafi Z, & Greenland S. (2020). ‘Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise’. BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
27. Greenland S, & Rafi Z. (2020). ‘To Aid Scientific Inference, Emphasize Unconditional Descriptions of Statistics. arXiv:190908583 [statME]. https://arxiv.org/abs/1909.08583


Help support the website!






comments powered by Disqus