M value

M value: why I decided to split the probes into "red" and "green"

SVD on the entire 27k by 511 patients M value matrix, plot 1st eigenarray (u matrix) and "color" the points according to the dye with with each CpG was labeled:

M value: analysis of red probes, identification of adjustment variables

Percent variance explained of the "red" dataset:

Potential adjustment variables: batch (sixth field in the patient ID barcode); center (second field in the patient ID barcode); day, month, year of shipment; concentration, amount, plate row, plate column. Potential biology: tumor stage, tumor grade, age.

Correlation of the red probes, first four eigengenes with the adjustment and bio variables:

(Kruskal-Wallist test for categorical variables and Spearman correlation for age)

It looks like removing the batch and the plate row did help some with the center effect but the plate column effect is still significantly higher. Need to remove that.

From my previous work with Brig we identified that day, month and year of shipment and center are highly correlated with batch. Therefore start by removing the batch effect. Percent variance explained after removing the batch effect:

> X<-model.matrix(~factor(batch))
> Xbc<-solve(t(X) %*% X) %*% t(X) %*% t(red)
> redB<- red-t(X %*% Xbc)

Looks like the first eigengene now explains ~20% of the variance. Correlation with the adjustment and bio variables after removing the batch effect:

Removing the batch effect didn't completely remove the center and the row/column effect. Day, month and year of shipment have been taken care of. Next remove the batch the the plate row effect. We identified with Justin the the entire 0652 batch doesn't have any plate row or column information:

> table(batch, adj$plate_row)

batch  A B C D E F G H
  0359 5 5 5 5 5 5 4 4
  0402 5 4 4 5 5 5 4 4
  0432 6 5 5 5 6 5 6 5
  0460 6 6 6 6 6 6 6 5
  0475 6 6 6 6 5 6 6 5
  0501 3 3 3 3 3 3 2 2
  0536 6 6 6 6 6 6 6 5
  0563 6 6 6 6 6 6 6 5
  0581 6 6 6 6 6 6 6 5
  0652 0 0 0 0 0 0 0 0
  0667 6 6 6 6 6 6 5 5
  0708 6 6 6 6 5 6 5 5
  0807 1 0 0 2 0 1 0 0
> table(batch, adj$plate_column)
batch  1 2 3 4 5 6
  0359 8 8 8 8 6 0
  0402 8 8 6 8 6 0
  0432 5 8 7 8 8 7
  0460 8 8 8 8 8 7
  0475 8 8 7 8 8 7
  0501 8 8 6 0 0 0
  0536 8 8 8 8 8 7
  0563 8 8 8 8 8 7
  0581 8 8 8 8 8 7
  0652 0 0 0 0 0 0
  0667 8 7 8 8 8 7
  0708 8 7 7 8 8 7
  0807 2 1 1 0 0 0
 

We decided to remove the patients from the batch 0652 from further analyses (43) and work with a smaller dataset (468 patients):

> mask<-batch!="0652"
> length(mask)
[1] 511
> table(mask)
mask
FALSE  TRUE
   43   468
> X<-model.matrix(~factor(batch[mask]) + adj$plate_row[mask])
> Xbcrw<-solve(t(X) %*% X) %*% t(X) %*% t(red[,mask])
> redBR<- red[,mask] - t(X %*% Xbcrw)

Percent variance explained after removing the batch and the plate row effects:

The first principal component is smaller but not significantly so. Lets look again at the variables:

Now remove the batch, plate row and plate column, look at the percent variance explained:

Now the first principal component explains a little less than 15% of the overall variability. Correlation with the adjustment and bio variables, see if the center effect still present:

Finally, the center effect needs to go. Variables to adjust for: batch, center, plate row, plate column. Percent Variance explained after the adjustment:

Under 14%! Still large effect, look at the variables:

Now lets take a look at the outliers of the first eigengene (patients 6 vs 367):

I guess it doesn't look too terrible. I also tried to remove all the listed variables as well as the first principal component and here is what I got in terms of the percent variance explained and the outliers:  
To me it looks worse than with the first principal component. Final: remove the batch, center, plate row and plate column from the data. 

M value: analysis of green probes, identification of adjustment variables

Percent Variance explained:

Remove the batch, center, plate row, plate column (also mask that one batch), looks at the first eigengene and the eigenarray:

P values after the adjustment:

Look at the outliers:

Weird!
Do one more test and remove the first eigengene together with the variables above:

Conclusion

Now scale (center=TRUE, scale=TRUE) bot datasets (red and green probes), combine them together for network construction.