Model A
|
TarBase model with randomized sequences and randomly picked data as negative set (without MirTif negative dataset).
|
Model B
|
miRecords model with randomized sequences and randomly picked sequences as negative dataset (without MirTif negative dataset).
|
Model C
|
TarBase model with MirTif negative dataset added to already existing negative dataset of model A.
|
Model D
|
miRecords model with MirTif negative dataset added to already existing negative dataset of Model B.
|
* Out of total 38 negative instances as the negative dataset used by MirTif, we could get 32 of them which we added to our negative dataset.
Abbreviations: |
TP=True Positive |
TN= True Negative |
FP=False Positive |
FN= False Negative |
Sn=Sensitivity |
C=Cost |
G=Gamma |
CV=Cross Validation |
MCC=Matthew Correlation Coefficient |
TEST-1
| C | G | CV | Accuracy | TP | TN | FP | FN | Sn=TP/ TP+FN | Sp=TN/ TN+FP | MCC | AUC |
Model A | 8.0 | 2.0 | 94.52% | 88.46% | 76 | 108 | 12 | 12 | 0.86 | 0.91 | 0.78 | 0.94 |
Model B | 2.0 | 2.0 | 93.18% | 89.33% | 64 | 95 | 10 | 9 | 0.88 | 0.90 | 0.75 | 0.98 |
The above results for TEST – 1 show details for our two models (Model A and Model B).Performance of Model A after adding of 32 negative dataset of MiRTif on our testing set(MiRecords) which had 120 negative class instances and 88 positive class instances. Similarly Model B testing set contained 73 positive class instances and 105 negative class instances containing 32 negative instances from Mirtif. Area Under the ROC Curve of the two models Model A and Model B: 0.94 and 0.98 respectively.
Conclusion: Without considering the MirTif negative dataset for training Model A and B classifiers, our Model A and B performance was noted to be better than MirTif.
TEST-2
| Number of interaction | True | False | Accuracy |
miRecords | 117 | 87 | 30 | 74.00% |
Assessed the performance of MiRTif by taking the interaction duplex from miRecords database (the experimentally validated database). The total number of interaction duplex collected from the miRecords is 117 and that had been used by us to train the model. We found 87 out of 117 positive interaction are correctly predicted and 30 out of 117 positive interaction duplex are incorrectly predicted by MiRTif webserver.
Conclusion: Our purpose of this test was to check the behavior of MirTif classifier over the unseen positive data as MirTif has been developed totally with Tarbase positive instances. Here in this test we subjected MirTif to the postitive unseen instances from MiRecords. We observed the fall in accuracy from ~82% to 74%. This implies that unlike our model, which ably predicts unseen data with almost same performance, MirTif's performances falls on unseen data. It has to be noted that performance of MirTif was measured on the same test set which was used to train it.
TEST-3
| C | G | CV | Accuracy | TP | TN | FP | FN | Sn=TP/ TP+FN | Sp=TN/ TN+FP | MCC | AUC |
Model C | 2 | 2 | 93.26% | 86.54% | 65 | 115 | 5 | 23 | 0.74 | 0.96 | 0.73 | 0.9473 |
Model D | 2 | 2 | 93.27% | 90.45% | 56 | 105 | 0 | 17 | 0.77 | 1 | 0.81 | 0.9850 |
Model C training set consists of 73 positive instances and 105 negative instances which contains 32 negatives instances of MirTif too. Out of 88 positive and 120 negative instances the model predicts 65 positive instances and 115 negative instances correctly, giving an accuracy of 86.54% with an AUC of 0.9473.
Similarly,Model D training set consists of 88 positive instances and 120 negative instances.
Out of 73 positive and 105 negative the models predicts 56 positive class and 105 negative giving an accuracy of 90.45 % with an AUC of 0.9850.
Conclusion: With test 3 our models' performance increased slightly and it was able to pick all negative instances correctly at the cost of number of true positives which could be seen as an impact of class imbalance problem.
Average Performance comparison between TAREF(with MirRecords models) and MirTif.
| Avg. Accuracy % | Sensitivity | Specificity | Area Under Curve |
MirTif | 81.97 | 0.835 | 0.736 | 0.86 |
TAREF | 91.65 | 0.85 | 0.96 | 0.98 |
ROC performance for various models:
|