Details in train.py output

[JonasLi-19]

Dear professor and developers, I come up against a few questions about the procedures in training.

1. I am not sure whether I should make these six folds almost the same sample amount, or should every train:test be 8:2 ?

2. After I train the model, one of the default2018*.out files contains: 0.801041 0.795815 0.558788 0.010000 1.746990 1.742589. the first two metirc are sequentially max_AUC min_AUC ... WAHT ARE THE FOUR NUMBERS.

[RJ-Li]

For the second question, you can find in the passage:

When evaluating a test set without counterexamples, models trained with counterexamples perform worse than their
counterparts trained without them at the pose selection task (0.885 to 0.845 AUC and 0.577 to 0.556 Top1), ...performance
improvements in affinity prediction (0.577 to 0.587 Pearson’s R and 1.463 to 1.457 RMSE) 

[drewnutt]

Dear professor and developers, I come up against a few questions about the procedures in training.

1. When I split my whole new dataset in train&validation, I use the default n_folds of '0,1,2' : `train_0, train_1, train_2,test_0,test_1,test_2`. However, I am not sure whether I should make these six folds almost the same sample amount, or should every train:test be 8:2 ?

2. After I train the model, one of the `default2018*.out` files contains: ` 0.801041 0.795815 0.558788 0.010000 1.746990 1.742589`. As I guess, the first two metirc are sequentially max_AUC min_AUC ... BUT I DO NOT KNOW WAHT ARE THE FOLLOWING FOUR NUMBERS.

1. The folds used in the paper were created via clustering of the protein pockets via the ProBIS algorithm. The clusters were then randomly assigned to folds. So for each fold in 0,1, and 2 the testing set is composed of only 1 fold and the training set is the other 2 folds. Each fold is about the same size in terms of number of examples, so the train test split is ~2:1.

2. The out file contains the following, in order: test_AUC, train_AUC, train_loss, learning_rate,test_RMSD_affinity, train_RMSD_affinity

   • If the model predicts the RMSD of a ligand, then there will be two more columns appended: test_RMSD_RMSE, train_RMSD_RMSE

[drewnutt]

I want to know more about the output rmsd, I didn't find an explanation for the default2018.***.rmsd.test number's meaning in the README.

I am not sure what default2018.***.rmsd.test is referring to here.

train.py will output a *.rmsd.finaltest at the end of training, that lists the affinity label and prediction for each example in the training set and the last line will show the RMSD of the affinity prediction on the test set.

[JonasLi-19]

If I want to train default2018.model on the crystal dataset, which is full of crystal ligands with RMSD=0. should I delete the RMSD column in types file, and set has_rmsd=false in the model?

[JonasLi-19]

I ask this question how to train on crystal dataset is because of the error I encountered:

I1102 07:12:28.331017 47833 layer_factory.hpp:77] Creating layer data
I1102 07:12:28.331130 47833 net.cpp:85] Creating Layer data
I1102 07:12:28.331151 47833 net.cpp:385] data -> data
I1102 07:12:28.331199 47833 net.cpp:385] data -> label
I1102 07:12:28.331220 47833 net.cpp:385] data -> affinity
I1102 07:12:28.331240 47833 net.cpp:385] data -> rmsd_true
Traceback (most recent call last):
  File "train.py", line 936, in <module>
    results = train_and_test_model(args, train_test_files[i], outname, cont)
  File "train.py", line 441, in train_and_test_model
    solver = caffe.get_solver(solverf)
ValueError: No valid examples found in training set.

My demand is python3 train.py -p crystal_2019_ -d /root/gnina_docker -m default2018.model --weights crossdock_default2018.caffemodel --dynamic

1 9.84 0 pdb2019_refi_train_gninatypes/2xef/2xef_rec.gninatypes pdb2019_refi_train_gninatypes/2xef/2xef_ligand.gninatypes
1 7.55 0 pdb2019_refi_train_gninatypes/3ni5/3ni5_rec.gninatypes pdb2019_refi_train_gninatypes/3ni5/3ni5_ligand.gninatypes
1 7.54 0 pdb2019_refi_train_gninatypes/4hu1/4hu1_rec.gninatypes pdb2019_refi_train_gninatypes/4hu1/4hu1_ligand.gninatypes

The rec and ligand gninatypes are in the correct address:

I am pretty sure the files are existed, because when I train other dataset with crystal gninatypes, they don't raise error!
SO, I guess, If I want to train default2018.model on the crystal dataset, which is full of crystal ligands with RMSD=0. should I delete the RMSD column in types file, and set has_rmsd=false in the model?

[dkoes]

If you are using default2018 unaltered it has balancing and stratification on be default, which makes no sense with an all crystal dataset.

[SanFran-Me]

Dear Professor, @dkoes
I trained crossdock_default2018 on PDBbind2018_refined-set, hoping it will be more powerful.
Actually it indeed perform better scoring CASF-2016, but when I use the result caffemodel to redock the PDBbind2018_refined, its output docked poses are more than 70% with more than 2A (TOP5 success rate around 30%).
What do you think of this result? I can only guess it is because the training set only contains label 1, so the CNNscore was trained to be worse?

[dkoes]

If you aren't training for pose selection, you won't get a model with good pose selection performance.

[Dadiao-shuai]

Hi, I noticed the --avg_rotations to test in the val-set for 24 rotations and get avg. My question is 'Does gnina train.py automatically use 24 rotations to train? if so, how can I increase this rotations num?'

[dkoes]

We train with random rotations enabled. There isn't a fixed number of rotations.

[Dadiao-shuai]

What should I do when generating gninatypes files if I couldn't calculate the RMSD?
I'm using default2018.model and its caffemodel to train on a new dataset. The only thing I find is to change: has_rmsd: true in default2018.model. IS THERE ANYTHING ELSE TO CHANGE?

layer {
  name: "data"
  type: "MolGridData"
  top: "data"
  top: "label"
  top: "affinity"
  top: "rmsd_true"
  include {
    phase: TEST
  }
  molgrid_data_param {
        source: "TESTFILE"
        batch_size: 50
        dimension: 23.5
        resolution: 0.500000
        shuffle: false
        ligmap: "completelig"
        recmap: "completerec"
        balanced: false
        has_affinity: true
        has_rmsd: true
        root_folder: "DATA_ROOT"
    }
  }
  
layer {
  name: "data"
  type: "MolGridData"
  top: "data"
  top: "label"
  top: "affinity"
  top: "rmsd_true"
  include {
    phase: TRAIN
  }
  molgrid_data_param {
        source: "TRAINFILE"
        batch_size:  50
        dimension: 23.5
        resolution: 0.500000
        shuffle: true
        balanced: true
        jitter: 0.000000
        ligmap: "completelig"
        recmap: "completerec"        
        stratify_receptor: true
        stratify_affinity_min: 0
        stratify_affinity_max: 0
        stratify_affinity_step: 1.000000
        has_affinity: true
        has_rmsd: true
        random_rotation: true
        random_translate: 6
        root_folder: "DATA_ROOT"       
    }
}
...
layer {
  name: "pose_output"
  type: "InnerProduct"
  bottom: "split"
  top: "pose_output"
  inner_product_param {
    num_output: 2
    weight_filler {
      type: "xavier"
    }
  }
}
...
layer {
  name: "output"
  type: "Softmax"
  bottom: "pose_output"
  top: "output"
}
layer {
  name: "labelout"
  type: "Split"
  bottom: "label"
  top: "labelout"
  include {
    phase: TEST
  }
}
layer {
  name: "rmsd"
  type: "AffinityLoss"
  bottom: "affinity_output"
  bottom: "affinity"
  top: "rmsd"
  affinity_loss_param {
    scale: 0.1
    gap: 0
    pseudohuber: false
    delta: 4
    penalty: 0
    ranklossmult: 0
    ranklossneg: 0    
  }
}

[dkoes]

Don't include the field in the input file either. What error do you get when you try it?

[Dadiao-shuai]

Thanks, I forget to exclude the rmsd field in the gninatypes.

[Dadiao-shuai]

failure when training on non-rmsd types

error:

I1219 08:58:55.233186 52309 layer_factory.hpp:77] Creating layer data
I1219 08:58:55.233378 52309 net.cpp:85] Creating Layer data
I1219 08:58:55.233404 52309 net.cpp:385] data -> data
I1219 08:58:55.233462 52309 net.cpp:385] data -> label
I1219 08:58:55.233489 52309 net.cpp:385] data -> affinity
I1219 08:58:55.233507 52309 net.cpp:385] data -> rmsd_true
F1219 08:58:55.233546 52309 layer.hpp:372] Check failed: ExactNumTopBlobs() == top.size() (3 vs. 4) MolGridData Layer produces 3 top blob(s) as output.
*** Check failure stack trace: ***
Aborted (core dumped)

this is my train|test types example: label, affinity, rec, lig
0 7.78 2020_m_pocket_gninatypes/5llo_m_pocket.gninatypes 2020_lig_mol2_gninatypes/5llo_ligand.gninatypes

this is the default2018.model, I set has_rmsd=false.

layer {
  name: "data"
  type: "MolGridData"
  top: "data"
  top: "label"
  top: "affinity"
  top: "rmsd_true"
  include {
    phase: TEST
  }
  molgrid_data_param {
        source: "TESTFILE"
        batch_size: 50
        dimension: 23.5
        resolution: 0.500000
        shuffle: false
        ligmap: "completelig"
        recmap: "completerec"
        balanced: false
        has_affinity: true
        has_rmsd: false
        root_folder: "DATA_ROOT"
    }
  }
  
layer {
  name: "data"
  type: "MolGridData"
  top: "data"
  top: "label"
  top: "affinity"
  top: "rmsd_true"
  include {
    phase: TRAIN
  }
  molgrid_data_param {
        source: "TRAINFILE"
        batch_size:  50
        dimension: 23.5
        resolution: 0.500000
        shuffle: true
        balanced: true
        jitter: 0.000000
        ligmap: "completelig"
        recmap: "completerec"        
        stratify_receptor: true
        stratify_affinity_min: 0
        stratify_affinity_max: 0
        stratify_affinity_step: 1.000000
        has_affinity: true
        has_rmsd: false
        random_rotation: true
        random_translate: 6
        root_folder: "DATA_ROOT"       
    }
}