Single parameter request not matching all parameter request

[rweigel]

See https://hapi-server.org/verify/?url=https://supermag.jhuapl.edu/hapi&dataset=indices_all

[rweigel]

is.FileContentSameOrConsistent(): Expect content from one parameter request to match content from all parameter request.
--
Got:Parameter 'SMU' does not match at time 2018-01-18T00:00Z: Single parameter request: 36.204342; All parameter request: 71.589996.Parameter 'SMU' does not match at time 2018-01-18T00:01Z: Single parameter request: 36.18755; All parameter request: 71.589996.Parameter 'SMU' does not match at time 2018-01-18T00:02Z: Single parameter request: 36.171463; All parameter request: 71.589996.Parameter 'SMU' does not match at time 2018-01-18T00:03Z: Single parameter request: 36.155609; All parameter request: 71.589996.(1436 additional messages suppressed.)

[is.FileContentSameOrConsistent()](https://github.com/hapi-server/verifier-nodejs/blob/master/is.js#Lundefined): Expect content from one parameter request to match content from all parameter request.
Got:
Parameter 'SMU' does not match at time 2018-01-18T00:00Z: Single parameter request: 36.204342; All parameter request: 71.589996.

Parameter 'SMU' does not match at time 2018-01-18T00:01Z: Single parameter request: 36.18755; All parameter request: 71.589996.

Parameter 'SMU' does not match at time 2018-01-18T00:02Z: Single parameter request: 36.171463; All parameter request: 71.589996.

Parameter 'SMU' does not match at time 2018-01-18T00:03Z: Single parameter request: 36.155609; All parameter request: 71.589996.

(1436 additional messages suppressed.)

[elwinter]

This comment is copied from an email from me to Bob, Jon, and Sandy on 2026-01-06.

Good afternoon folks. I was able to dig down to find the lines in supermag_api.py which actually fetch data from the native (non-HAPI) SuperMag server. To recap, issue #35 refers to inconsistencies in data returned from the SuperMag HAPI server when the indices_all table is requested as a whole, compared to the data returned when single parameters are requested. I extracted the bytestrings returned from the http requests to the native SuperMagserver at these URLs:

Single parameter (SMLmlt)
'https://supermag.jhuapl.edu/services/indices.php?python&nohead&start=2018-01-18T00:00&logon=superhapi&extent=000000086400&indices=sml,mlt'

returned the first few lines:

[{"tval": 1516233600.000000, "SML": -35.743866, "SMLmlt": 12.847858},
{"tval": 1516233660.000000, "SML": -35.306236, "SMLmlt": 12.864515},
{"tval": 1516233720.000000, "SML": -33.762257, "SMLmlt": 12.881180},
...

Whole indices_all table
'https://supermag.jhuapl.edu/services/indices.php?python&nohead&start=2018-01-18T00:00&logon=superhapi&extent=000000086400&indices=sme,sml,smlmlt,smlglon,smu,smumlt,smuglon,num,smes,smls,smlsmlt,smlsglon,smus,smusmlt,smusglon,nums,smed,smld,smldmlt,smldglon,smud,smudmlt,smudglon,numd,smer,smlr,smlrmlt,smlrglon,smur,smurmlt,smurglon,numr,smr,smrnum,ltsmr,ltnum,sml,mlat,sml,glat,sml,stid,smu,mlat,smu,glat,smu,stid,smls,mlats,smls,glats,smls,stids,smus,mlats,smus,glats,smus,stids,smld,mlatd,smld,glatd,smld,stidd,smud,mlatd,smud,glatd,smud,stidd,smlr,mlatd,smlr,glatr,smlr,stidr,smur,mlatr,smur,glatr,smur,stidr&swi=clockgse,clockgsm,density,pdyn,epsilon,newell&imf=bgse,bgsm,vgse,vgsm'

returned the first few lines:

[{"tval": 1516233600.000000, "SME": 71.948212, "SML": -35.743866, "SMLmlat": 66.594109, "SMLglat": 64.779999, "SMLstid": "EAG", "SMU": 36.204342, "SMUmlat": 66.707436, "SMUglat": 71.589996, "SMUstid": "TIK", "SMEnum": 144, "SMEs": 71.948212, "SMLs": -35.743866, "SMLsmlat": 66.594109, "SMLsglat": 64.779999, "SMLsstid": "EAG", "SMUs": 36.204342, "SMUsmlat": 66.707436, "SMUsglat": 71.589996, "SMUsstid": "TIK", "SMEsnum": 47, "SMEd": 29.912941, "SMLd": -11.732411, "SMLdmlat": 75.615707, "SMLdglat": 78.059998, "SMLdstid": "BBG", "SMUd": 18.180529, "SMUdmlat": 70.473801, "SMUdglat": 70.900002, "SMUdstid": "JAN", "SMEdnum": 97, "SMEr": [24.754791,29.912941,20.859982,19.022781,16.775768,16.775768,34.626537,34.285248,39.506004,10.787916,9.734932,40.993832,40.993832,41.435184,12.931376,19.142252,19.981009,19.981009,15.605640,12.464585,9.685377,6.034775,5.281168,19.142288], "SMLr": [-6.574262,-11.732411,-11.732411,-11.732411,-2.014430,-2.014430,1.577804,1.919093,-3.301662,-3.301662,-6.734826,-35.743866,-35.743866,-35.743866,-6.636868,-6.636868,-7.475625,-7.475625,-7.475625,-4.334569,-3.786185,-0.895956,-0.961760,-0.961760], "SMLrmlat": [71.889503,75.615707,75.615707,75.615707,67.354294,67.354294,74.552505,47.791527,54.299080,54.299080,70.641609,66.594109,66.594109,66.594109,58.144970,58.144970,72.447830,72.447830,72.447830,73.469017,57.829525,63.903595,65.008995,65.008995], "SMLrglat": [74.500000,78.059998,78.059998,78.059998,71.160004,71.160004,79.300003,52.169998,59.970001,59.970001,71.320000,64.779999,64.779999,64.779999,50.061001,50.061001,62.824001,62.824001,62.824001,64.199997,48.650002,56.400002,64.180000,64.180000], "SMLrstid": ["BJN","BBG","BBG","BBG","KHS","KHS","VIZ","IRT","MGD","MGD","BRW","EAG","EAG","EAG","C04","C04","RAN","RAN","RAN","CDC","T50","NAN","LRV","LRV"], "SMUr": [18.180529,18.180529,9.127569,7.290369,14.761339,14.761339,36.204342,36.204342,36.204342,7.486254,3.000106,5.249967,5.249967,5.691319,6.294508,12.505384,12.505384,12.505384,8.130016,8.130016,5.899192,5.138820,4.319407,18.180529], "SMUrmlat": [70.473801,70.473801,68.194489,61.935825,69.341019,69.341019,66.707436,66.707436,66.707436,41.771515,65.586121,63.398941,63.398941,67.471809,42.633930,61.836475,61.836475,61.836475,59.507896,59.507896,59.252960,74.088173,69.982544,70.473801], "SMUrglat": [70.900002,70.900002,71.089996,65.529999,73.529999,73.529999,71.589996,71.589996,71.589996,47.799999,66.970001,62.389999,62.389999,61.756001,37.090000,52.035000,52.035000,52.035000,49.389999,49.389999,49.810001,66.099998,64.169998,70.900002], "SMUrstid": ["JAN","JAN","NOR","NAD","DIK","DIK","TIK","TIK","TIK","BRN","T41","GAK","GAK","FSP","FRN","C05","C05","C05","T32","T32","T30","PGC","GHB","JAN"], "SMErnum": [49,53,37,11,6,6,5,4,5,5,5,12,13,16,20,28,26,23,19,20,19,14,14,22], "smr": 2.110451, "smrnum": 80, "smrnum00": 26, "smrnum06": 20, "smrnum12": 12, "smrnum18": 22, "bgse": {"X": 0.380000, "Y": 3.030000, "Z": -1.440000}, "bgsm": {"X": 0.380000, "Y": 3.350000, "Z": -0.230000}, "vgse": {"X": -324.899994, "Y": -16.000000, "Z": -8.200000}, "vgsm": {"X": -324.899994, "Y": -11.899240, "Z": -13.477688}, "clockgse": 0.492502, "clockgsm": 21.862360, "density": 5.660000, "dynpres": 1.230000, "epsilon": 7.894627, "newell": 1114.162964},
{"tval": 1516233660.000000, "SME": 71.493790, "SML": -35.306236, "SMLmlat": 66.594109, "SMLglat": 64.779999, "SMLstid": "EAG", "SMU": 36.187550, "SMUmlat": 66.707436, "SMUglat": 71.589996, "SMUstid": "TIK", "SMEnum": 144, "SMEs": 71.493790, "SMLs": -35.306236, "SMLsmlat": 66.594109, "SMLsglat": 64.779999, "SMLsstid": "EAG", "SMUs": 36.187550, "SMUsmlat": 66.707436, "SMUsglat": 71.589996, "SMUsstid": "TIK", "SMEsnum": 47, "SMEd": 26.731359, "SMLd": -10.656343, "SMLdmlat": 64.774460, "SMLdglat": 67.970001, "SMLdstid": "LOZ", "SMUd": 16.075016, "SMUdmlat": 70.473801, "SMUdglat": 70.900002, "SMUdstid": "JAN", "SMEdnum": 97, "SMEr": [20.910454,26.731359,20.070686,20.070686,18.570881,18.570881,34.591969,34.299885,39.381248,10.662943,8.902601,40.536446,40.536446,42.229702,13.218773,19.028961,19.913296,19.913296,15.548108,13.053182,9.752694,6.041216,5.238159,17.025091], "SMLr": [-4.835438,-10.656343,-10.656343,-10.656343,-4.023838,-4.023838,1.595583,1.887665,-3.193697,-3.193697,-6.070404,-35.306236,-35.306236,-35.306236,-6.295306,-6.295306,-7.179641,-7.179641,-7.179641,-4.684714,-3.605592,-0.918101,-0.950075,-0.950075], "SMLrmlat": [71.889503,64.774460,64.774460,64.774460,67.354294,67.354294,74.552505,47.791527,54.299080,54.299080,70.641609,66.594109,66.594109,66.594109,58.144970,58.144970,72.447830,72.447830,72.447830,73.469017,57.829525,63.903595,65.008995,65.008995], "SMLrglat": [74.500000,67.970001,67.970001,67.970001,71.160004,71.160004,79.300003,52.169998,59.970001,59.970001,71.320000,64.779999,64.779999,64.779999,50.061001,50.061001,62.824001,62.824001,62.824001,64.199997,48.650002,56.400002,64.180000,64.180000], "SMLrstid": ["BJN","LOZ","LOZ","LOZ","KHS","KHS","VIZ","IRT","MGD","MGD","BRW","EAG","EAG","EAG","C04","C04","RAN","RAN","RAN","CDC","T50","NAN","LRV","LRV"], "SMUr": [16.075016,16.075016,9.414343,9.414343,14.547043,14.547043,36.187550,36.187550,36.187550,7.469246,2.832198,5.230210,5.230210,6.923467,6.923467,12.733656,12.733656,12.733656,8.368467,8.368467,6.147101,5.123116,4.288084,16.075016], "SMUrmlat": [70.473801,70.473801,68.194489,68.194489,69.341019,69.341019,66.707436,66.707436,66.707436,41.771515,65.586121,63.398941,63.398941,67.471809,67.471809,61.836475,61.836475,61.836475,59.507896,59.507896,59.252960,74.088173,69.982544,70.473801], "SMUrglat": [70.900002,70.900002,71.089996,71.089996,73.529999,73.529999,71.589996,71.589996,71.589996,47.799999,66.970001,62.389999,62.389999,61.756001,61.756001,52.035000,52.035000,52.035000,49.389999,49.389999,49.810001,66.099998,64.169998,70.900002], "SMUrstid": ["JAN","JAN","NOR","NOR","DIK","DIK","TIK","TIK","TIK","BRN","T41","GAK","GAK","FSP","FSP","C05","C05","C05","T32","T32","T30","PGC","GHB","JAN"], "SMErnum": [48,53,37,12,6,6,5,4,5,5,5,12,13,16,20,26,26,23,21,20,19,14,14,22], "smr": 2.139009, "smrnum": 80, "smrnum00": 26, "smrnum06": 20, "smrnum12": 12, "smrnum18": 22, "bgse": {"X": 0.420000, "Y": 2.910000, "Z": -1.940000}, "bgsm": {"X": 0.420000, "Y": 3.420000, "Z": -0.740000}, "vgse": {"X": -327.799988, "Y": -13.500000, "Z": -1.400000}, "vgsm": {"X": -327.799988, "Y": -12.056726, "Z": -6.232601}, "clockgse": 334.580719, "clockgsm": 356.072418, "density": 5.390000, "dynpres": 1.140000, "epsilon": 21.062252, "newell": 2171.203857},
{"tval": 1516233720.000000, "SME": 69.933716, "SML": -33.762257, "SMLmlat": 66.594109, "SMLglat": 64.779999, "SMLstid": "EAG", "SMU": 36.171463, "SMUmlat": 66.707436, "SMUglat": 71.589996, "SMUstid": "TIK", "SMEnum": 144, "SMEs": 69.933716, "SMLs": -33.762257, "SMLsmlat": 66.594109, "SMLsglat": 64.779999, "SMLsstid": "EAG", "SMUs": 36.171463, "SMUsmlat": 66.707436, "SMUsglat": 71.589996, "SMUsstid": "TIK", "SMEsnum": 46, "SMEd": 25.007875, "SMLd": -10.669405, "SMLdmlat": 64.774460, "SMLdglat": 67.970001, "SMLdstid": "LOZ", "SMUd": 14.338470, "SMUdmlat": 69.341019, "SMUdglat": 73.529999, "SMUdstid": "DIK", "SMEdnum": 98, "SMEr": [14.997700,22.227242,20.443569,20.443569,17.399904,17.399904,34.557636,34.217724,39.241638,11.520597,8.632284,39.021038,39.021038,40.752090,13.802654,19.507328,19.664127,19.664127,15.216376,13.106757,9.748007,5.568404,5.829934,12.496017], "SMLr": [-3.439864,-10.669405,-10.669405,-10.669405,-3.061435,-3.061435,1.613829,1.953738,-3.070175,-3.070175,-5.423613,-33.762257,-33.762257,-33.762257,-6.812819,-6.812819,-6.969620,-6.969620,-6.969620,-4.860001,-3.540334,-0.676652,-0.938181,-0.938181], "SMLrmlat": [71.889503,64.774460,64.774460,64.774460,67.354294,67.354294,74.552505,47.791527,54.299080,54.299080,70.641609,66.594109,66.594109,66.594109,58.144970,58.144970,72.447830,72.447830,72.447830,73.469017,57.829525,63.903595,65.008995,65.008995], "SMLrglat": [74.500000,67.970001,67.970001,67.970001,71.160004,71.160004,79.300003,52.169998,59.970001,59.970001,71.320000,64.779999,64.779999,64.779999,50.061001,50.061001,62.824001,62.824001,62.824001,64.199997,48.650002,56.400002,64.180000,64.180000], "SMLrstid": ["BJN","LOZ","LOZ","LOZ","KHS","KHS","VIZ","IRT","MGD","MGD","BRW","EAG","EAG","EAG","C04","C04","RAN","RAN","RAN","CDC","T50","NAN","LRV","LRV"], "SMUr": [11.557836,11.557836,9.774164,9.774164,14.338470,14.338470,36.171463,36.171463,36.171463,8.450421,3.208672,5.258780,5.258780,6.989835,6.989835,12.694508,12.694508,12.694508,8.246757,8.246757,6.207673,4.891753,4.891753,11.557836], "SMUrmlat": [70.473801,70.473801,68.194489,68.194489,69.341019,69.341019,66.707436,66.707436,66.707436,41.771515,65.586121,63.398941,63.398941,67.471809,67.471809,61.836475,61.836475,61.836475,59.507896,59.507896,59.252960,69.982544,69.982544,70.473801], "SMUrglat": [70.900002,70.900002,71.089996,71.089996,73.529999,73.529999,71.589996,71.589996,71.589996,47.799999,66.970001,62.389999,62.389999,61.756001,61.756001,52.035000,52.035000,52.035000,49.389999,49.389999,49.810001,64.169998,64.169998,70.900002], "SMUrstid": ["JAN","JAN","NOR","NOR","DIK","DIK","TIK","TIK","TIK","BRN","T41","GAK","GAK","FSP","FSP","C05","C05","C05","T32","T32","T30","GHB","GHB","JAN"], "SMErnum": [48,53,37,12,6,6,5,4,5,5,5,12,13,16,20,25,26,22,22,20,20,14,14,22], "smr": 2.156304, "smrnum": 80, "smrnum00": 26, "smrnum06": 20, "smrnum12": 12, "smrnum18": 22, "bgse": {"X": 0.730000, "Y": 2.930000, "Z": -3.070000}, "bgsm": {"X": 0.730000, "Y": 3.850000, "Z": -1.790000}, "vgse": {"X": -327.600006, "Y": -13.700000, "Z": -1.800000}, "vgsm": {"X": -327.600006, "Y": -12.097867, "Z": -6.676177}, "clockgse": 326.309937, "clockgsm": 347.790894, "density": 5.670000, "dynpres": 1.220000, "epsilon": 29.711775, "newell": 2669.940918},

I compared the bytestrings returned by these calls. Two things struck me as odd:

1. The request URL sent for the single-parameter request requested parameters SML and SMLmlt - why is that? Does this correspond to the checkbox on the web interface that specifies including MLT in the response?

2. In the all-parameter response, there is no SMLmlt or smlmlt column, but that column does exist in the single-parameter request for SMLmlt.

On the face of it, this seems to be an inconsistency in the native SuperMag server, rather than the HAPI code. Are there parameters yhat are treated specially on either side? For example, is SMLmlt a computed value on the native server side, and thus not returned when all parameters of the indices_all table are requested? Remind me again who I should talk to about the native SuperMag server?

I'll keep poking. At Sandy's suggestion, I moved my work from the supoermag branchto a new branch off master; the supermag branch was merged into master a few weeks ago. The new branch I am working on )no changes committed yet) is called 035-inconsistent_supermag_results.

Take care,
Eric

[elwinter]

In addition to my previous comment, the error message from verifier.js indicates that the single-parameter-request SMLmlt value is being compared to the full-table-request SMLglat value, which is obviously incorrect - there is no connection between magnetic local time and magnetic latitude. They measure completely different physical quantities.

[elwinter]

So based on these two observations, we may have 2 distinct problems to solve.