Benchmarks

icon

13

pages

icon

English

icon

Documents

Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres

icon

13

pages

icon

English

icon

Documents

Le téléchargement nécessite un accès à la bibliothèque YouScribe Tout savoir sur nos offres

1 Benchmarks
The following requirements apply to all benchmarks. If a benchmark program spans
multiple machines, each machine must satisfy all of these requirements.
• Times are measured in real time, with a resolution of no worse than one second.
• The benchmark program and all related processes may use no more than 75%
of the machine’s physical memory, and in no event more than two gigabytes of
physical memory.
• The benchmark program and all related processes may use no more than ten
gigabytes of disk space.
• The benchmark program must produce the correct answer.
• The program must not encounter an error or prematurely terminate
during or after the benchmark.
• The benchmark program must be in a usable condition after the benchmark.
• The program may not degrade or disable the user interface, or in other
manner cause less work to be done than as expected by normal user interaction.
1.1 Test Environment
Processor Intel Pentium 4 stepping D1
3.2 GHz clock speed
800 MHz front side bus speed
Cache 8 KB L1 data
12 KB L1 instruction
512 KB L2 unified
Memory 1 GB DDR 2700 RAM
Disk 80 GB capacity
100 MB/s interface bandwidth
49.3 MB/s average transfer rate
19.2 ms access time
Operating System Microsoft Windows XP Service Pack 1a
Table 1: Machine Configuration
The processor HyperThreading extension was enabled during the benchmarks al
though this does not significantly affect performance. Almost all of the benchmarking
work is done on a single thread.
Execution time was ...
Voir icon arrow

Publié par

Nombre de lectures

50

Langue

English

1BenchmarksThefollowingrequirementsapplytoallbenchmarks.Ifabenchmarkprogramspansmultiplemachines,eachmachinemustsatisfyalloftheserequirements.Timesaremeasuredinrealtime,witharesolutionofnoworsethanonesecond.Thebenchmarkprogramandallrelatedprocessesmayusenomorethan75%ofthemachine’sphysicalmemory,andinnoeventmorethantwogigabytesofphysicalmemory.Thebenchmarkprogramandallrelatedprocessesmayusenomorethantengigabytesofdiskspace.Thebenchmarkprogrammustproducethecorrectanswer.Thebenchmarkprogrammustnotencounteranerrororprematurelyterminateduringorafterthebenchmark.Thebenchmarkprogrammustbeinausableconditionafterthebenchmark.Thebenchmarkprogrammaynotdegradeordisabletheuserinterface,orinothermannercauselessworktobedonethanasexpectedbynormaluserinteraction.1.1TestEnvironmentProcessorIntelPentium4steppingD13.2GHzclockspeed800MHzfrontsidebusspeedCache8KBL1data12KBL1instruction512KBL2uniedMemory1GBDDR2700RAMDisk80GBcapacity100MB/sinterfacebandwidth49.3MB/saveragetransferrate19.2msaccesstimeOperatingSystemMicrosoftWindowsXPServicePack1aTable1:MachineCongurationTheprocessorHyperThreadingextensionwasenabledduringthebenchmarksal-thoughthisdoesnotsignicantlyaffectperformance.Almostallofthebenchmarkingworkisdoneonasinglethread.Executiontimewasdeterminedbythesystemhighperformancetimer.Thehighperformancetimerhasaresolutionofoneprocessorclockcycle.Theruntimeenvi-ronmentwasconguredtoterminatethebenchmarkprogramifmorethan75%(768MB)ofphysicalmemorywasallocatedfromtheheap.Duringbenchmarkexecution,thepeakheapallocationsizewasrecorded.Notethatthisvaluemaybemorethan1
ProgramJigCellversion04302004CompilerIBMJikesJavacompilerversion1.19OptimizerSunJavaHotSpotClientVMversion1.4.2.03-b02,mixedmodeRuntimeSunJavaRuntimeEnvironmentSEversion1.4.2.03-b02Table2:SoftwareCongurationtheamountofmemoryrequiredbythebenchmarkasfreedobjectsneednotbeimme-diatelyreclaimedbytheruntimeenvironmentifmemoryisavilable.Additionally,alowerboundonthememoryrequiredwasobtainedbydecreasingtheamountofheapmemoryavailabletotheruntimeenvironmentuntilthebenchmarkwasunabletocom-pleteduetomemoryexhaustion.1.2ModelTestDataAmodelconsistsofthefollowingelements:ModelnameListofcompartments.Eachcompartmenthasanameanddescription.Listofreactions.Eachreactionhasaname,description,kineticinformation,andparameters.ThekineticequationforrectioniisdSdit+1=kiSiki+1Si+1startingwithi=0.Eachparameterhasanameanddescription.Thevalueofparameteriisi+12.Listofspeciesusedinreactions.Eachspecieshasanameanddescription.Thesizeofamodelisdeterminedbythenumberofreactions.1.3ModelLoadingPerformanceModelloadingperformancemeasuresthesizeofmodelssupportedbytheModelBuilder.Amodelissupportediftheloadingtimeislessthan300seconds.TheModelBuildermustsupportamodelcontaining1000reactionstosatisfythemodelbuildingperfor-mancerequirements.Theprocedureformeasuringmodelloadingperformanceis:1.CreateatestmodelcontainingNreactions2.Writetestmodeltodisk3.ReadtestmodelfromdiskStep3oftheprocedureistimed.AsofMay1st,theJigCellModelBuilderdoesnotpassthisbenchmark.Thetimetoloadamodelcontaining1000reactionsexceeds626secondsusinglessthan720MBofmemoryand686KBofdiskspace.2
1.4ReactionEntryPerformanceReactionentryperformancemeasuresthetimerequiredtoaddanewreactiontothemodel.TheModelBuildermustsupportaddingareactiontoamodelcontaining200reactionsin120secondstosatisfythemodelbuildingperformancerequirements.Theprocedureformeasuringreactionentryperformanceis:1.CreateatestmodelcontainingNreactions2.Writetestmodeltodisk3.Readtestmodelfromdisk4.AddspeciesN+15.AddreactionN+16.AddparameterN+17.WritetestmodeltodiskSteps3through7oftheprocedurearetimed.AsofMay1st,theJigCellModelBuilderhasnotbeenrunforthisbenchmark.1.5RunTestData1.6RunSetLoadingPerformanceRunsetloadingperformancemeasuresthesizeofrunsets,parametersets,andinitialconditionsetssupportedbytheModelBuilder.Asetissupportediftheloadingtimeislessthan300seconds.TheModelExecutormustsupportarunsetcontaining10000runs,500parametersets,and500initialconditionsetstosatifsythemodelexecutionperformancerequirements.TheModelExecutordoesnotneedtosupporttheselimitssimultaneously.Therearethreeproceduresformeasuringrunsetloadingperformance.Therstproceduremeasuresrunsetloadingperformancewiththemaximumrunsetsize:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create1testparameterset4.Create1testinitialconditionset5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.Readtestrunsetfromdisk3
Thesecondproceduremeasuresrunsetloadingperformancewiththemaximumparametersetsize:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.CreateNtestparametersets4.Create1testinitialconditionset5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.ReadtestrunsetfromdiskThethirdproceduremeasuresrunsetloadingperformancewiththemaximumini-tialconditionsetsize:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create1testparameterset4.CreateNtestinitialconditionsets5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.ReadtestrunsetfromdiskStep7ofeachoftheseproceduresistimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthesebenchmarks.1.7RunEntryPerformanceRunentryperformancemeasuresthetimerequiredtoaddanewruntoarunset.TheModelExecutormustsupportaddingaruntoarunsetcontaining2000runsin300secondstosatisfythemodelexecutionperformancerequirements.1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create100testparametersets4.Create100testinitialconditionsets5.CreateatestrunsetcontainingNruns4
6.Writetestrunsettodisk7.Readtestrunsetfromdisk8.AddrunN+19.WritetestrunsettodiskSteps7through9oftheprocedurearetimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark.1.8RunExecutionPerformanceRunexecutionperformancemeasuresthetimerequiredtoexecuteanexistingrun.TheModelExecutormustsupportexecutingarunforaworkingmodelfromarunsetcontaining2000runsin60secondstosatisfythemodelexecutionperformancerequirements.Theprocedureformeasuringrunexecutionperformanceis:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create100testparametersets4.Create100testinitialconditionsets5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.Readtestrunsetfromdisk8.ExecuterunNSteps7and8oftheprocedurearetimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark.1.9RunModicationPerformanceRunmodicationperformancemeasuresthetimerequiredtomodifyanexistingrun.TheModelExecutormustsupportmodifyingarunforaworkingmodelfromarunsetcontaining2000runsin60secondstosatisfythemodelexecutionperformancerequirements.Theprocedureformeasuringreactionmodicationperformanceis:1.Createatestmodelcontaining200reactions2.Writetestmodeltodisk3.Create100testparametersets5
4.Create100testinitialconditionsets5.CreateatestrunsetcontainingNruns6.Writetestrunsettodisk7.Readtestrunsetfromdisk8.Settheparameterforreaction200inparameterset100to09.Settheinitialconditionforspecies200ininitialconditionset100to010.WritetestrunsettodiskSteps7through10oftheprocedurearetimed.AsofMay1st,theJigCellRunManagerhasnotbeenrunforthisbenchmark.1.10ExperimentSetTestDataAnexperimentsetconsistsofexperiments.Eachexperimenthasthefollowingele-ments:ExperimentnameGloballyuniqueidentierExperimentalobservations.Theexperimentalvalueisatwovariabletimeseriescontaining4observations.Therstvariableofexperimentobservationiistheintegeriandthesecondvariableisanite,normalizedrealnumber.Experimentaltype.Thetypemustbecompatiblewiththetimeseriesformatoftheexperimentalvalue.ExperimentdescriptionThesizeofanexperimentsetisdeterminedbythenumberofexperiments.1.11ExperimentLoadingPerformanceExperimentloadingperformancemeasuresthesizeofexperimentsetssupportedbytheModelAnalyzer.Anexperimentsetissupportediftheloadingtimeislessthan300seconds.TheModelAnalyzermustsupportanexperimentsetcontaining10000experimentstosatisfythemodelanalysisperformancerequirements.Theprocedureformeasuringexperimentloadingperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.ReadtestexperimentsetfromdiskStep3oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadanexperimentsetcontaining10000experimentsislessthan5secondsusinglessthan64MBofmemoryand5011KBofdiskspace.6
1.12ExperimentEntryPerformanceExperimententryperformancemeasuresthetimerequiredtoaddanewexperimenttoanexperimentset.TheModelAnalyzermustsupportaddinganexperimenttoanex-perimentsetcontaining2000experimentsin120secondstosatisfythemodelanalysisperformancerequirements.Theprocedureformeasuringexperimententryperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.Readtestexperimentsetfromdisk4.AddexperimentN+15.WritetestexperimentsettodiskStep3through5oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoaddanewexperimenttoanexperimentsetcontaining2000experimentsislessthan3secondsusinglessthan64MBofmemoryand998KBofdiskspace.1.13TransformSetTestDataAtransformsetconsistsoftransformsandamappingfunctionfromexperimentnamestotransforms.Eachtransformhasthefollowingelements:TransformnameGloballyuniqueidentierTransformationprocedure.Thetransformationprocedureisafunctionalspeci-cationofwhatthetransformdoes.ParametersusedbythetransformationprocedureReferencetothetransformthatprovidesinputtothistransform,ifneededListoftransformscontainedinthistransformTheproceduredenedforeachnamedexperimentconsistsoftwotransformsandrequiresoneparameter.Thersttransformreceivedinputfromthesecondtransform;thesecondtransformdoesnotrequireinput.Neithertransformcontainsanytrans-forms.Theoutputoftheprocedureisatimeseriesofthesameformatasanexperimen-talobservation.Thesizeofatransformsetisdeterminedbythenumberofmappingsfromexperimentnamestotransforms.7
1.14TransformLoadingPerformanceTransformloadingperformancemeasuresthesizeoftransformsetssupportedbytheModelAnalyzer.Atransformsetissupportediftheloadingtimeislessthan300sec-onds.TheModelAnalyzermustsupportatransformsetfor10000experiments,ofwhichatleast500ofthetransformsaredifferent,tosatisfythemodelanalysisperfor-mancerequirements.Whenrunningthisbenchmark,allofthegeneratedtransformsaredifferent.Theprocedureformeasuringloadingloadingperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.CreateatesttransformsetcontainingNtransforms4.Writetesttransformsettodisk5.Readtestexperimentsetfromdisk6.ReadtesttransformsetfromdiskStep6oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadatransformsetfor10000experimentsislessthan25secondsusinglessthan256MBofmemoryand15971KBofdiskspace.1.15ObjectiveSetTestDataAnobjectivesetconsistsofobjectivesandamappingfunctionfromexperimentnamestoobjectives.Eachobjectivehasthefollowingelements:ObjectivenameGloballyuniqueidentierObjectiveprocedure.Theobjectiveprocedureisafunctionalspecicationofwhattheobjectivedoes.ParametersusedbytheobjectiveprocedureTheproceduredenedforeachnamedexperimentrequiresoneparameter.Thesizeofanobjectivesetisdeterminedbythenumberofmappingsfromexperimentnamestoobjectives.8
1.16ObjectiveFunctionLoadingPerformanceObjectivefuntionloadingperformancemeasuresthesizeofobjectivesetssupportedbytheModelAnalyzer.Anobjectivesetissupportediftheloadingtimeislessthan300seconds.TheModelAnalyzermustsupportanobjectivesetfor10000experiments,ofwhichatleast50oftheobjectivefunctionsaredifferent,tosatisfythemodelanal-ysisperformancerequirements.Whenrunningthisbenchmark,allofthegeneratedobjectivefunctionsaredifferent.Theprocedureformeasuringobjectivefunctionloadingperformanceis:1.CreateatestexperimentsetcontainingNexperiments2.Writetestexperimentsettodisk3.CreateatesttransformsetcontainingNtransforms4.Writetesttransformsettodisk5.CreateatestobjectivesetcontainingNobjectives6.Writetestobjectivesettodisk7.Readtestexperimentsetfromdisk8.Readtesttransformsetfromdisk9.ReadtestobjectivesetfromdiskStep9oftheprocedureistimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetoloadanobjectivesetfor10000experimentsislessthan10secondsusinglessthan256MBofmemoryand15169KBofdiskspace.1.17ExperimentCongurationPerformanceExperimentcongurationperformancemeasuresthetimerequiredtocongurethetransformandobjectivefunctionforanewexperiment.TheModelAnalyzermustsupportconguringanexperimentinanexperimentsetcontaining2000experimentsin300secondstosatisfythemodelanalysisperformancerequirements.Theprocedureformeasuringexperimentcongurationperformanceis:1.CreateatestexperimentsetcontainingN+1experiments2.Writetestexperimentsettodisk3.CreateatesttransformsetcontainingNtransforms4.Writetesttransformsettodisk5.CreateatestobjectivesetcontainingNobjectives9
6.Writetestobjectivesettodisk7.Readtestexperimentsetfromdisk8.Readtesttransformsetfromdisk9.CreatetransformN+110.AddtransformN+1toexperimentN+111.Writetesttransformsettodisk12.Readtestobjectivesetfromdisk13.CreateobjectiveN+114.AddobjectiveN+1toexperimentN+115.WritetestobjectivesettodiskSteps7through16oftheprocedurearetimed.AsofMay1st,theJigCellComparatorpassesthisbenchmark.Thetimetocon-gureanexperimentinanexperimentsetcontaining2000experimentsislessthan20secondsusinglessthan256MBofmemoryand5207KBofdiskspace.1.18SummaryofResultsAsofMay1st,6ofthe13benchmarkshavebeencompletedandrun.JigCellpasses5ofthebenchmarksandfails1ofthebenchmarks.#Time(s)Memory(MB)Disk(KB)Passes1.3626.28512-7206864.1a6.1b6.1c6.17.18.19.11.115.0032-4950111.122.8932-419981.1424.03128-187159711.169.94128-150151691.1719.4532-2565207Table3:BenchmarkResultsSummary01
FigureFigure:1:2BenchmarkBenchmark113.111.1
Voir icon more
Alternate Text