{"paragraphs":[{"text":"%md\n### Note\n\nPlease view the [README](https://github.com/eclipse/deeplearning4j-examples/blob/master/tutorials/README.md) to learn about installing, setting up dependencies, and importing notebooks in Zeppelin","user":"anonymous","dateUpdated":"2017-11-14T02:37:10+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Note</h3>\n<p>Please view the <a href=\"https://github.com/eclipse/deeplearning4j-examples/blob/master/tutorials/README.md\">README</a> to learn about installing, setting up dependencies, and importing notebooks in Zeppelin</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103095_971599260","id":"20171110-193224_343490486","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:37:10+0000","dateFinished":"2017-11-14T02:37:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:8661"},{"text":"%md\n\n### Background\n\nNeural network hyperparameters are parameters set prior to training. They include the learning rate, batch size, number of epochs, regularization, weight initialization, number of hidden layers, number of nodes, and etc. Unlike the weights and biases of the nodes of the neural network, they cannot be estimated directly using the data. Setting an optimal or near-optimal configuration of the hyperparameters can significantly affect neural network performance. Thus, time should be set aside to tune these hyperparameters.\n\nDeeplearning4j (DL4J) provides functionality to do exactly this task. Arbiter was created explicitly for tuning neural network models and is part of the DL4J suite of deep learning tools. In this tutorial, we will show an example of using Arbiter to tune the learning rate and the number of hidden nodes or layer size of a neural network model. We will use the MNIST dataset (images of handwritten digits) to train the neural network. ","user":"anonymous","dateUpdated":"2017-11-14T02:37:12+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Background</h3>\n<p>Neural network hyperparameters are parameters set prior to training. They include the learning rate, batch size, number of epochs, regularization, weight initialization, number of hidden layers, number of nodes, and etc. Unlike the weights and biases of the nodes of the neural network, they cannot be estimated directly using the data. Setting an optimal or near-optimal configuration of the hyperparameters can significantly affect neural network performance. Thus, time should be set aside to tune these hyperparameters.</p>\n<p>Deeplearning4j (DL4J) provides functionality to do exactly this task. Arbiter was created explicitly for tuning neural network models and is part of the DL4J suite of deep learning tools. In this tutorial, we will show an example of using Arbiter to tune the learning rate and the number of hidden nodes or layer size of a neural network model. We will use the MNIST dataset (images of handwritten digits) to train the neural network.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103114_1799846894","id":"20171110-192451_237014421","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:37:12+0000","dateFinished":"2017-11-14T02:37:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8662"},{"text":"%md\n\n### Imports","user":"anonymous","dateUpdated":"2017-11-14T02:37:14+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Imports</h3>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103116_-133482987","id":"20171110-193118_152075211","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:37:14+0000","dateFinished":"2017-11-14T02:37:14+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8663"},{"text":"import org.deeplearning4j.api.storage.StatsStorage\nimport org.deeplearning4j.arbiter.MultiLayerSpace\nimport org.deeplearning4j.arbiter.layers.DenseLayerSpace\nimport org.deeplearning4j.arbiter.layers.OutputLayerSpace\nimport org.deeplearning4j.arbiter.optimize.api.CandidateGenerator\nimport org.deeplearning4j.arbiter.optimize.api.OptimizationResult\nimport org.deeplearning4j.arbiter.optimize.api.ParameterSpace\nimport org.deeplearning4j.arbiter.optimize.api.data.DataProvider\nimport org.deeplearning4j.arbiter.data.MnistDataProvider\nimport org.deeplearning4j.arbiter.optimize.api.saving.ResultReference\nimport org.deeplearning4j.arbiter.optimize.api.saving.ResultSaver\nimport org.deeplearning4j.arbiter.optimize.api.score.ScoreFunction\nimport org.deeplearning4j.arbiter.optimize.api.termination.MaxCandidatesCondition\nimport org.deeplearning4j.arbiter.optimize.api.termination.MaxTimeCondition\nimport org.deeplearning4j.arbiter.optimize.api.termination.TerminationCondition\nimport org.deeplearning4j.arbiter.optimize.config.OptimizationConfiguration\nimport org.deeplearning4j.arbiter.optimize.generator.RandomSearchGenerator\nimport org.deeplearning4j.arbiter.optimize.parameter.continuous.ContinuousParameterSpace\nimport org.deeplearning4j.arbiter.optimize.parameter.integer.IntegerParameterSpace\nimport org.deeplearning4j.arbiter.optimize.runner.IOptimizationRunner\nimport org.deeplearning4j.arbiter.optimize.runner.LocalOptimizationRunner\nimport org.deeplearning4j.arbiter.saver.local.FileModelSaver\nimport org.deeplearning4j.arbiter.scoring.impl.TestSetAccuracyScoreFunction\nimport org.deeplearning4j.arbiter.task.MultiLayerNetworkTaskCreator\nimport org.deeplearning4j.datasets.iterator.MultipleEpochsIterator\nimport org.deeplearning4j.datasets.iterator.impl.MnistDataSetIterator\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.dataset.api.iterator.DataSetIterator\nimport org.nd4j.linalg.lossfunctions.LossFunctions\nimport org.nd4j.shade.jackson.annotation.JsonProperty\nimport org.nd4j.linalg.factory.Nd4j\nimport org.nd4j.linalg.cpu.nativecpu.CpuAffinityManager\n\n\nimport java.io.File\nimport java.io.IOException\nimport java.util.List\nimport java.util.Map\nimport java.util.concurrent.TimeUnit\n","user":"anonymous","dateUpdated":"2017-11-14T02:39:14+0000","config":{"tableHide":false,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"import org.deeplearning4j.api.storage.StatsStorage\nimport org.deeplearning4j.arbiter.MultiLayerSpace\nimport org.deeplearning4j.arbiter.layers.DenseLayerSpace\nimport org.deeplearning4j.arbiter.layers.OutputLayerSpace\nimport org.deeplearning4j.arbiter.optimize.api.CandidateGenerator\nimport org.deeplearning4j.arbiter.optimize.api.OptimizationResult\nimport org.deeplearning4j.arbiter.optimize.api.ParameterSpace\nimport org.deeplearning4j.arbiter.optimize.api.data.DataProvider\nimport org.deeplearning4j.arbiter.data.MnistDataProvider\nimport org.deeplearning4j.arbiter.optimize.api.saving.ResultReference\nimport org.deeplearning4j.arbiter.optimize.api.saving.ResultSaver\nimport org.deeplearning4j.arbiter.optimize.api.score.ScoreFunction\nimport org.deeplearning4j.arbiter.optimize.api.termination.MaxCandidatesCondition\nimport org.deeplearning4j.arbiter.optimize.api.termination.MaxTimeCondition\nimport org.deeplearning4j.arbiter.optimize.api.termination.TerminationCondition\nimport org.deeplearning4j.arbiter.optimize.config.OptimizationConfiguration\nimport org.deeplearning4j.arbiter.optimize.generator.RandomSearchGenerator\nimport org.deeplearning4j.arbiter.optimize.parameter.continuous.ContinuousParameterSpace\nimport org.deeplearning4j.arbiter.optimize.parameter.integer.IntegerParameterSpace\nimport org.deeplearning4j.arbiter.optimize.runner.IOptimizationRunner\nimport org.deeplearning4j.arbiter.optimize.runner.LocalOptimizationRunner\nimport org.deeplearning4j.arbiter.saver.local.FileModelSaver\nimport org.deeplearning4j.arbiter.scoring.impl.TestSetAccuracyScoreFunction\nimport org.deeplearning4j.arbiter.task.MultiLayerNetworkTaskCreator\nimport org.deeplearning4j.datasets.iterator.MultipleEpochsIterator\nimport org.deeplearning4j.datasets.iterator.impl.MnistDataSetIterator\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.dataset.api.iterator.DataSetIterator\nimport org.nd4j.linalg.lossfunctions.LossFunctions\nimport org.nd4j.shade.jackson.annotation.JsonProperty\nimport org.nd4j.linalg.factory.Nd4j\nimport org.nd4j.linalg.cpu.nativecpu.CpuAffinityManager\nimport java.io.File\nimport java.io.IOException\nimport java.util.List\nimport java.util.Map\nimport java.util.concurrent.TimeUnit\n"}]},"apps":[],"jobName":"paragraph_1510626103117_-596898894","id":"20171110-193241_134607717","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:39:14+0000","dateFinished":"2017-11-14T02:39:24+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8664"},{"text":"%md\n\nOur goal of this tutorial is to tune the learning rate and the layer size. We can start by setting up the parameter space of the learning rate and the layer size. We will consider values between 0.0001 and 0.1 for the learning rate and integer values between 16 and 256 for the layer size. \n\nNext, we set up a MultiLayerSpace, which is similar in structure to the MultiLayerNetwork class we've seen below. Here, we can set the hyperparameters of the neural network model. However, we can set the learning rate and the number of hidden nodes using the ParameterSpaces we've initialized before and not a set value like the other hyperparameters.\n\nLastly, we use the CandidateGenerator class to configure how candidate values of the learning rate and the layer size will be generated. In this tutorial, we will use random search; thus, values for the learning rate and the layer size will be generated uniformly within their ranges.","user":"anonymous","dateUpdated":"2017-11-14T02:27:10+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p>Our goal of this tutorial is to tune the learning rate and the layer size. We can start by setting up the parameter space of the learning rate and the layer size. We will consider values between 0.0001 and 0.1 for the learning rate and integer values between 16 and 256 for the layer size. </p>\n<p>Next, we set up a MultiLayerSpace, which is similar in structure to the MultiLayerNetwork class we&rsquo;ve seen below. Here, we can set the hyperparameters of the neural network model. However, we can set the learning rate and the number of hidden nodes using the ParameterSpaces we&rsquo;ve initialized before and not a set value like the other hyperparameters.</p>\n<p>Lastly, we use the CandidateGenerator class to configure how candidate values of the learning rate and the layer size will be generated. In this tutorial, we will use random search; thus, values for the learning rate and the layer size will be generated uniformly within their ranges.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103118_-1664482364","id":"20171110-193336_1672644219","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:27:12+0000","dateFinished":"2017-11-14T02:27:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8665"},{"text":"val learningRateHyperparam  = new ContinuousParameterSpace(0.0001, 0.1)\nval layerSizeHyperparam  = new IntegerParameterSpace(16,256)            \n\n\nval hyperparameterSpace  = new MultiLayerSpace.Builder()\n    //These next few options: fixed values for all models\n    .weightInit(WeightInit.XAVIER)\n    .regularization(true)\n    .l2(0.0001)\n    //Learning rate hyperparameter: search over different values, applied to all models\n    .learningRate(learningRateHyperparam)\n    .addLayer( new DenseLayerSpace.Builder()\n            //Fixed values for this layer:\n            .nIn(784)  //Fixed input: 28x28=784 pixels for MNIST\n            .activation(Activation.LEAKYRELU)\n            //One hyperparameter to infer: layer size\n            .nOut(layerSizeHyperparam)\n            .build())\n    .addLayer( new OutputLayerSpace.Builder()\n            .nOut(10)\n            .activation(Activation.SOFTMAX)\n            .lossFunction(LossFunctions.LossFunction.MCXENT)\n            .build())\n    .build()\n    \nval candidateGenerator = new RandomSearchGenerator(hyperparameterSpace, null)   \n\n","user":"anonymous","dateUpdated":"2017-11-14T02:39:21+0000","config":{"tableHide":false,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"learningRateHyperparam: org.deeplearning4j.arbiter.optimize.parameter.continuous.ContinuousParameterSpace = ContinuousParameterSpace(min=1.0E-4,max=0.1)\nlayerSizeHyperparam: org.deeplearning4j.arbiter.optimize.parameter.integer.IntegerParameterSpace = IntegerParameterSpace(min=16,max=256)\nhyperparameterSpace: org.deeplearning4j.arbiter.MultiLayerSpace =\nregularization: FixedValue(true)\nweightInit: FixedValue(XAVIER)\nlearningRate: ContinuousParameterSpace(min=1.0E-4,max=0.1)\nl2: FixedValue(1.0E-4)\nLayer config 0: (Number layers:FixedValue(1), duplicate: true), DenseLayerSpace(nIn: FixedValue(784), nOut: IntegerParameterSpace(min=16,max=256), activationFunction: ActivationParameterSpaceAdapter(activation=FixedValue(LEAKYRELU)))\nLayer config 1: (Number layers:FixedValue(1), duplicate: true), OutputLayerSpace(nOut: FixedValue(10), activationFunction: ActivationParameterSpaceAdapter(activation=FixedValue(SOFTMAX)))\nLayer config 0: (Number layers:FixedValue(1), duplicate: true), DenseLayerSpace(nIn: FixedValue(784), nOut: IntegerParameterSpace(min=16,max=256), activationFuncti...candidateGenerator: org.deeplearning4j.arbiter.optimize.generator.RandomSearchGenerator = RandomSearchGenerator\n"}]},"apps":[],"jobName":"paragraph_1510626103119_967228248","id":"20171110-200136_1726515359","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:39:24+0000","dateFinished":"2017-11-14T02:39:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8666"},{"text":"%md \n\nTo obtain the data, we will use the built-in MnistDataProvider class and use two training epochs or complete passes through the data and a batch size of 64 for training.","user":"anonymous","dateUpdated":"2017-11-14T02:27:18+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p>To obtain the data, we will use the built-in MnistDataProvider class and use two training epochs or complete passes through the data and a batch size of 64 for training.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103120_-2072142594","id":"20171110-200830_25205398","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:27:18+0000","dateFinished":"2017-11-14T02:27:18+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8667"},{"text":"val nTrainEpochs = 2\nval batchSize = 64\n\nval dataProvider = new MnistDataProvider(nTrainEpochs, batchSize)","user":"anonymous","dateUpdated":"2017-11-14T02:39:30+0000","config":{"editorSetting":{"language":"scala"},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"nTrainEpochs: Int = 2\nbatchSize: Int = 64\ndataProvider: org.deeplearning4j.arbiter.data.MnistDataProvider = MnistDataProvider(numEpochs=2, batchSize=64, rngSeed=-1419655116)\n"}]},"apps":[],"jobName":"paragraph_1510626103120_1501331519","id":"20171110-200622_1951663417","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:39:30+0000","dateFinished":"2017-11-14T02:39:32+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8668"},{"text":"%md \n\nWe've set how we are going to generate new values of the two hyperparameters we are considering but there still remains the question of how to evaluate them. We will use the accuracy score metric to evaluate different configurations of the hyperparameters so we initialize a TestSetAccuracyScoreFunction.","user":"anonymous","dateUpdated":"2017-11-14T02:27:22+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p>We&rsquo;ve set how we are going to generate new values of the two hyperparameters we are considering but there still remains the question of how to evaluate them. We will use the accuracy score metric to evaluate different configurations of the hyperparameters so we initialize a TestSetAccuracyScoreFunction.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103121_-1142868171","id":"20171110-200934_482187011","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:27:22+0000","dateFinished":"2017-11-14T02:27:22+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8669"},{"text":"val scoreFunction = new TestSetAccuracyScoreFunction()\n","user":"anonymous","dateUpdated":"2017-11-14T02:39:33+0000","config":{"tableHide":false,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"scoreFunction: org.deeplearning4j.arbiter.scoring.impl.TestSetAccuracyScoreFunction = TestSetAccuracyScoreFunction()\n"}]},"apps":[],"jobName":"paragraph_1510626103121_-379721325","id":"20171110-201010_344510588","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:39:33+0000","dateFinished":"2017-11-14T02:39:34+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8670"},{"text":"%md\n\nWe also want to set how long the hyperparameter search will last. There are infinite configurations of the learning rate and hidden layer size, since the learning rate space is continuous. Thus, we set a termination condition of 15 minutes.","user":"anonymous","dateUpdated":"2017-11-14T02:27:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p>We also want to set how long the hyperparameter search will last. There are infinite configurations of the learning rate and hidden layer size, since the learning rate space is continuous. Thus, we set a termination condition of 15 minutes.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103122_-895159118","id":"20171110-201242_1188027252","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:27:25+0000","dateFinished":"2017-11-14T02:27:25+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8671"},{"text":"val terminationConditions = { new MaxTimeCondition(15, TimeUnit.MINUTES)}","user":"anonymous","dateUpdated":"2017-11-14T02:39:36+0000","config":{"tableHide":false,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"terminationConditions: org.deeplearning4j.arbiter.optimize.api.termination.MaxTimeCondition = MaxTimeCondition(15,MINUTES\")\n"}]},"apps":[],"jobName":"paragraph_1510626103122_719761601","id":"20171110-201204_1138596401","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:39:36+0000","dateFinished":"2017-11-14T02:39:37+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8672"},{"text":"%md\nTo save the best model, we can set the directory to save it in.","user":"anonymous","dateUpdated":"2017-11-14T02:27:28+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p>To save the best model, we can set the directory to save it in.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103122_-1771639992","id":"20171110-201228_1675790178","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:27:28+0000","dateFinished":"2017-11-14T02:27:28+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8673"},{"text":"val baseSaveDirectory = \"arbiterExample/\"\nval f = new File(baseSaveDirectory)\nif(f.exists()) f.delete()\nf.mkdir()\nval modelSaver = new FileModelSaver(baseSaveDirectory)","user":"anonymous","dateUpdated":"2017-11-14T02:39:38+0000","config":{"tableHide":false,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"baseSaveDirectory: String = arbiterExample/\nf: java.io.File = arbiterExample\nres0: AnyVal = true\nres1: Boolean = true\nmodelSaver: org.deeplearning4j.arbiter.saver.local.FileModelSaver = FileModelSaver(path=arbiterExample/)\n"}]},"apps":[],"jobName":"paragraph_1510626103123_935521002","id":"20171110-201446_559608063","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:39:38+0000","dateFinished":"2017-11-14T02:39:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8674"},{"text":"%md\n\nGiven all the configurations we have already set, we need to put them together using the OptimizationConfiguration. To execute the hyperparameter search, we initialize an IOptimizaitonRunner using the OptimizationConfiguration. ","user":"anonymous","dateUpdated":"2017-11-14T02:27:36+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p>Given all the configurations we have already set, we need to put them together using the OptimizationConfiguration. To execute the hyperparameter search, we initialize an IOptimizaitonRunner using the OptimizationConfiguration.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103123_2073716403","id":"20171110-201524_1984648792","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:27:36+0000","dateFinished":"2017-11-14T02:27:36+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8675"},{"text":"val configuration = new OptimizationConfiguration.Builder()\n                .candidateGenerator(candidateGenerator)\n                .dataProvider(dataProvider)\n                .modelSaver(modelSaver)\n                .scoreFunction(scoreFunction)\n                .terminationConditions(terminationConditions)\n                .build()\n\nval runner = new LocalOptimizationRunner(configuration, new MultiLayerNetworkTaskCreator())\n\n//Start the hyperparameter optimization\n\nrunner.execute()\n\n","user":"anonymous","dateUpdated":"2017-11-14T02:39:41+0000","config":{"editorSetting":{"language":"scala"},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"configuration: org.deeplearning4j.arbiter.optimize.config.OptimizationConfiguration = OptimizationConfiguration(dataProvider=MnistDataProvider(numEpochs=2, batchSize=64, rngSeed=-1419655116), candidateGenerator=RandomSearchGenerator, resultSaver=FileModelSaver(path=arbiterExample/), scoreFunction=TestSetAccuracyScoreFunction(), terminationConditions=[MaxTimeCondition(15,MINUTES\")], rngSeed=null, executionStartTime=0)\nrunner: org.deeplearning4j.arbiter.optimize.runner.LocalOptimizationRunner = org.deeplearning4j.arbiter.optimize.runner.LocalOptimizationRunner@21e91304\n"}]},"apps":[],"jobName":"paragraph_1510626103124_-1658909985","id":"20171110-201515_253387615","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T02:39:42+0000","dateFinished":"2017-11-14T02:54:46+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8676"},{"text":"%md \nLastly, we can print out the details of the best model and the results.\n","dateUpdated":"2017-11-14T02:44:28+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<p>Lastly, we can print out the details of the best model and the results.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1510626103124_482259524","id":"20171110-201557_1875435291","dateCreated":"2017-11-14T02:21:43+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8677","user":"anonymous","dateFinished":"2017-11-14T02:55:04+0000","dateStarted":"2017-11-14T02:54:46+0000"},{"text":"val s = \"Best score: \" + runner.bestScore() + \"\\n\" + \"Index of model with best score: \" + runner.bestScoreCandidateIndex() + \"\\n\" + \"Number of configurations evaluated: \" + runner.numCandidatesCompleted() + \"\\n\"\nprintln(s)\n\n\n//Get all results, and print out details of the best result:\nval indexOfBestResult = runner.bestScoreCandidateIndex()\nval allResults = runner.getResults()\n\nval bestResult = allResults.get(indexOfBestResult).getResult()\nval bestModel = bestResult.getResult().asInstanceOf[MultiLayerNetwork]\n\n\nprintln(\"\\n\\nConfiguration of best model:\\n\")\nprintln(bestModel.getLayerWiseConfigurations().toJson())","user":"anonymous","dateUpdated":"2017-11-14T03:10:20+0000","config":{"tableHide":false,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"s: String =\n\"Best score: 0.9607\nIndex of model with best score: 17\nNumber of configurations evaluated: 70\n\"\nBest score: 0.9607\nIndex of model with best score: 17\nNumber of configurations evaluated: 70\n\nindexOfBestResult: Int = 17\nallResults: java.util.List[org.deeplearning4j.arbiter.optimize.api.saving.ResultReference] = [LocalFileNetResultReference(/usr/zeppelin/arbiterExample/0), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/1), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/2), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/3), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/4), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/5), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/6), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/7), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/8), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/9), LocalFileNetResultReference(/usr/zeppelin/arbiterExample/10), LocalFileNetResultReference(/us...bestResult: org.deeplearning4j.arbiter.optimize.api.OptimizationResult =\nOptimizationResult(candidate=Candidate(value=DL4JConfiguration(multiLayerConfiguration={\n  \"backprop\" : true,\n  \"backpropType\" : \"Standard\",\n  \"cacheMode\" : \"NONE\",\n  \"confs\" : [ {\n    \"cacheMode\" : \"NONE\",\n    \"iterationCount\" : 0,\n    \"l1ByParam\" : {\n      \"b\" : 0.0,\n      \"W\" : 0.0\n    },\n    \"l2ByParam\" : {\n      \"b\" : 0.0,\n      \"W\" : 1.0E-4\n    },\n    \"layer\" : {\n      \"dense\" : {\n        \"activationFn\" : {\n          \"LReLU\" : {\n            \"alpha\" : 0.01\n          }\n        },\n        \"adamMeanDecay\" : \"NaN\",\n        \"adamVarDecay\" : \"NaN\",\n        \"biasInit\" : 0.0,\n        \"biasLearningRate\" : 0.09608075748379281,\n        \"dist\" : null,\n        \"dropOut\" : 0.0,\n        \"epsilon\" : \"NaN\",\n        \"gradientNo...bestModel: org.deeplearning4j.nn.multilayer.MultiLayerNetwork = org.deeplearning4j.nn.multilayer.MultiLayerNetwork@65a2e660\n\n\nConfiguration of best model:\n\n{\n  \"backprop\" : true,\n  \"backpropType\" : \"Standard\",\n  \"cacheMode\" : \"NONE\",\n  \"confs\" : [ {\n    \"cacheMode\" : \"NONE\",\n    \"iterationCount\" : 0,\n    \"l1ByParam\" : {\n      \"b\" : 0.0,\n      \"W\" : 0.0\n    },\n    \"l2ByParam\" : {\n      \"b\" : 0.0,\n      \"W\" : 1.0E-4\n    },\n    \"layer\" : {\n      \"dense\" : {\n        \"activationFn\" : {\n          \"LReLU\" : {\n            \"alpha\" : 0.01\n          }\n        },\n        \"adamMeanDecay\" : \"NaN\",\n        \"adamVarDecay\" : \"NaN\",\n        \"biasInit\" : 0.0,\n        \"biasLearningRate\" : 0.09608075748379281,\n        \"dist\" : null,\n        \"dropOut\" : 0.0,\n        \"epsilon\" : \"NaN\",\n        \"gradientNormalization\" : \"None\",\n        \"gradientNormalizationThreshold\" : 1.0,\n        \"iupdater\" : {\n          \"@class\" : \"org.nd4j.linalg.learning.config.Sgd\",\n          \"learningRate\" : 0.09608075748379281\n        },\n        \"l1\" : 0.0,\n        \"l1Bias\" : 0.0,\n        \"l2\" : 1.0E-4,\n        \"l2Bias\" : 0.0,\n        \"layerName\" : \"layer0\",\n        \"learningRate\" : 0.09608075748379281,\n        \"learningRateSchedule\" : null,\n        \"momentum\" : \"NaN\",\n        \"momentumSchedule\" : null,\n        \"nin\" : 784,\n        \"nout\" : 250,\n        \"rho\" : \"NaN\",\n        \"rmsDecay\" : \"NaN\",\n        \"updater\" : \"SGD\",\n        \"weightInit\" : \"XAVIER\"\n      }\n    },\n    \"leakyreluAlpha\" : 0.0,\n    \"learningRateByParam\" : {\n      \"b\" : 0.09608075748379281,\n      \"W\" : 0.09608075748379281\n    },\n    \"learningRatePolicy\" : \"None\",\n    \"lrPolicyDecayRate\" : \"NaN\",\n    \"lrPolicyPower\" : \"NaN\",\n    \"lrPolicySteps\" : \"NaN\",\n    \"maxNumLineSearchIterations\" : 5,\n    \"miniBatch\" : true,\n    \"minimize\" : true,\n    \"numIterations\" : 1,\n    \"optimizationAlgo\" : \"STOCHASTIC_GRADIENT_DESCENT\",\n    \"pretrain\" : false,\n    \"seed\" : 1510627477634,\n    \"stepFunction\" : null,\n    \"useDropConnect\" : false,\n    \"useRegularization\" : true,\n    \"variables\" : [ \"W\", \"b\" ]\n  }, {\n    \"cacheMode\" : \"NONE\",\n    \"iterationCount\" : 0,\n    \"l1ByParam\" : {\n      \"b\" : 0.0,\n      \"W\" : 0.0\n    },\n    \"l2ByParam\" : {\n      \"b\" : 0.0,\n      \"W\" : 1.0E-4\n    },\n    \"layer\" : {\n      \"output\" : {\n        \"activationFn\" : {\n          \"Softmax\" : { }\n        },\n        \"adamMeanDecay\" : \"NaN\",\n        \"adamVarDecay\" : \"NaN\",\n        \"biasInit\" : 0.0,\n        \"biasLearningRate\" : 0.09608075748379281,\n        \"dist\" : null,\n        \"dropOut\" : 0.0,\n        \"epsilon\" : \"NaN\",\n        \"gradientNormalization\" : \"None\",\n        \"gradientNormalizationThreshold\" : 1.0,\n        \"iupdater\" : {\n          \"@class\" : \"org.nd4j.linalg.learning.config.Sgd\",\n          \"learningRate\" : 0.09608075748379281\n        },\n        \"l1\" : 0.0,\n        \"l1Bias\" : 0.0,\n        \"l2\" : 1.0E-4,\n        \"l2Bias\" : 0.0,\n        \"layerName\" : \"layer1\",\n        \"learningRate\" : 0.09608075748379281,\n        \"learningRateSchedule\" : null,\n        \"lossFn\" : {\n          \"MCXENT\" : {\n            \"softmaxClipEps\" : 1.0E-10\n          }\n        },\n        \"lossFunction\" : \"MCXENT\",\n        \"momentum\" : \"NaN\",\n        \"momentumSchedule\" : null,\n        \"nin\" : 250,\n        \"nout\" : 10,\n        \"rho\" : \"NaN\",\n        \"rmsDecay\" : \"NaN\",\n        \"updater\" : \"SGD\",\n        \"weightInit\" : \"XAVIER\"\n      }\n    },\n    \"leakyreluAlpha\" : 0.0,\n    \"learningRateByParam\" : {\n      \"b\" : 0.09608075748379281,\n      \"W\" : 0.09608075748379281\n    },\n    \"learningRatePolicy\" : \"None\",\n    \"lrPolicyDecayRate\" : \"NaN\",\n    \"lrPolicyPower\" : \"NaN\",\n    \"lrPolicySteps\" : \"NaN\",\n    \"maxNumLineSearchIterations\" : 5,\n    \"miniBatch\" : true,\n    \"minimize\" : true,\n    \"numIterations\" : 1,\n    \"optimizationAlgo\" : \"STOCHASTIC_GRADIENT_DESCENT\",\n    \"pretrain\" : false,\n    \"seed\" : 1510627477634,\n    \"stepFunction\" : null,\n    \"useDropConnect\" : false,\n    \"useRegularization\" : true,\n    \"variables\" : [ \"W\", \"b\" ]\n  } ],\n  \"inferenceWorkspaceMode\" : \"SEPARATE\",\n  \"inputPreProcessors\" : { },\n  \"iterationCount\" : 2814,\n  \"pretrain\" : false,\n  \"tbpttBackLength\" : 20,\n  \"tbpttFwdLength\" : 20,\n  \"trainingWorkspaceMode\" : \"NONE\"\n}\n"}]},"apps":[],"jobName":"paragraph_1510626103125_-1728714426","id":"20171110-201710_2013958491","dateCreated":"2017-11-14T02:21:43+0000","dateStarted":"2017-11-14T03:10:20+0000","dateFinished":"2017-11-14T03:10:22+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:8678"},{"dateUpdated":"2017-11-14T03:07:14+0000","config":{"editorSetting":{"language":"scala"},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1510626103125_-1585885365","id":"20171114-004959_823380322","dateCreated":"2017-11-14T02:21:43+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:8679","text":""}],"name":"Hyperparameter Optimization","id":"2CXJSDZ5P","angularObjects":{"2CZ7EPFUB:shared_process":[],"2CY7EK3T1:shared_process":[],"2CYEM1VFC:shared_process":[],"2D1KS2US3:shared_process":[],"2D13K563R:shared_process":[],"2CYZYPWPM:shared_process":[],"2CZU13TNZ:shared_process":[],"2CXYB8YE8:shared_process":[],"2CXX6PEK1:shared_process":[],"2CYZYN2RE:shared_process":[],"2CYYY8FND:shared_process":[],"2CZVCG3RN:shared_process":[],"2CY8JTYZJ:shared_process":[],"2CZDXVTTZ:shared_process":[],"2CXPC41YC:shared_process":[],"2D1BYTVAP:shared_process":[],"2CX1VR83V:shared_process":[],"2CZ1CJR7R:shared_process":[],"2CWXB18PA:shared_process":[],"2CZRA1GGD:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}