{"paragraphs":[{"text":"%md\n### Note\n\nView the README.md [here](https://github.com/eclipse/deeplearning4j-examples/blob/master/tutorials/README.md) to learn about installing, setting up dependencies and importing notebooks in Zeppelin","dateUpdated":"2017-10-30T09:13:13-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Note</h3>\n<p>View the README.md <a href=\"https://github.com/eclipse/deeplearning4j-examples/blob/master/tutorials/README.md\">here</a> to learn about installing, setting up dependencies and importing notebooks in Zeppelin</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1509379993874_-1529791372","id":"20171029-230537_325264422","dateCreated":"2017-10-30T09:13:13-0700","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:429"},{"text":"%md\n\n### Background\n\nRecurrent neural networks (RNN's) are used when the input is sequential in nature. Typically RNN's are much more effective than regular feed forward neural networks for sequential data because they can keep track of dependencies in the data over multiple time steps. This is possible because the output of a RNN at a time step depends on the current input and the output of the previous time step. \n\nRNN's can also be applied to situations where the input is sequential but the output isn't. In these cases the output of the last time step of the RNN is typically taken as the output for the overall observation. For classification, the output of the last time step will be the predicted class label for the observation. \n\nIn this notebook we will show how to build a RNN using the MultiLayerNetwork class of deeplearning4j (DL4J). This tutorial will focus on applying a RNN for a classification task. We will be using the MNIST data, which is a dataset that consists of images of handwritten digits, as the input for the RNN. Although the MNIST data isn't time series in nature, we can interpret it as such since there are 784 inputs. Thus, each observation or image will be interpreted to have 784 time steps consisting of one scalar value for a pixel. Note that we use a RNN for this task for purely pedagogical reasons. In practice, convolutional neural networks (CNN's) are better suited for image classification tasks. \n\n","user":"anonymous","dateUpdated":"2017-10-30T09:13:25-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Background</h3>\n<p>Recurrent neural networks (RNN&rsquo;s) are used when the input is sequential in nature. Typically RNN&rsquo;s are much more effective than regular feed forward neural networks for sequential data because they can keep track of dependencies in the data over multiple time steps. This is possible because the output of a RNN at a time step depends on the current input and the output of the previous time step. </p>\n<p>RNN&rsquo;s can also be applied to situations where the input is sequential but the output isn&rsquo;t. In these cases the output of the last time step of the RNN is typically taken as the output for the overall observation. For classification, the output of the last time step will be the predicted class label for the observation. </p>\n<p>In this notebook we will show how to build a RNN using the MultiLayerNetwork class of deeplearning4j (DL4J). This tutorial will focus on applying a RNN for a classification task. We will be using the MNIST data, which is a dataset that consists of images of handwritten digits, as the input for the RNN. Although the MNIST data isn&rsquo;t time series in nature, we can interpret it as such since there are 784 inputs. Thus, each observation or image will be interpreted to have 784 time steps consisting of one scalar value for a pixel. Note that we use a RNN for this task for purely pedagogical reasons. In practice, convolutional neural networks (CNN&rsquo;s) are better suited for image classification tasks.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1509379993880_-1533638861","id":"20171029-230912_1114990421","dateCreated":"2017-10-30T09:13:13-0700","dateStarted":"2017-10-30T09:13:25-0700","dateFinished":"2017-10-30T09:13:26-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:430"},{"text":"%md\n### Imports","dateUpdated":"2017-10-30T09:13:13-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Imports</h3>\n</div>"}]},"apps":[],"jobName":"paragraph_1509379993880_-1533638861","id":"20171029-231007_1771351264","dateCreated":"2017-10-30T09:13:13-0700","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:431"},{"text":"import org.deeplearning4j.eval.Evaluation\nimport org.deeplearning4j.nn.api.OptimizationAlgorithm\nimport org.deeplearning4j.nn.conf.MultiLayerConfiguration\nimport org.deeplearning4j.nn.conf.NeuralNetConfiguration\nimport org.deeplearning4j.nn.conf.Updater\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.deeplearning4j.nn.conf.layers.{DenseLayer, GravesLSTM, OutputLayer, RnnOutputLayer}\nimport org.deeplearning4j.nn.conf.distribution.UniformDistribution\nimport org.deeplearning4j.nn.conf.layers.GravesLSTM\nimport org.deeplearning4j.nn.conf.layers.RnnOutputLayer\nimport org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator\nimport org.deeplearning4j.optimize.listeners.ScoreIterationListener\n\nimport org.datavec.api.split.NumberedFileInputSplit\nimport org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader\n\nimport org.nd4j.linalg.dataset.DataSet\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction\nimport org.nd4j.linalg.api.ndarray.INDArray\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.dataset.api.iterator.DataSetIterator\n\nimport org.slf4j.Logger\nimport org.slf4j.LoggerFactory\nimport org.apache.commons.io.IOUtils\n\nimport java.nio.charset.Charset\nimport java.util.Random\nimport java.net.URL","user":"anonymous","dateUpdated":"2017-11-04T15:19:04-0700","config":{"tableHide":false,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"import org.deeplearning4j.eval.Evaluation\nimport org.deeplearning4j.nn.api.OptimizationAlgorithm\nimport org.deeplearning4j.nn.conf.MultiLayerConfiguration\nimport org.deeplearning4j.nn.conf.NeuralNetConfiguration\nimport org.deeplearning4j.nn.conf.Updater\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.deeplearning4j.nn.conf.layers.{DenseLayer, GravesLSTM, OutputLayer, RnnOutputLayer}\nimport org.deeplearning4j.nn.conf.distribution.UniformDistribution\nimport org.deeplearning4j.nn.conf.layers.GravesLSTM\nimport org.deeplearning4j.nn.conf.layers.RnnOutputLayer\nimport org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator\nimport org.deeplearning4j.optimize.listeners.ScoreIterationListener\nimport org.datavec.api.split.NumberedFileInputSplit\nimport org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader\nimport org.nd4j.linalg.dataset.DataSet\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction\nimport org.nd4j.linalg.api.ndarray.INDArray\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.dataset.api.iterator.DataSetIterator\nimport org.slf4j.Logger\nimport org.slf4j.LoggerFactory\nimport org.apache.commons.io.IOUtils\nimport java.nio.charset.Charset\nimport java.util.Random\nimport java.net.URL\n"}]},"apps":[],"jobName":"paragraph_1509379993881_-1534023610","id":"20171029-231320_1088615359","dateCreated":"2017-10-30T09:13:13-0700","dateStarted":"2017-11-04T15:19:04-0700","dateFinished":"2017-11-04T15:19:08-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:432"},{"text":"%md\n\n### Download the dataset\n\nUCI has a number of datasets available for machine learning, make sure you have enough space on your local disk. The UCI synthetic control dataset can be found at [http://archive.ics.uci.edu/ml/datasets/synthetic+control+chart+time+series](http://archive.ics.uci.edu/ml/datasets/synthetic+control+chart+time+series). The code below will check if the data already exists and download the file.","user":"anonymous","dateUpdated":"2017-11-04T14:36:25-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Download the dataset</h3>\n<p>UCI has a number of datasets available for machine learning, make sure you have enough space on your local disk. The UCI synthetic control dataset can be found at <a href=\"http://archive.ics.uci.edu/ml/datasets/synthetic+control+chart+time+series\">http://archive.ics.uci.edu/ml/datasets/synthetic+control+chart+time+series</a>. The code below will check if the data already exists and download the file.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1509831304842_-1096828513","id":"20171104-143504_114487850","dateCreated":"2017-11-04T14:35:04-0700","dateStarted":"2017-11-04T14:36:25-0700","dateFinished":"2017-11-04T14:36:25-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:433"},{"text":"val dataPath = new File(cache, \"/uci_synthetic_control/\")\n\nif(!dataPath.exists()) {\n    val url = \"https://archive.ics.uci.edu/ml/machine-learning-databases/synthetic_control-mld/synthetic_control.data\"\n    println(\"Downloading file...\")\n    val data = IOUtils.toString(new URL(url), Charset.defaultCharset())\n    val lines = data.split(\"\\n\")\n\n    var lineCount = 0;\n    var index = 0\n\n    val linesList = scala.collection.mutable.ListBuffer.empty[String]\n    println(\"Extracting file...\")\n\n    for (line <- lines) {\n        val count = new java.lang.Integer(lineCount / 100)\n        var newLine: String = null\n        newLine = line.replaceAll(\"\\\\s+\", \", \" + count.toString() + \"\\n\")\n        newLine = line + \", \" + count.toString()\n        linesList.add(newLine)\n        lineCount += 1\n    }\n    util.Random.shuffle(linesList)\n\n    for (line <- linesList) {\n        val outPath = new File(dataPath, index + \".csv\")\n        FileUtils.writeStringToFile(outPath, line, Charset.defaultCharset())\n        index += 1\n    }\n    println(\"Done.\")\n} else {\n    println(\"File already exists.\")\n}","user":"anonymous","dateUpdated":"2017-11-04T15:03:04-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala"},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"dataPath: java.io.File = /home/justin/.deeplearning4j/uci_synthetic_control\nFile already exists.\n"}]},"apps":[],"jobName":"paragraph_1509830845426_-754256973","id":"20171104-142725_182654854","dateCreated":"2017-11-04T14:27:25-0700","dateStarted":"2017-11-04T15:03:04-0700","dateFinished":"2017-11-04T15:03:04-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:434"},{"text":"%md\n\n### Iterating from disk\n\nNow that we've saved our dataset to a CSV sequence format, we need to set up a `CSVSequenceRecordReader` and iterator that will read our saved sequences and feed them to our network. If you have already saved your data to disk, you can run this code block (and remaining code blocks) as much as you want without preprocessing the dataset again. Convenient!","user":"anonymous","dateUpdated":"2017-11-04T15:01:36-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Iterating from disk</h3>\n<p>Now that we&rsquo;ve saved our dataset to a CSV sequence format, we need to set up a <code>CSVSequenceRecordReader</code> and iterator that will read our saved sequences and feed them to our network. If you have already saved your data to disk, you can run this code block (and remaining code blocks) as much as you want without preprocessing the dataset again. Convenient!</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1509832464846_1910733676","id":"20171104-145424_2009227852","dateCreated":"2017-11-04T14:54:24-0700","dateStarted":"2017-11-04T15:01:36-0700","dateFinished":"2017-11-04T15:01:36-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:435"},{"text":"val batchSize = 128\nval numLabelClasses = 6\n\n// training data\nval trainRR = new CSVSequenceRecordReader(0, \", \")\ntrainRR.initialize(new NumberedFileInputSplit(dataPath.getAbsolutePath() + \"/%d.csv\", 0, 449))\nval trainIter = new SequenceRecordReaderDataSetIterator(trainRR, batchSize, numLabelClasses, 1)\n\n// testing data\nval testRR = new CSVSequenceRecordReader(0, \", \")\ntestRR.initialize(new NumberedFileInputSplit(dataPath.getAbsolutePath() + \"/%d.csv\", 450, 599))\nval testIter = new SequenceRecordReaderDataSetIterator(testRR, batchSize, numLabelClasses, 1)","user":"anonymous","dateUpdated":"2017-11-04T15:03:13-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"scala"},"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"batchSize: Int = 128\nnumLabelClasses: Int = 6\ntrainRR: org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader = org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader@26cfecb3\ntrainIter: org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator = org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator@59d132c5\ntestRR: org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader = org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader@24c47b9c\ntestIter: org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator = org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator@3e378717\n"}]},"apps":[],"jobName":"paragraph_1509832376757_-1054280910","id":"20171104-145256_1072151754","dateCreated":"2017-11-04T14:52:56-0700","dateStarted":"2017-11-04T15:03:13-0700","dateFinished":"2017-11-04T15:03:14-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:436"},{"text":"%md\n### Configuring a RNN for Classification\nOnce everything needed is imported we can jump into the code. To build the neural network, we can use a set up like what is shown below. Because there are 784 timesteps and 10 class labels, nIn is set to 784 and nOut is set to 10 in the MultiLayerNetwork configuration. \n","dateUpdated":"2017-10-30T09:13:13-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Configuring a RNN for Classification</h3>\n<p>Once everything needed is imported we can jump into the code. To build the neural network, we can use a set up like what is shown below. Because there are 784 timesteps and 10 class labels, nIn is set to 784 and nOut is set to 10 in the MultiLayerNetwork configuration.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1509379993881_-1534023610","id":"20171029-231327_1470602188","dateCreated":"2017-10-30T09:13:13-0700","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:437"},{"text":"val conf = new NeuralNetConfiguration.Builder()\n    .seed(123)    //Random number generator seed for improved repeatability. Optional.\n    .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)\n    .iterations(1)\n    .weightInit(WeightInit.XAVIER)\n    .updater(Updater.NESTEROVS)\n    .learningRate(0.005)\n    .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue)  //Not always required, but helps with this data set\n    .gradientNormalizationThreshold(0.5)\n    .list()\n    .layer(0, new GravesLSTM.Builder().activation(Activation.TANH).nIn(1).nOut(10).build())\n    .layer(1, new RnnOutputLayer.Builder(LossFunctions.LossFunction.MCXENT)\n            .activation(Activation.SOFTMAX).nIn(10).nOut(numLabelClasses).build())\n    .pretrain(false).backprop(true).build();\n\nval model: MultiLayerNetwork = new MultiLayerNetwork(conf)\nmodel.setListeners(new ScoreIterationListener(20))","user":"anonymous","dateUpdated":"2017-11-04T15:18:42-0700","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":false,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"conf: org.deeplearning4j.nn.conf.MultiLayerConfiguration =\n{\n  \"backprop\" : true,\n  \"backpropType\" : \"Standard\",\n  \"cacheMode\" : \"NONE\",\n  \"confs\" : [ {\n    \"cacheMode\" : \"NONE\",\n    \"iterationCount\" : 0,\n    \"l1ByParam\" : { },\n    \"l2ByParam\" : { },\n    \"layer\" : {\n      \"gravesLSTM\" : {\n        \"activationFn\" : {\n          \"TanH\" : { }\n        },\n        \"adamMeanDecay\" : \"NaN\",\n        \"adamVarDecay\" : \"NaN\",\n        \"biasInit\" : 0.0,\n        \"biasLearningRate\" : 0.005,\n        \"dist\" : null,\n        \"dropOut\" : 0.0,\n        \"epsilon\" : \"NaN\",\n        \"forgetGateBiasInit\" : 1.0,\n        \"gateActivationFn\" : {\n          \"Sigmoid\" : { }\n        },\n        \"gradientNormalization\" : \"ClipElementWiseAbsoluteValue\",\n        \"gradientNormalizationThreshold\" : 0.5,\n        \"iupdater\" : {\n   ...model: org.deeplearning4j.nn.multilayer.MultiLayerNetwork = org.deeplearning4j.nn.multilayer.MultiLayerNetwork@7bdc18cd\n"}]},"apps":[],"jobName":"paragraph_1509379993882_-1532869364","id":"20171029-231542_1864385323","dateCreated":"2017-10-30T09:13:13-0700","dateStarted":"2017-11-04T15:12:24-0700","dateFinished":"2017-11-04T15:12:26-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:438"},{"text":"%md\n### Training the classifier\n\nTo train the model, pass the training iterator to the model's `fit()` method. We can use a loop to train the model using a prespecified number of epochs or passes through the training data. \n","user":"anonymous","dateUpdated":"2017-11-04T15:04:39-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Training the classifier</h3>\n<p>To train the model, pass the training iterator to the model&rsquo;s <code>fit()</code> method. We can use a loop to train the model using a prespecified number of epochs or passes through the training data.</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1509379993884_-1535177857","id":"20171029-231638_317062042","dateCreated":"2017-10-30T09:13:13-0700","dateStarted":"2017-11-04T15:04:39-0700","dateFinished":"2017-11-04T15:04:39-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:439"},{"text":"val numEpochs = 1\n(1 to numEpochs).foreach(_ => model.fit(trainIter) )","user":"anonymous","dateUpdated":"2017-11-04T15:19:30-0700","config":{"editorSetting":{"language":"scala"},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":"org.apache.thrift.transport.TTransportException","apps":[],"jobName":"paragraph_1509379993884_-1535177857","id":"20171029-231649_668010364","dateCreated":"2017-10-30T09:13:13-0700","dateStarted":"2017-11-04T15:19:30-0700","dateFinished":"2017-11-04T15:31:15-0700","status":"ERROR","errorMessage":"org.apache.thrift.transport.TTransportException\n\tat org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)\n\tat org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)\n\tat org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)\n\tat org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)\n\tat org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)\n\tat org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)\n\tat org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_interpret(RemoteInterpreterService.java:266)\n\tat org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.interpret(RemoteInterpreterService.java:250)\n\tat org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:373)\n\tat org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97)\n\tat org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:406)\n\tat org.apache.zeppelin.scheduler.Job.run(Job.java:175)\n\tat org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\n","progressUpdateIntervalMs":500,"$$hashKey":"object:440"},{"text":"%md\n### Model Evaluation\nOnce training is complete we only a couple lines of code to evaluate the model on a test set. Using a test set to evaluate the model typically needs to be done in order to avoid overfitting on the training data. If we overfit on the training data, we have essentially fit to the noise in the data. \n\nAn `Evaluation` class has more built-in methods if you need to extract a confusion matrix, and other tools are also available for calculating the Area Under Curve (AUC).","user":"anonymous","dateUpdated":"2017-11-04T15:06:50-0700","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","fontSize":9,"editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>Model Evaluation</h3>\n<p>Once training is complete we only a couple lines of code to evaluate the model on a test set. Using a test set to evaluate the model typically needs to be done in order to avoid overfitting on the training data. If we overfit on the training data, we have essentially fit to the noise in the data. </p>\n<p>An <code>Evaluation</code> class has more built-in methods if you need to extract a confusion matrix, and other tools are also available for calculating the Area Under Curve (AUC).</p>\n</div>"}]},"apps":[],"jobName":"paragraph_1509379993885_-1535562606","id":"20171029-231657_433209818","dateCreated":"2017-10-30T09:13:13-0700","dateStarted":"2017-11-04T15:06:50-0700","dateFinished":"2017-11-04T15:06:50-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:441"},{"text":"val evaluation = model.evaluate(testIter)\n\n// print the basic statistics about the trained classifier\nprintln(\"Accuracy: \"+evaluation.accuracy())\nprintln(\"Precision: \"+evaluation.precision())\nprintln(\"Recall: \"+evaluation.recall())","user":"anonymous","dateUpdated":"2017-11-04T15:18:32-0700","config":{"tableHide":true,"editorSetting":{"language":"scala","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/scala","fontSize":9,"editorHide":false,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"ERROR","msg":[{"type":"TEXT","data":"java.lang.IllegalStateException: Cannot evaluate network with no output layer\n  at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:2974)\n  at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:2921)\n  at org.deeplearning4j.nn.multilayer.MultiLayerNetwork.evaluate(MultiLayerNetwork.java:2823)\n  ... 70 elided\n"}]},"apps":[],"jobName":"paragraph_1509379993885_-1535562606","id":"20171029-231714_1749373270","dateCreated":"2017-10-30T09:13:13-0700","dateStarted":"2017-11-04T15:10:30-0700","dateFinished":"2017-11-04T15:10:30-0700","status":"ERROR","progressUpdateIntervalMs":500,"$$hashKey":"object:442"},{"text":"%md\n\n### What's next?\n\n- Check out all of our tutorials available [on Github](https://github.com/eclipse/deeplearning4j/tree/master/dl4j-examples/tutorials). Notebooks are numbered for easy following.","user":"anonymous","dateUpdated":"2017-11-04T15:05:47-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<div class=\"markdown-body\">\n<h3>What&rsquo;s next?</h3>\n<ul>\n  <li>Check out all of our tutorials available <a href=\"https://github.com/eclipse/deeplearning4j/tree/master/dl4j-examples/tutorials\">on Github</a>. Notebooks are numbered for easy following.</li>\n</ul>\n</div>"}]},"apps":[],"jobName":"paragraph_1509833121055_99635236","id":"20171104-150521_881726694","dateCreated":"2017-11-04T15:05:21-0700","dateStarted":"2017-11-04T15:05:47-0700","dateFinished":"2017-11-04T15:05:47-0700","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:443"},{"text":"%md\n","user":"anonymous","dateUpdated":"2017-11-04T15:05:47-0700","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1509833147131_-1874465398","id":"20171104-150547_1333839171","dateCreated":"2017-11-04T15:05:47-0700","status":"READY","progressUpdateIntervalMs":500,"$$hashKey":"object:444"}],"name":"RNNs: Sequence Classification of Synthetic Control Data","id":"2CVU7NPYV","angularObjects":{"2CY73X3S5:shared_process":[],"2CWWRYD9Z:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}