FENNIX

Fast Experimentation with Neural Networks


Getting started

This section will guides you through the first steps for using FENNIX. This is not a complete guide, but just an introductory tutorial about how to use the most common FENNIX tools. Detailed information about each specific plug-in can be found by using the function help() giving as argument the name of the desired plug-in.


Download

The last version of FENNIX can be downloaded from sourceforge.

Back to top

Launch FENNIX

FENNIX is written in Java, so it can be executed by using the command:

	java -jar ./dist/FENNIX.jar
However, when you download FENNIX you get batch files for an easier execution:
	FENNIX_GUI.sh
	FENNIX_GUI.bat
Back to top

The GUI

The graphical user interface of FENNIX is very simple. It has two panels for describing the experience: one for entering the script in text format (numbered as 2 in the image), and one for entering the script in a graphic tree-like format (numbered as 3 in the image). Moreover. it has a small command line (numbered as 6 in the image) that allows you to enter single commands that are executed directly after pressing enter.


The GUI contains buttons (numbered as 5 in the image) that allows you to control the execution of the script:
Messages and results are shown in the black panel numbered as 4 in the image.
More details about the graphical user interface can be found the section GUI.
Back to top

Manipulating datasets

So far, there is only one available plug-in for reading a dataset from a text file. This plug-in is called FNNXDatasetLoader. If you type in the command line:


	help(FNNXDatasetLoader)
	
You will obtain a description of the plug-in, and some instructions about how to use it, and the format of the files it accepts.
	-------------------------------------------------------------
	FNNXDatasetLoader implements a FNNXFunction that reads a text
	file and stores its content in an appropriate object. Each
	row of the file is considered as an observation, and columns
	can be separated by tabulation, comma (,) or semicolon (;).
	semicolons (;). The file must contain a header in order to
	facilitate its interpretation, and to provide a means to
	check its consistency. There are different header formats:

	-FENNIX_005:
	    FENNIX
	    #PAT: n_observations
	    #INPUT: n_inputs
	    #OUTPUT: n_outputs

	-FENNIX_008:
	    FENNIX
	    ->newBlock(n_observations, n_inputs, n_outputs)

	Hence, the first n_inputs columns are considered as inputs,
	and the last n_outputs columns are considered as outputs.

	Usage:
	dataset_name = call(FNNXDatasetLoader) {
	    file = "file_name.fxd"
	    format = "FENNIX_008"
	}
	-------------------------------------------------------------
This is an example of one of the formats:
FENNIX

->newBlock(4, 2, 1)

0	0	0
0	1	1
1	0	1
1	1	0
The plug-in FNNXDatasetLoader reads a text file, decodes its content, and stores the result in an appropriate FENNIX object. For example, the following script will read the file wines.fxd and stores the resulting dataset in a variable called data:

	data = call(FNNXDatasetLoader) {
		file = "file.fxd"
		format = "FENNIX_008"
	}
	
The dataset is part of the UCI Machine Learning Repository, and contains data resulting of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. The analysis determined the quantities of 13 constituents found in each of the three types of wines. The goal of the dataset is to classify the three types of wines. More information about the dataset can be found here, and the original dataset can be found here.

Moreover, the dataset may need to be normalized. The plug-in FNNXNormalizer normalizes datasets.

	help(FNNXNormalizer)
	
	-------------------------------------------------------------
	FNNXNormalizer takes a dataset and normalizes each one of its
	columns. Several normalization methods are available:

	-MIN_MAX: Transforms the data changing their minimum and
	    maximum to the values given as input parameters.
	-AVG_RANGE: Transforms the data changing their average and
	    range to the values given as input parameters.
	-PLUS_MULT: Transforms the data by adding the first parameter
	    and multiplying by the second parameter.
	-CLIP_MIN_MAX: Modifies the data by clipping all values
	    exceding the two input parameters (min and max). This is
	    a non-linear transformation.

	Usage:
	call(FNNXNormalizer) {
	    dataset = dataset_name
	    method = "MIN_MAX"
	    p1 = -0.95
	    p2 = 0.95
	}
	-------------------------------------------------------------
The following example reads the dataset in the file wines.fxd, and normalizes it between -0.95 and 0.95:

	data = call(FNNXDatasetLoader) {
		file = "wines.fxd"
		format = "FENNIX_008"
	}
	call(FNNXNormalizer) {
		dataset = 	data
		method = 	"MIN_MAX"
		p1 = 	-0.95
		p2 = 	0.95
	}
	
You can have more information about the variable "data" by using the function show():

	show(data)

DataBlock: [178x13]
Max = [0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95]
Mean = [0.035, -0.350, 0.073, -0.079, -0.336, -0.088, -0.273, -0.119, -0.242, -0.338, -0.212, -0.016, -0.315]
Min = [-0.95, -0.95, -0.95, -0.95, -0.95, -0.95, -0.95, -0.95, -0.95, -0.95, -0.95, -0.95, -0.95]
Range = [1.9, 1.9, 1.9, 1.9, 1.9, 1.9, 1.9, 1.9, 1.9, 1.9, 1.9, 1.9, 1.9]
MSStot = [0.164, 0.175, 0.077, 0.106, 0.086, 0.167, 0.159, 0.198, 0.117, 0.140, 0.124, 0.243, 0.181]

DataBlock: [178x3]
Max = [0.95, 0.95, 0.95]
Mean = [-0.320, -0.192, -0.438]
Min = [-0.95, -0.95, -0.95]
Range = [1.9, 1.9, 1.9]
MSStot = [0.8, 0.866, 0.711]
Back to top

Creating a multilayer perceptron

The plug-in FNNXMLP implements a multilayer perceptron.

	
	help(FNNXMLP)
	
	-------------------------------------------------------------
	FNNXMLP is a FNNXObject containing a multi-layer perceptron.
	After created, the MLP is fully connected and its weights
	are randomly initialized. Bias units are automatically added.
	The user must specify some  mandatory parameters:

	inputs: number of inputs
	hidden: number of hidden units (can be a vector)
	actFuncHidden: activation function for the hidden units
	outputs: number of output units
	actFuncOutput: activation function for the output units
	shortcut: must input units be connected to output units?
	Several activation functions are available:
	LINEAR: linear activation function
	SIGM: sigmoidal activation function
	TANH: hyperbolic tangent activation function
	EXP: exponential activation funcion

	Usage:
	network_name = new(FNNXMLP) {
	    inputs =	?
	    hidden = ?
	    actFuncHidden = "TANH"
	    outputs = ?
	    actFuncOutput = "TANH"
	    shortcut = true
	}
	-------------------------------------------------------------
The following code creates a multilayer perceptron and stores it in a variable called "mlp":

	mlp = new(FNNXMLP) {
		inputs = 	13
		hidden = 	4
		actFuncHidden = 	"TANH"
		outputs = 	3
		actFuncOutput = 	"TANH"
		shortcut = 	false
	}
	
Once the variable is created, you can get information about the multilayer perceptron by using the function show():

	help(mlp)
	
	FNNXMLP{
		inputs = 13.0
		hidden = [4.0]
		actFuncHidden = [TANH]
		outputs = 3.0
		actFuncOutput = TANH
		shortcut = false
	}
Back to top

Training a multilayer perceptron

The most widely used technique for training multilayer perceptrons is the back-propagation algorithm. FENNIX has several plug-ins implementing this algorithm. FNNXOnlineBackprop peforms online back-propagation (weights change after the presentation of each example) and FNNXBatchBackprop performs batch back-propagation (weights change after the presentation of the whole set of examples). The function help() gives information about them:


	help(FNNXOnlineBackprop)
	
	-------------------------------------------------------------
	FNNXOnlineBackprop executes the back-propagation algorithm
	for a given neural network, by using a given dataset. The
	back-propagation algorithm is executed in an online manner
	with weight modifications after the presentation of each
	training example. This implementation provides stochastic
	learning and the Fahlman's trick. The user must specify some
	mandatory parameters:

	network: the feed-forward network to train
	dataset: the dataset containing the examples
	etaIni: initial value for the learning rate
	etaEnd: final value for the learning rate (stochastic)
	mu: the momentum term (inertia)
	epochs: how many times the dataset has to be presented
	weightDecay: regularization term
	minErrorPercent: the error is not back-propagated if it is
	smaller that this value

	Usage:
	network_name = new(FNNXMLP) {
	    inputs =	?
	    hidden = ?
	    actFuncHidden = "TANH"
	    outputs = ?
	    actFuncOutput = "TANH"
	}
	dataset_name = call(FNNXDatasetLoader) {
	    file = file_name.fxd
	    format = "FENNIX_008"
	}
	testTraining = new(FNNXSSETester) {
	    network = network_name
	}
	testValidation = new(FNNXSSETester) {
	    network = network_name
	}
	plot_name = new(FNNXPlotter)
	save_name = new(FNNXSaver) {
	    file = "file_name.fxr"
	}
	call(FNNXOnlineBackprop) {
	    network = network_name
	    dataset = dataset_name
	    tester = test_training
	    validator = test_validation
	    etaIni = 0.05
	    etaEnd = 0.001
	    mu = 0.7
	    epochs = ?
	    weightDecay = 0
	    minErrorPercent = 2
	    stopIfTesterBelow = 0
	    logCounter = 100
	    plotter = plot_name
	    saver = save_name
	}
	-------------------------------------------------------------
FNNXOnlineBackprop performs online back-propagation on a multilayer perceptron by using a given dataset. In order to monitor the learning process, FNNXOnlineBackprop executes the test given by the parameter tester after a certain number of iterations given by the parameter logCounter. The results of the test are sent to a plotting and/or saving object specified by the parameter plotter and saver.
The following script reads the file wines.fxd, normalizes it between -0.95 and -0.95, creates a multilayer perceptron, and trains it for 100 epochs with this dataset. Every 50 iterations (examples) the plug-in measures the sum of squared errors (FNNXSSETester) on the training data and plots (FNNXPlotter) it.

	data = call(FNNXDatasetLoader) {
		file = "wines.fxd"
		format = "FENNIX_008"
	}
	call(FNNXNormalizer) {
		dataset = 	data
		method = 	"MIN_MAX"
		p1 = 	-0.95
		p2 = 	0.95
	}
	mlp = new(FNNXMLP) {
		inputs = 	13
		hidden = 	4
		actFuncHidden = 	"TANH"
		outputs = 	3
		actFuncOutput = 	"TANH"
		shortcut = 	false
	}
	test = new(FNNXSSETester) {
		network = mlp
	}
	plot = new(FNNXPlotter)
	call(FNNXOnlineBackprop) {
		network =	mlp
		dataset =	data
		tester =	test
		etaIni =	0.05
		etaEnd =	0.001
		mu =	0.7
		epochs =	100
		weightDecay =	0
		minErrorPercent =	2
		stopIfTesterBelow =	0
		logCounter =	50
		plotter =	plot
	}

Back to top

Testing a multilayer perceptron

There are several plug-ins allowing the user to perform a test after or during training. The plug-in FNNXOutputTester computes the output of a multilayer perceptron using a given dataset.


	help(FNNXOutputTester)
	
	-------------------------------------------------------------
	FNNXOutputTester feeds input data from a given dataset into a
	given supervised neural network and computes its output.

	Usage:
	plot_name = new(FNNXPlotter)
	save_name = new(FNNXSaver) {
	    file = "file_name.fxr"
	}
	call(FNNXOutputTester) {
	    network = network_name
	    dataset = dataset_name
	    plotter = plot_name
	    saver = save_name
	}
	-------------------------------------------------------------
Executing the FNNXOutputTester on the multilayer perceptron from the last example will give us the following result:

	call(FNNXOutputTester) {
		network = 	mlp
		dataset = 	data
		plotter = 	plot
	}


The figure shows three plots, one for each output of the multilayer perceptron. The green lines represent the data in the dataset (target), and the cyan points represent the output of the multilayer perceptron. We can see that in the last example the neural network reproduces the training data almost perfectly.

Another useful plug-in is FNNXPerformanceTester which computes the amount of examples correctly classified by the neural network.

	help(FNNXPerformanceTester)
	
	-------------------------------------------------------------
	FNNXPerformanceTester feeds input data from a given dataset
	into a given supervised neural network and computes its
	performance. The performance is computed as the percent of
	examples correctly fitted by the neural network. A value for
	the allowed error must be given to assess whether an output
	is correct or not.

	Usage:
	plot_name = new(FNNXPlotter)
	save_name = new(FNNXSaver) {
	    file = "file_name.fxr"
	}
	call(FNNXPerformanceTester) {
	    network = network_name
	    dataset = dataset_name
	    allowedError = ?
	    plotter = plot_name
	    saver = save_name
	}
	-------------------------------------------------------------
Executing the FNNXPerformanceTester on the multilayer perceptron from the last example will give us the following result:

	call(FNNXPerformanceTester) {
		network = 	mlp
		dataset = 	data
		allowedError = 40
		plotter = 	plot
	}


The figure shows four vertical bars, one bar for each output of the neural network, plus an additional bar summarizing all the outputs of the network (logical AND). Each bar shows the amount of examples correctly classified using the percent of allowed error given by the parameter allowedError.
Back to top

Hold-out validation

The plug-in FNNXSampler allows to split a dataset in two parts by randomly sampling it without replacing.

	
	help(FNNXSampler)
	
	-------------------------------------------------------------
	FNNXSampler takes a given percent of random samples from a
	dataset, without replacement. It returns a vector with two
	elements: the first element contains the selected samples,
	and the second element contains the rest of the dataset.

	Usage:
	vector_name = call(FNNXSampler) {
	    dataset = dataset_name
	    percent = 80
	}
	-------------------------------------------------------------
Hence, it is possible to train the multilayer perceptron by using one part of the dataset, and then to validate the resulting network by using the other part of the dataset (which was not seen before by the network). The following example shows how to do this with FENNIX:

	data = call(FNNXDatasetLoader) {
		file = "wines.fxd"
		format = "FENNIX_008"
	}
	call(FNNXNormalizer) {
		dataset = 	data
		method = 	"MIN_MAX"
		p1 = 	-0.95
		p2 = 	0.95
	}
	trainVal = call(FNNXSampler) {
		dataset =	data
		percent =	80
	}
	dataTrain = trainVal[0]
	dataValid = trainVal[1]
	mlp = new(FNNXMLP) {
		inputs = 	13
		hidden = 	4
		actFuncHidden = 	"TANH"
		outputs = 	3
		actFuncOutput = 	"TANH"
		shortcut = 	false
	}
	test = new(FNNXSSETester) {
		network = mlp
	}
	plot = new(FNNXPlotter)
	call(FNNXOnlineBackprop) {
		network =	mlp
		dataset =	data
		tester =	test
		etaIni =	0.05
		etaEnd =	0.001
		mu =	0.7
		epochs =	100
		weightDecay =	0
		minErrorPercent =	2
		stopIfTesterBelow =	0
		logCounter =	50
		plotter =	plot
	}
	plotTrain = new(FNNXPlotter)
	call(FNNXOutputTester) {
		network = 	mlp
		dataset = 	dataTrain
		plotter = 	plotTrain
	}
	plotValid = new(FNNXPlotter)
	call(FNNXOutputTester) {
		network = 	mlp
		dataset = 	dataValid
		plotter = 	plotValid
	}

We obtain the following images after running this example:


The leftmost figure shows the output of the neural network for the training dataset, and the rightmost figure shows the output of the neural network for the validation dataset.

However, a single hold-out validation is not a good idea if the dataset is small. In small datasets it is more useful to perform several independent hold-out validations and compute the average of the individual validation results (leave-one-out validation, bootstrap validation). The following example shows how to perform 100 independent validations, and store the results in a text file for a further analysis (averaging):

	data = call(FNNXDatasetLoader) {
		file = "wines.fxd"
		format = "FENNIX_008"
	}
	call(FNNXNormalizer) {
		dataset = 	data
		method = 	"MIN_MAX"
		p1 = 	-0.95
		p2 = 	0.95
	}
	plot = new(FNNXPlotter)
	save = new(FNNXSaver) {
		file =	"validation.fxr"
	}
	i = 0
	while (i < 100) {
		show(i)
		trainVal = call(FNNXSampler) {
			dataset =	data
			percent =	80
		}
		dataTrain = trainVal[0]
		dataValid = trainVal[1]
		
		mlp = new(FNNXMLP) {
			inputs = 	13
			hidden = 	4
			actFuncHidden = 	"TANH"
			outputs = 	3
			actFuncOutput = 	"TANH"
			shortcut = 	false
		}
		test = new(FNNXSSETester) {
			network = mlp
		}
	    
		call(FNNXOnlineBackprop) {
			network =	mlp
			dataset =	dataTrain
			tester =	test
			etaIni =	0.05
			etaEnd =	0.001
			mu =	0.7
			epochs =	100
			weightDecay =	0
			minErrorPercent =	2
			stopIfTesterBelow =	0
			logCounter =	50
			plotter =	plot
		}
		call(FNNXPerformanceTester) {
			network = 	mlp
			dataset = 	dataValid
			allowedError =	40
			saver = 	save
		}
		i++
	}

This script will generate the file "validation.fxr" where the results of the individual validation tests are stored. The average of this results is easily computed by using a statistical software like R.
Back to top

Last modification 16 April 2012
by Héctor Satizábal.