- Notifications
You must be signed in to change notification settings - Fork3
Jacobian-Enhanced Neural Networks (JENN) are fully connected multi-layer perceptrons, whose training process was modified to account for gradient information. Specifically, the parameters are learned by minimizing the Least Squares Estimator (LSE), modified to minimize prediction error of both response values and partial derivatives.
License
shb84/JENN
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Jacobian-Enhanced Neural Networks (JENN) are fully connected multi-layerperceptrons, whose training process is modified to predict partialderivatives accurately. This is accomplished by minimizing a modified versionof the Least Squares Estimator (LSE) that accounts for Jacobian prediction error (seepaper).The main benefit of jacobian-enhancement is better accuracy withfewer training points compared to standard fully connected neural nets, as illustrated below.
If you use JENN in a scientific publication, please consider citing it:
@misc{berguin2024jacobianenhanced, title={Jacobian-Enhanced Neural Networks}, author={Steven H. Berguin}, year={2024}, eprint={2406.09132}, archivePrefix={arXiv}, primaryClass={id='cs.LG' full_name='Machine Learning' is_active=True alt_name=None in_archive='cs' is_general=False description='Papers on all aspects of machine learning research (supervised, unsupervised, reinforcement learning, bandit problems, and so on) including also robustness, explanation, fairness, and methodology. cs.LG is also an appropriate primary category for applications of machine learning methods.'}}
- Multi-Task Learning : predict more than one output with same model Y = f(X) where Y = [y1, y2, ...]
- Jacobian prediction : analytically compute the Jacobian (i.e. forward propagation of dY/dX)
- Gradient-Enhancement: minimize prediction error of partials (i.e. back-prop accounts for dY/dX)
pip install jenn
Seedemo notebooks for more details
Import library:
import jenn
Generate example training and test data:
x_train, y_train, dydx_train = jenn.synthetic.Sinusoid.sample( m_lhs=0, m_levels=4, lb=-3.14, ub=3.14,)x_test, y_test, dydx_test = jenn.synthetic.Sinusoid.sample( m_lhs=30, m_levels=0, lb=-3.14, ub=3.14,)
Train a model:
nn = jenn.model.NeuralNet( layer_sizes=[1, 12, 1],).fit( x=x_train, y=y_train, dydx=dydx_train, lambd=0.1, # regularization parameter is_normalize=True, # normalize data before fitting it)
Make predictions:
y, dydx = nn.evaluate(x)# OR y = nn.predict(x)dydx = nn.predict_partials(x)
Save model (parameters) for later use:
nn.save('parameters.json')
Reload saved parameters into new model:
reloaded = jenn.model.NeuralNet(layer_sizes=[1, 12, 1]).load('parameters.json')
Optionally, ifmatplotlib
is installed, import plotting utilities:
from jenn.utils import plot
Optionally, ifmatplotlib
is installed, check goodness of fit:
plot.goodness_of_fit( y_true=dydx_test[0], y_pred=nn.predict_partials(x_test)[0], title="Partial Derivative: dy/dx (JENN)")
Optionally, ifmatplotlib
is installed, show sensitivity profiles:
plot.sensitivity_profiles( f=[jenn.synthetic.Sinusoid.evaluate, nn.predict], x_min=x_train.min(), x_max=x_train.max(), x_true=x_train, y_true=y_train, resolution=100, legend=['true', 'pred'], xlabels=['x'], ylabels=['y'],)
JENN is intended for the field of computer aided design, where there is oftena need to replace computationally expensive, physics-based models with so-calledsurrogate models inorder to save time down the line. Since thesurrogate model emulates the original model accuratelyin real time, it offers a speed benefit that can be used to carry out orders of magnitudemore function calls quickly, opening the door to Monte Carlo simulation of expensive functions for example.
In general, the value proposition of a surrogate is that the computationalexpense of generating training data to fit the modelis much less than the computational expense of performing the analysis with the original physics-based model itself.However, in the special case of gradient-enhanced methods, there is the additional value proposition that partialsare accurate which is a critical property for one important use-case:surrogate-based optimization. The field ofaerospace engineering is rich inapplications of such a use-case.
Gradient-enhanced methods require responses to be continuous and smooth,but they are only beneficial if the cost of obtaining partialsis not excessive in the first place (e.g. adjoint methods), or if the need for accuracy outweighs the cost ofcomputing the partials. Users should therefore carefully weigh the benefit ofgradient-enhanced methods relative to the needs of their application.
Distributed under the terms of the MIT License.
This code used the code by Prof. Andrew Ng in theCoursera Deep Learning Specializationas a starting point. It then built upon it to include additional features suchas line search and plotting but, most of all, it fundamentally changed the formulationto include gradient-enhancement and made sure all arrays were updated in place (data is never copied).The author would like to thank Andrew Ng foroffering the fundamentals of deep learning on Coursera, which took a complicatedsubject and explained it in simple terms that even an aerospace engineer could understand.
About
Jacobian-Enhanced Neural Networks (JENN) are fully connected multi-layer perceptrons, whose training process was modified to account for gradient information. Specifically, the parameters are learned by minimizing the Least Squares Estimator (LSE), modified to minimize prediction error of both response values and partial derivatives.