Basically, this file contains stuff that should be documented, but is not.
Feel free to contribute things that you want documented, as well as to add or correct documentation.
Let’s talk about defining the grad() function in an Op, using an illustrative example.
In Poisson regression (Ranzato and Szummer, 2008), the target t is integer data, which we predict using a continuous output o. In the negative log likelihood of the Poisson regressor, there is a term:
Let’s say we write a logfactorial Op. We then compute the gradient
You should define gradient, even if it is undefined. [give log factorial example]
If an Op does not define grad, but this Op does not appear in the path when you compute the gradient, then there is no problem.
If an Op does not define grad, and this Op does appear in the path when you compute the gradient, WRITEME.
Gradients for a particular variable can be one of four kinds: 1) forgot to implement it
You will get an exception of the following form.
theano.gof.utils.MethodNotDefined: (‘grad’, <class ‘pylearn.algorithms.sandbox.cost.LogFactorial’>, ‘LogFactorial’)
currently, there is no way for a grad() method to distinguish between cases 3 and 4 but the distinction is important because graphs with type-3 gradients are ok to run, whereas graphs with type-4 gradients are not. so I suggested that Joseph return a type-4 gradient by defining an Op with no perform method. the idea would be that this would suit the graph-construction phase, but would prevent linking. how does that sound to you?
This documentation is useful when we show users how to write Ops.
st_impl is an optional method in an Op. @staticmethod is a Python decorator for a class method that does not implicitly take the class instance as a first argument. Hence, st_impl can be used for Op implementations when no information from the Op instance is needed. This can be useful for testing an implementation. See the XlogX class below for an example.
This documentation is useful when we show users how to write Ops. Olivier says this behavior should be discouraged but I feel that st_impl should be encouraged where possible.
Olivier says that XlogX gives a good example. In fact, I would like to beef xlogx up into our running example for demonstrating how to write an Op:
class XlogX(scalar.UnaryScalarOp):
"""
Compute X * log(X), with special case 0 log(0) = 0.
"""
@staticmethod
def st_impl(x):
if x == 0.0:
return 0.0
return x * numpy.log(x)
def impl(self, x):
return XlogX.st_impl(x)
def grad(self, inp, grads):
x, = inp
gz, = grads
return [gz * (1 + scalar.log(x))]
def c_code(self, node, name, inp, out, sub):
x, = inp
z, = out
if node.inputs[0].type in [scalar.float32, scalar.float64]:
return """%(z)s =
%(x)s == 0.0
? 0.0
: %(x)s * log(%(x)s);""" % locals()
raise NotImplementedError('only floatingpoint is implemented')
scalar_xlogx = XlogX(scalar.upgrade_to_float, name='scalar_xlogx')
xlogx = tensor.Elemwise(scalar_xlogx, name='xlogx')
It is also necessary to talk about UnaryScalarOp vs. BinaryOp.
UnaryScalarOp is the same as scalar.ScalarOp with member variable nin=1. give an example of this
Guillaume can you make sure to hit these points:
What are canonical examples of tests?
- What are the different test patterns?
nnet.py:
- What is going on with test1, test2, test3, test4?
What is the right eq function to use?
- There are a lot of tests that define their own epsilon, but this should be standardized. e.g. in test_elemwise.py self.assertTrue((numpy.abs(f(xv) - zv) < 1e-10).all())
If the expected variable of a test is that an Exception is thrown, how do we correctly detect and handle that?
nosetests has assertRaises
- Convention is that all test files must start with test_, not
_test_, so rename all that use the old convention?
** This is also useful in the How to write an Op tutorial. **
This is internal documentation. Guillaume can you make sure to hit these points:
export THEANO_BLAS_LDFLAGS=’-lmkl -liomp5 -fopenmp’
Do we want the following:
export OMP_NUM_THREADS=2
- Are there functions for doing type checking?
like dtype of this matrix is an int-type (not just int32 or int64) “if isinstance(item, int):” is the preferred way to do it in python now, so mimic this If the type is wrong, what exception should be raised?
- If we have a matrix with only one row, how do we convert it to a vector?
x.reshape(x.size) You can also use resize but there is not reason to ‘’resize’‘
- How do you convert the type of a numpy array?
theano._asarray(x, dtype = 'int32') Note that using numpy.asarray is potentially dangerous, due to a problem in numpy where the type may not be properly set (see numpy’s Track ticket #870).
theano.compile.io.Out(gw1, borrow = True) for that value in compile.function