| Trees | Indices | Help |
|
|---|
|
|
object --+
|
NgramModel
A general n-gram model, trained on some labelled data. Generate models using NgramModel.train().
Note that backoff assumes that an estimator with discounting is being used. The backed off probabilities are scaled according to the smoothing probability that would have been assigned to a zero count. If you use this with MLE, the backoff is effectively disabled.
The estimator should be picklable. This means you can't use a lambda, for example.
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
| list of (label sequence,probability) pairs |
|
||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
|||
|
Inherited from |
|||
|
|||
|
|||
|
|||
|
|||
|
model_type model_type gives a sensible name to the order of n-gram. |
|||
|
Inherited from |
|||
|
|||
x.__init__(...) initializes x; see help(type(x)) for signature
|
repr(x)
|
model_type gives a sensible name to the order of n-gram. Uses unigram, bigram, trigram, or x-gram. |
Returns the amount to scale the backed off probabilities by when backing off to an order n-1 model in the given context. This is presented as alpha in Jurafsky and Martin. Returned as a base 2 log. A more efficient way to do this would be to supply a function of the context specific to the discounting technique. In this case it wouldn't be necessary to sum the discounted mass each time. |
Gives the probability P(emission | label). Returned as a base 2 log. |
Removed. See forward_log_probabilities. Use either normal_forward_backward_probabilities or gamma_probabilities. |
Removed. Use normal_forward_probabilities and take logs if you're happy with normalized probabilities. Otherwise, normal_forward_probabilities needs to be made to return the sums it normalizes by. |
Reproduces an n-gram model that was converted to a picklable form using to_picklable_dict. Extra args/kwargs are passed to the class constructor. |
State-occupation probabilities.
|
Applies the N-best variant of the Viterbi algorithm to return N sequences of states that maximize the probability of the sequence of observations.
See Also: Generalization of the Viterbi Algorithm, Foreman, 1993 |
Generate a sequence of emissions at random using the n-gram model. If labels=True, outputs a sequence of (emission,label) pairs indicating what hidden labels emitted the emissions. The sequence will have maximum length |
Produces a matrix of the probability of each timestep's emission from each state. matrix[t,i] = p(o_t | state=i) |
Produces a matrix of the transition probabilities from every
(n-1)-gram to every state. Matrix indices are based on enumeration of
The matrix has n dimensions. The first index is the current state, the second the previous, etc. Thus, matrix[i,j,...] = p(state_t = i | state_(t-1) = j, ...). Probabilities are not logs. |
Computes the joint probability that the model assigns to a sequence and its gold standard labels. Probability is a log, because we'd get underflow otherwise. |
Return the backward probability matrices a Numpy array. This is faster than backward_log_probabilities because it uses Numpy arrays with non-log probabilities and normalizes each timestep.
|
A faster implementation of forward_backward_probabilities for the case where we're normalizing, using Numpy and non-log probabilities. This is still an S-dimensional matrix, not the state-occupation probabilities. Use gamma_probabilities to get that. |
Return the forward probability matrix as a Numpy array. This is equivalent to forward_probabilities, but much faster. It doesn't need logs because it's normalizing at each timestep.
|
Produces a picklable representation of model as a dict. You can't just pickle the object directly because some of the NLTK classes can't be pickled. You can pickle this dict and reconstruct the model using NgramModel.from_picklable_dict(dict). |
Trains the n-gram model given some labelled data. The data should be in the form of a sequence of sequences of tuples (e,t), where e is an emission and t a tag. If ignore_list is given, all ngrams containing a label in ignore_list will be ignored altogether. One use for this is to ignore blank labels, so we don't learn the gaps in the labelled data. |
Gives the probability P(label_i | label_(i-1), ..., label_(i-n)), where the previous labels are given in the sequence label_context. The context should be in reverse order, i.e. with the most recent label at the start. Note that this is the probability of a label given the previous n-1 labels, which is the same as the probability of the n-gram [label_i, ..., label_(i-n+1)] given the ngram [label_(i-1), ..., label_(i-n)], since all but the last element of the ngram overlaps with the condition, so has probability 1. Caches all computed transition probabilities. This is particularly important for backoff models. Many n-grams will back off to the same (n-1)-gram and we don't want to recompute the transition probability for that each time. |
Debugging version of the above. Use this only for debugging. It prints stuff out and doesn't cache. |
|
|||
model_typemodel_type gives a sensible name to the order of n-gram. Uses unigram, bigram, trigram, or x-gram.
|
| Trees | Indices | Help |
|
|---|
| Generated by Epydoc 3.0.1 on Mon Nov 26 16:04:58 2012 | http://epydoc.sourceforge.net |