Contents
This guide is written for Linux users and should suit Mac users well enough. I have not run the parser under Windows much myself, but have tested the basic running of the thing. There may be problems with running under Windows that I don't know about.
I'm assuming you understand basic terminal usage. The commands shown below include a ">" command prompt: don't type this.
First download the parser and all of its dependencies.
Basic Usage
Running the parser to parse a short sequence of chords is simple. Once you've downloaded the code, open up a terminal and change to the bin directory. To run the parser's command line interface, run the following command:
> ./jazzparser
If you're using Windows, fire up cmd to get a terminal. You can do all the same things described below using the batch file instead of the Bash script in the previous example, like so:
> jazzparser.bat
If all goes well, you should see a prompt for a chord sequence. We've not given any input, so you'll just see the message "No input string given" and it will wait for input from the command line.
You can give input in various forms. One way is to run the parser as you just did and type a chord sequence, followed by <enter>. You can do this for as many chord sequences as you like. Press Ctrl+D to exit.
Alternatively, you can give a chord sequence as an argument. Here's some simple input to show the parser in action:
> ./jazzparser "A E7 A"
This is performing a full CKY parse with no statistical models. It's using the default parser module, which is a plain CKY parser (called cky), and the default supertagger, which just supplies all possible categories to the parser (called full).
You should get some output like this for this first example:
Processing input: A E7 A (commandline) Tagging sequence (3 timesteps) >>> Parsing iteration: 1 Parse finished with 31 results Parsed: A E7 A Results: 0> Db^T : [<3,1>, <0,1>, <3,1>] ... 26> A^T : [<3,0>, leftonto(<3,0>)] ... 30> A^T : [<3,0>, leftonto(leftonto(<3,0>))] commandline Complete Completed 1/1 parses
That result numered 26 is the one we're looking for. Since there's no probabilistic model involved in the parsing, the parser has no idea of how plausible each of these results is.
What you see here are CCG categories: syntactic type before the : and logical form after. The logical form gives the tonal space analysis. In result 26, the sequence starts at A (coordinate (3,2)), then cadences back to A again using a single left step.
Use the -h (or --help) option to see all the command line options the parser can take.
Using a Parsing Model
Use the -p option (or --parser) to select a parsing module. The default, cky, uses no probabilistic model. Instead, let's use pcfg.
> ./jazzparser -p pcfg "A E7 A"
This will cause an error. Each parsing module has a set of options that you can specify using --popt (or --parser-options) on the command line. Some parser modules require certain options to be given. To see a list of all a module's options, type help where the options would be:
> ./jazzparser -p pcfg --popt help
This applies in general wherever a module takes options in this form (for example, supertagger options). Module options can be given in the form option=value and multiple options may be given using multiple --popt options or by separating them with colons (option1=value1:option2=value2).
In the parser option help, we can see that the model option is required for the pcfg parser. This specifies a trained parsing model. A different script is used to train models, but fortunately one is included pre-trained with the parser, called chords.
Now we can load a model and parse using the probabilistic parser.
> ./jazzparser -p pcfg --popt model=chords "A E7 A" ... Parse finished with 13 results Parsed: A E7 A Results: 0> A^T : [<3,0>, leftonto(<3,0>)] ... 12> E^T-Db^T : [<0,1>, <3,1>]
This time the parser has ranked the results by their probability. It's still entertained some whacky interpretations, but there are far fewer results than before, because low-probability interpretations have been eliminated, and the one we want is at the top of the list.
The model we've told it to use is a statistical parsing model trained using a corpus of jazz chord sequences annotated with analyses (a treebank). For now, we'll be using the pre-trained models distributed with the code.
Let's try a slightly longer chord sequence, using some different chord types:
> ./jazzparser -p pcfg --popt model=chords "A F#o7 Bm7 E7 AM7" ... 0> A^T : [<3,0>, leftonto(leftonto(leftonto(<3,0>)))]
The parser has again managed to get the interpretation we want, treating it as an extended cadence, at the top - it is, after all, a pretty simple example.
Using a Supertagging Model
The sort of parsing we've tried so far starts by looking up all possible grammatical categories for each individual chord from the grammar's lexicon (lexical categories). It then tried combining them into a full interpretation. Before we introduced the -p option, it was looking for all possible combinations. And then, when we specified a parsing model, it was able to select just the most promising combinations.
A supertagging model helps the parser out by cutting down the selection of lexical categories the parser has to start with. It does this by looking at the context of each chord and comparing to interpretations previously used in similar contexts.
The parser did fine on its own with the short examples above. Try parsing this chord sequence:
> ./jazzparser "A Bm7 E7 Bm7 E7 AM7"
You'll find it takes much longer, and that's still a very short chord sequence if you compare to real songs. What's more, the parser's already running with very constrained settings (the default settings), so its ranking of the final results is not very good.
Take a look at the parameters that the parser is using when you run it with no options:
> ./jazzparser No input string given: accepting input from stdin. Hit Ctrl+d to exit ===== The Jazz Parser ===== Supertagger: full ...
It's already using a supertagger. In fact, the Jazz Parser always uses a supertagger. It's just that here it's using a special case, the full supertagger, that just gives the parser all the lexical categories that the grammar allows.
Let's speed things up by using a statistical supertagger:
> ./jazzparser -t ngram-multi --topt model=bigram "A Bm7 E7 Bm7 E7 AM7" -p pcfg --popt model=chords ... Parse finished with 2 results Parsed: A Bm7 E7 Bm7 E7 AM7 Results: 0> A^T : [<3,0>, ((\x1.leftonto(leftonto(x1)) & \x1.leftonto(leftonto(x1))) <3,0>)]
Here we used the ngram-multi supertagger, which uses an ngram model (the generalization of an HMM) to choose the categories. The model we chose was a bigram model (i.e. an HMM), trained on the same corpus as the parsing model. The result should have popped out pretty quickly this time.
The parser can now handle much longer chord sequences. Both the supertagger and the parser can produce better quality results if you change their beam settings. By default, these are set so that a result is produced quickly, which is good for firing off parses from the command line like this. When running experiments, it's better to use less constrained settings and leave the parser going longer.
For more information on N-gram models, see NgramSupertaggingModels.
For more general info on supertagging, see ParsingModels.
Parser and Supertagger Options
Above, we chose a parsing module using -p and a supertagging module using -t. Each module comes with its own setting of parameters. We had to set one for both the parser and supertagger to tell them what statistical models to use (--topt model=bigram and --popt model=chords).
There are other parsing and supertagging modules as well as pcfg and ngram-multi and each has its own list of parameters. The parameters are specified using --popt and --topt respectively. You can get a list of the available parameters with --popt help (along with -p to select the parsing module).
> ./jazzparser -p pcfg --popt help
Available options for selected parser
=====================================
model model=X, where X is the name of the Name of a trained PCFG model to use for
model. (REQUIRED) parsing.
max_iter max_iter=X, where X is an integer. Maximum number of parser iterations to
...> ./jazzparser -t ngram-multi --topt help
Available options for selected tagger
=====================================
model model=X, where X is the name of a Model name. This model must have been
trained model (REQUIRED) previously trained. Required
partition partition=P, where P is an int If given, a partitioned version of the
...You can specify multiple parameter settings by separating them with a colon (e.g. --popt model=chords:inspect=T), or by giving multiple --popts (e.g. --popt model=chords --popt inspect=T). The same goes for --topt and other module options that work in this way.
More Topics
That covers the basic usage of the parser. The following pages document other features.
ConfigFiles: it's a pain to have to specify all the parser's options every time you run it; instead you can put them in a file.
InteractiveShell: once the parser's done, you can examine its output interactively.
FileInput: the parser can take chord sequence (and other) input from files, including multiple songs at once.
FileOutput: the parser can write its results to a file that you can load and analyse later.
The rest of this is old
The rest of this should be moved to other pages.
Derivations
For the sake of parsing time and space, the parser by default does not store the necessary information to display derivation trees once parsing has finished. However, the -d option makes it store this information.
Use this with the -i option: then you can use the d command in the interactive shell to print out a textual representation of the derivation tree for a specific result. This includes all the possible derivations of the result, condensed into one tree, so this tree can get pretty huge!
File Input
Instead of typing in individual chord sequences, you can get input from a file using the -f option, followed by the filename. This can allow you to parse multiple sequences in one go.
The Jazz Parser accepts several types of input. The default is just a textual representation of chords, like that we parsed from the command line above. All input types can be given as input files, included textual chords.
Select a filetype using the --ft option. --ft help shows a list of filetypes.
Select an input file with the -f option. Some filetypes have their own options, which you can specify using the --fopt option. Use --fopt help (in conjunction with --ft) to see available options.
Chord Input Files
The simplest input format is plain text chords. This is the same as the input we gave on the command line above. You can just put a chord sequence in a file and take it as input using -f <filename>. (This is filetype chords, but it's the default, so no need for --ft.)
More interestingly, you can put multiple chord sequences in a single file and the parser will process them each in turn. This is filetype bulk-chords.
The input file should contain lines of chord sequences, like this:
Im(6) II%7, V7 Im V%7, I7 IV7 #IVo7 Im(6) bIIIm7, bVI7 II%7 V7 Im(6) Im(6)
Other useful things you can put in there are:
- "|" symbols to mark barlines (this is for readability only - the parser ignores them)
IM7 | IIm7 V7 | IM7 | IIm7 V7 || IM7 I7 | IVM7 #IVo7 | V7 | IM7
- comments - any line starting "//" will not be parsed
// Here are some more chord sequences. This line will be ignored.
outputting comments - lines starting ">>" will not be parsed, but the whole line, after the ">>", will be printed out during parsing. This is handy for dividing up and explaining the output from the parser.
>>Rhythm changes A sections
- name assignments - any comment (outputting or non-outputting) that starts with the symbol "=" will be used as a name for the chord sequence that follows it and this will be displayed in the result summary (see below).
>>=Rhythm changes example 1 ... //=Autumn Leaves A ...