DOCUMENTATION

tokens.csv represents the relative aggregate distribution of tokens as
seen when parsing a nominal ten billion documents. The data was
collected by Google in July 2007, using a parser based on r967 of the
HTML5 specification.

The file is a comma-delimited file with five columns. The first column
is the phase and the second column is the insertion mode, if the phase
was the main phase. The third column is the token. The token names are
self-explanatory, except that successive characters are combined into
one token. Parse errors are not treated as tokens and do not figure in
this data. Start tags and end tags have the tag name in the fourth
column; the tag name "other" is used for elements that are in the
phrasing category. The last column is the relative frequency of that
token. Tokens were only counted when they were emitted from the
tokeniser; tokens that are "reprocessed", and fake tokens that are
emitted when the tree construction code is required to "act as if" a
particular token had been seen, were not counted.

The parser was run with scripting enabled but no scripts supported, so
<noscript> elements were treated as CDATA blocks but no script blocks
were executed.


METHODOLOGY

The sample data consisted of several billion documents from Google's
index. An attempt was made to eliminate duplicates from the
sample. Documents that did not appear, a priori, to be HTML documents
(e.g. PDF documents, Flash animations, Word documents) were not
included in the sample. Documents which, when parsed, resulted in DOMs
with more than 65535 elements were only parsed up to the creation of
the 65535th element.

It is hoped that this data will help implementors of HTML5 parsers to
optimise their branches to better serve actual Web content.

For more information, please contact Ian Hickson <ian@hixie.ch>.


LICENSE

Copyright 2007 Google, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You may
obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License.