Despite not technically being spec-compliant, tl was able to parse most of the CC-MAIN-2023-40 (September/October 2023) of CommonCrawl. The archive contains 3.40 billion web pages (3 384 335 454 to be exact) totalling of 98.38 TiB of compressed material, though that includes the entire raw HTTP conversation between the crawler and the server. By comparison, the resulting set of forms plus metadata is 54 GB compressed, large enough that just summarising the data takes considerable time. 51 152 471 (0.0151%) web pages in the dataset could not be parsed at all due to invalid HTML encoding, invalid character encodings, or bugs in the parser.
Материалы по теме:
，推荐阅读新收录的资料获取更多信息
Laura CressTechnology reporter
When another data point is inserted and the Voronoi diagram reconstructed, the newly created region displaces the area that once belonged to the old regions. Those points whose regions were displaced are considered natural neighbours to the new point. The weight of each natural neighbour is given by the area taken from the total area occupied by the new region. In 3D, we measure polyhedral volumes instead of areas.