papers.adligo.com

SimpleZigguratDiskStructures

Author: Scott Morgan
Edited: 2025-11-25
Id: 1.3.6.1.4.1.33097.1.1.100
Copywrite 2021 Adligo Inc
Document URLs:
Questions Comments:

papers.adligo.com/issues

Papers Index

Summary

   This paper codifies the Ziggurat Disk Structure which is an abstract on-disk data structure for files and folders (aka Directories). The Ziggurat Disk Structure stores files which are identified by a positive integer identifier (i.e. 0,1,2,3,4, etc). It is designed to be human and machine readable, however the emphasis is machine readability and speed.
   There are several possible use-cases for the Ziggurat Disk Structure including;

   The Ziggurat Disk Structure is designed to be layered on standard UNIX (Apple/Mac) and Windows file systems as well as GFS (Google File System), HDFS (Hadoop File System) and can be adapted to other physical disk structures. It consists of a virtual associative array structure inspired by Peter van Emde Boaz Trees. However, it makes use of a fractal like disk pattern as the data elements are NOT expected to be known at the time of structure initialization.

Table of Contents

The Virtual Ziggurat Disk Structure Elements

   Ziggurat Disk Structures map a virtual infinite associative array structure to a physical disk structure. The virtual arrays may be of different sizes. However, it is recommended that they are all of the same size 2^X. In the following example they are all 2^3, creating 8 slots per array. To illustrate this we start at the bottom of the Ziggurat with the level zero arrays. Below in diagram #1 we illustrate the parent array of the first three bottom child arrays and their association.

Diagram 1

Diagram 1

   There MAY be any number of virtual parent arrays to create a unbounded universe of slots (provided your programming language supports infinite positive numbers). Note most programming languages do NOT support infinite positive integers. However, very large positive integers are supported in code like Java’s BigInteger class.

   Diagram #2 illustrates additional levels of virtual arrays and their association. With levels 0-3 we would be able to store 64 data elements with a height under level 1, 512 after growing the virtual associative array structure to level 2 and 4,096 after growing the virtual associative array structure to level 2. This growing virtual array structure can go on forever to continue to grow the structure to any size as necessary.

Diagram 2

Diagram 2

The Physical to Virtual Ziggurat Disk Structure Mappings

   At the top of the physical Ziggurat Disk Structure some sort of manifest file should exist which identifies the size of the arrays. This file could also contain other metadata.    Each data element is identified as being stored in a slot. The slot numbers are abbreviated as sX, and the X comes from the respective virtual array slot (i.e. 0-7 in our example diagrams). The levels are also abbreviated as lX and the X comes from the respective virtual associative arrays level (i.e. 0-3 in our examples. This gives us names like;

   Under the top parent folder of the Ziggurat Disk Structure the slot zero folders are created for each virtual parent array. You can see in Diagram #3 that the three files (l0s0.json, l0s1.json and l0s2.json) have been added under the l1s0 folder. These are different files than the files in green at the right under the l1s0 folder. Diagram #3 illustrates a entity style structure where each entity is stored in a separate physical file (i.e. l0s0.json).    In this diagram the summary information would be included in the respective summary files (i.e. l1s0.bin). Symlinks would be optional as they are supported by some file systems but not others.

Diagram 3 Entity per File

Diagram 3

Diagram 4 Multiple Entities per File / RDBMS Style

   Diagram #4 illustrates what a Ziggurat Disk Structure might look like if you were storing one data entity per row in binary files similar to the way most RDBMS (Relational Database Management Systems) do this. For example l1s0.bin would contain 8 rows one row for each slot in the virtual array. This is often how ACID (Atomic Consistent Independent Durable) transaction code is implemented.    In this diagram the summary information would be included in the .bin file (i.e. l1s0.bin). Symlinks would be optional as they are supported by some file systems but not others.

Diagram 4

Diagram 5 Hybrid Entity Relational

   Diagram #5 illustrates a hybrid Ziggurat Disk Structure. In this structure the traditional RDBMS binary file with one row per entity is augmented with the addition of one file per data entity. This give the flexibility of faster query implementations (i.e. SQL) in the table structure, with the addition of more complex tree like data structures in the Json (or XML, etc) formats.    In this diagram the summary information would be included in the .bin file (i.e. l1s0.bin). Symlinks would be optional as they are supported by some file systems but not others. Diagram 5

Conclusion

   The Ziggurat Disk Structure is a simple versatile file and folder management system that can be leveraged to facilitate unbounded heap prioritization systems, blockchains and flexible RDBMS, entity and hybrid data storage systems.

Citations / References

“The Google File System” by Sanjay Ghemawat et al. (2003): This paper presents the design of the Google File System (GFS), a distributed file system that has inspired many modern distributed storage systems.

https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html

Tromp, John & van Emde Boas, Peter. (1992). Associative Storage Modification Machines.. 291-313.

© Copyright 2025 Adligo Inc