subtleties of state
I found a type error in the database system I am implementing. The type error only is thrown after a certain number of pages are added to a table. This was, as one might expect, a fun exercise in debugging.
The actual trail of the path looks as follows:
Some context onto this problem:
The general idea is that a table has a set of pages, which may or may not be loaded into memory. However, this implies having some extremely complex requirements about visibility and lifetimes of your data. A table scan will thus ripple through your pages as you bring them in and out of memory. Further, indexes have to be overlaid on your columns in the table, which then should have - in some fashion - references to the page data. This means that typical GC approaches won't work, as you need a kind-of weak-reference - a weak reference such that when you access it, it becomes alive.
Another lesson that can be taken away here is that simple tests are not enough; complex semantic tests that exercise the boundary conditions of your system need to be put in place from the beginning; early drafts of this sytem were written on the fly and without tests.
Further, an adequately powerful type system can prevent certain classes of bugs. In this case, an adequate type system would have limited the expression of the idea that a pointer can be NIL from time to time (Of course, this means that the machinery for whether the data is present or not has to be encoded into the type system: no silver bullet).
As I noted above, this problem has a gnarly essential complexity - data has to come and go, transparent to the other layers of the architecture.. Right now, solving that essential complexity is not high on the priority list, so the caching feature will get cut, to be added at a later date (possibly at a different layer in the data archicture).