Introduction
For decades, the historians of cartography have exhorted us to see maps not as objective views of a place but as texts that make interpretations and arguments about the world they represent. At the very same time, scholars from various disciplines have increasingly applied computational analytics and machine learning to raise new questions about the written word through techniques like text mining, XML encoding and data analytics. Putting these trends together, this project was inspired to ask: can one read maps computationally?
By far, the most common—almost ubiquitous—approach to digital analysis of historical cartography is a process known as “georectification” or “georeferencing,” in which maps of the past are layered on top of modern base maps and contorted to fit features that are shared by the two maps. This process is primarily intended to expose the infelicities of the historical in reference to “what is actually there.” As useful as such an approach can be, it is filled with many problematic assumptions. For instance, for the basemap to act as a control in the experiment, it must be treated as “accurate” and “objective,” while conversely, the historical map must be treated as a source that needs to be “fixed” or rectified.” This process overwhelmingly privileges, both technically and conceptually, maps of the past that, despite their differences, largely anticipate maps of the present, it being far easier and common to georectify historical maps that stick to an orthographic, Cartesian representation, often (but not exclusively) over large territories.
Thus, the irony of georectification is that it both seeks to expose the imperfections of the historical map while also reinforcing scholars’ fundamental assumptions about what a “map” is and should be. What then, do we do with “maps” that don’t fit into this narrow definition: itineraries, panoramas, views, and, as we came to call them, chorographies: three-dimensional renderings of urban spaces and their exurban and rural hinterlands? These images were omnipresent through the medieval and early modern period, but for the most part, they have been treated as decorative illustrations rather than studied as visual, (and above all) spatial texts.
Sandcastle Workflow
Project PIs, Phil Stern and Ed Triplett, both had longstanding interests in the ways premodern mapmakers represented places through visual systems other than top-down, orthographic maps, as well as the opportunities that computational methods present for the creation of new knowledge about the past. Individually, they had each been working in this space through Triplett’s "Book of Fortresses” project – a Wired Lab supported project that sought to disassemble and spatially reconstruct a 16th century book of drawings and plans of more than 55 fortresses and fortified towns on the Portuguese border with Castile – and Stern’s “Modeling Sovereignty” project – a database of places associated with the British empire that (as a side-effect more than a directed search) began collecting maps and views of colonial sites.
The “Sandcastle Workflow” grew directly out of the tension between layer-based Historical GIS practices, and the unsuitability of perspectival, “chorographic” maps for this paradigm. Inspired by recent great leaps in game engines and procedural modeling, as well as other virtual, augmented, and extended reality applications, we began with the exceedingly quixotic, if not audacious, idea to ask: what if we could do more than “rectify” images to a Cartesian plane? What if we could climb into historical views of cities and experience the worlds they represent? Could we design digital workflows and tools that reconstruct perspectival historical representations in 3D space, that would help us to analyze, understand, and experience those maps on their own terms?
The project begins with the assumption that previous (digital) methods for studying “Bird’s eye” or “Chorographic” views have either separated them out from spatial analysis altogether, or conversely, used modern base maps to stress how “inaccurate” or “pictorial” they are. With Sandcastle, the PIs set out to develop a much more malleable, sandbox-like digital environment that could translate these digital images into 3D environments. To accomplish this proposed workflow, all objects in a given view, “panorama” or chorography had to be traced and classified according to a set of controlled vocabularies that described each part geometrically (whether they be a south-facing building façade, or a west-facing water-wheel) as well as contextually – i.e. as residences, pieces of infrastructure, or religious buildings.
Collectively we had a great deal of experience with the challenges of data-driven projects, especially the difficulty of doing such work when the data themselves did not exist in ready-made data sets or sources. There was also no obvious short-cut for the kind of visual “annotation” we needed, such as OCR or HTR for visual sources. Machine learning (AI) options were intriguing for us, but identification of features in hand-drawn images remains a difficult task for this technology. Lacking a single “out of the box” solution, we came to the conclusion that what we needed was a multiple-software workflow, and a custom “Sandcastle Toolkit” to bridge between a 2D annotation system and a 3D game engine.
Knowing that this kind of application development requires a great deal of time, expertise and funding, we applied to the National Endowment for the Huminites Office of Digital Humanities, for a Phase II development grant. We also received two related grants from our own institution, one, the “Data+” project for summer 2020 and another through the Bass Connections program, to help us enlist a team of graduate and undergraduate students that would execute the bulk of the annotation work, research example chorographies and chorographers, test our theories in GIS software, and to begin developing the Sandcastle Toolkit.
Summer 2020
The first giant step was to create the data. The data-creation phase began in the summer of 2020 through a 10-week “Data +” project at Duke University. With five wonderful students, (one Ph.D. student and four undergraduates) we jumped into closely dissecting the iconography of the Book of Fortresses and two other complex, early seventeenth-century chorographic views—one of London, one of Lisbon. This allowed the team to come up with a classification schema for breaking down the views into their fundamental elements, and then identify those elements on the images. This was a painstaking process of meticulously drawing around each element, tagging and categorizing it as we went.
Yet, this was hardly as straightforward as it sounds. We needed to constantly refine and troubleshoot techniques for identifying and annotating those elements. For example, how were we to annotate “houses” when those they blend into block-long rows of houses? How was one to draw boundaries within more fluid elements, such as “water” or “sky”? How did one properly tag elements-within-elements, like people on a ship? The goal here was to identify just how the chorographer “viewed” the city, both to hopefully be able to “read” the map statistically—what proportion of the view is made of water? Churches? Trees?—and then ultimately, to translate the annotations into adjustable (procedural) 3D models. Along the way, we confirmed that image annotation was also an extremely useful pedagogical tool that forced all of us to do both intensive and extensive close looking. This close looking also revealed many interesting features of the maps we would not have noticed otherwise.
Yet, if Max Weber famously thought of politics as the “strong and slow boring of hard boards,” he never met image annotation. He also, to the best of our knowledge, had not anticipated COVID-19. The very same week the PIs were scheduled to have a meeting with a few collaborators to develop an annotation system in March 2020, the world shut down. We were now faced with radical revisions to our plan, which would change the trajectory of the project in crucial ways.
Our vision of students huddled around high-powered computers in an on-campus lab was no more. We needed to accommodate students working from home on a wide range of expertise, devices, and even internet speeds. We also needed the output to be nimble enough to allow us to export our data and port it elsewhere if need be, all the while facing unprecedented personal and professional challenges. And, of course, as it turns out, no ready-made tool existed even for this first step of annotation. After considering and experimenting with several options, we landed on Supervisely, a web application largely intended for training machine learning applications through visual annotation of many images. Through a great deal of experimentation and patience on the part of the entire team, the Data+ ended the summer with a solid and tested annotation process, as well as a great deal of raw data on both the Book of Fortresses, London, and Lisbon.
Meanwhile, a related team of graduate and undergraduate students, some simply volunteering their time, undertook critical preliminary research for us, developing databases of “chorographic” images and their archival and print locations, doing some analysis of some larger collections, beginning environmental scans of similar projects, and piloting some techniques for enriching the annotation process with historical research on these places.
At the end of the summer, Eric Monson at the Duke Library’s Center for Data and Visualization Sciences developed an absolutely critical piece of the Sandcastle Workflow – a Python script that used the traced annotations from Supervisely to create folders full of hundreds of individual files that can be reassembled in “Houdini,” the procedural 3D modeling software we selected to build the Sandcastle Toolkit.
Fall 2020
We then took that experience and data into another audacious undertaking, a yearlong project-based course under the rubric of Duke’s Bass Connections program: “Mapping History: Seeing Pre-modern Cartography through GIS and Game Engines.” Of course, we could not just start where we had left off. Our team of nearly two dozen undergraduate and graduate students were coming from various disciplines and vastly different academic backgrounds. We were deeply committed to striking the right balance between the “deliverable” goals of a research team and the pedagogical goals of a university class. On top of this, we were still doing all of this in the terra incognita of remote and virtual learning and the ongoing stresses of the pandemic. Students and faculty were spread across time zones in the United States and abroad, working under varied conditions, and very little could be predicted from week to week. Our watchword was “flexibility.”
The Fall course thus offered an introduction to materials in the history and theory of cartography, medieval and early modern urbanization, and research methods in history, art history, architectural history and others, as well as critical approaches to digital mapping, procedures of map annotation and markup, and Python-based techniques for importing markup data into a procedural modeling environment (Houdini). Students also helped troubleshoot and develop the project workflow as well by using this process to research, annotate, and analyze their own map drawn from medieval or early modern European and European colonial history.
It was a lot to expect of these students, even under normal circumstances. The team, however, rose to the challenge. We produced an extended annotated bibliography of related sources trained (with immense help from students who continued from the Data+ project) on the Supervisely annotation method, and began developing the 3D translation code networks in Houdini. In pairs, students also completed final project “mapographies” on eight different chorographic views, which they had selected to analyze, many of which came from the famous late sixteenth century atlas Civitates Orbis Terrarum.
Spring 2021
In the Spring semester, we pivoted away from a focus on the individual cities, and allowed students to specialize in the research tasks that suited their intellectual interests and skills best. After 4 additional weeks devoted to completing the annotation work on the chorographic views that were selected in the Fall, we broke the team into 5 sub-groups: a "mapography" team, a "historiography and theory" team, a "GIS" team, a "procedural modeling" team, and a "web visualization" team.
The “mapography” team collected additional chorographies related to our subset of annotated views from Lisbon, London, Ceuta, Aden, Goa, Marburg, Nice, Istanbul & Turin, and conducted historical research on the creators of those views, especially Braun and Hogenberg and their Civitates Orbis Terrarum.
The historiography and theory team began our project glossary by researching the meaning of problematic terms related to the project, such as “Chorography,” “Bird’s-eye-view,” “Map,” and “Panorama.” This team also conducted their own independent research on topics within the field of historical chorography, such as Ottoman cartography, how to “read a map” the relationship between religion and cartography, definitions of chorography, and iconography on chorography (relating to structures, like Hospitals, and movement (like ships).
The GIS team used multiple methods to illustrate the challenge of contextualizing the 9 chorographies in relation to a modern basemap. The primary mode of representing the maps spatially in the GIS was colloquially called “billboarding” – a technique that was first used for the Book of Fortresses project by Dr. Triplett. This process refuses to warp the historical views via georectification, choosing instead to represent them in a imaginary vantage point that approximates what is drawn by the chorographer.
The procedural modeling team used the Houdini procedural modeling software to develop the “Sandcastle Toolkit” – a plugin for the Unreal or Unity game engine that is the essential bridge in middle of the sandcastle workflow between traced and annotated iconography in Supervisely, and an explorable 3D environment in the Unity or Unreal game engine. At the end of the spring semester, the students developed techniques that took a series of inputs and converted them into the beginnings of a 3D scene with procedural houses, curving walls, and round, rectangular or polygonal towers.
The visualization and web development team was tasked with an overhaul of the “Sandcastle3D.org” website. This team began with an environmental scan of other project websites that dealt with historical maps and views and researched additional data visualization techniques that could help us interrogate our annotation data. They then moved on to developing wireframes of the new site and trained themselves in Webflow – a content management and web development application that allowed them to develop a unique site to their specifications. Finally, this team gathered textual and visual assets from the other sub-teams and developed a working draft of what has become the Sandcastle3D.org website.
Summer 2021
After a year of grueling data collection and research, we are now in the final stage of this overlapping, complex project that blends the Book of Fortresses, Sandcastle, and the “Mapping History” Bass Connections project. Over the summer of 2021, we plan to double down on the development of the Sandcastle Workflow through extended and concentrated work in Houdini and the Unreal game engine. Through the “Code +” program at Duke, 6 new team members will spend 10 weeks working full time on the Sandcastle toolkit in Houdini during the summer of 2021.
As any of our Bass Connections students will testify to, the learning curve for Houdini can be steep. Nevertheless, Houdini offered flexibility, a node-based development environment, and easy connection to Unity, Unreal, or other CAD software. This makes it an excellent “bridge” to translate the hard work we’ve already done in Supervisely into 3D environments that obey the “rules” set by the creators of our chorographies. We are looking forward to this more intensive phase of Houdini development for many reasons – not least of all because we are so eager to “climb into” these historical views — but mostly because at the end of Spring 2021, we finally began to see our first images from the Book of Fortresses raise themselves up off of the 2D image plane.