Paleography refers to the study of all forms of ancient and historical writing, of which musical notation is, to my mind, one of the most interesting. Imagine the problem: here you are faced with the problem of keeping an accurate record of, say, the unaccompanied plainchant of the monks, and all of a sudden they start adding voices and creating complicated music. And all you have had to work with were some little wiggly lines and dots that probably originally were nothing more than accents and punctuation marks accompanying ordinary writing. After some early and very ambiguous examples from the 9th century, one of the earliest notations that is clear enough for us to transcribe into modern notation is an example from the Abbey of Saint Martial in Limoges. The manuscript shown below is a two-part piece of music, an organum, written in Aquitanian neumes. The staff lines are just scratched in the parchment and have been drawn in so they can be visible in the facsimile:
The text is "Viderunt omnes", a Christmas gradual. Here is what it looks like in modern notation:
As you can see, while we can be fairly clear about what the actual pitches are, the rhythms are anyone's guess because the original notation does not show any rhythmic values. Figuring out how to do that took the next few hundred years! One of the early attempts at a solution was the idea of rhythmic "modes"--basically small cells of various combinations of long and short note values. Here is a sample of what that kind of notation looked like. This is a piece that I studied in a paleography course and you can see some of my faint pencil notes on the scan showing what lines are what notes and where the rhythmic modes start. I have also numbered the iterations of the modes:
And here is how the beginning looks in transcription:
Just to show you how complex some of the notation systems were, here is a piece by Jacopinus Selesses that uses white (meaning hollow notes), black (meaning filled in notes) and red (meaning colored red) notation:
All these examples are from the Apel book, still the most thorough treatment even now, though there is certainly a lot of recent research on the problems.
So that is my very brief and sketchy introduction to musical paleography. The book by Apel is over 400 pages long and very detailed.
This might be a performance of the first example above, but I'm not sure. It sounds like the notes, but as the rhythms can be different for each performance, I'm not positive.
And here is another organum from the St. Martial manuscript: