Friday, August 25, 2006
The originality problem
As the "easier" problems with ALG (grammaticality, surface realization, form) come closer to being solved, we're left with the fundamental problem, the essential problem, of how to make new texts seem original. Poetry generators produce boring work, not the first time, but somewhere between the 5th and 100th time. Simply put, an artificial author's poems start to all look alike. Jump over to my ETC site and see. Erica's work is full of participials and personal pronoun subjects and objects--so many that, even though ETC is not template based, it starts to appear to be.
Many electronic text products rely on randomization or weights to decide what trope or form they should use at a given point in the poem. ETC's abstraction index is an example. That index is really a probability, not 0-100, but 0-1. For lines other than questions, the software simply does a weighted selection on the index and if it comes back true, writes a fragment. Over time an abstraction index of 75 will result in 3/4 of the lines being fragments. That's just way too predictable.
If you read a collection of Wallace Steven's poems, you find way too many references to snow. It works in "The Snow Man," but after a very brief while, becomes tedious. Exaggerate the effect via the machine and the results pretty much suck.
I'm beginning to think that ALG systems, to be effective need another component in addition to the standard document planner, micro-planner, and surface realizer: a trope and word choice memory. A way for the machine to say to itself, Damn! That sounds really cool, but I used it in another poem--I can't use it here. But thinking that and designing it (let alone implementing it) are quite different things.
# posted by Jim @ 8:55 AM 
Comments:
<< Home
Ah yeah. This Way Madness Lies. At least this is my personal experience, from two years of trying to out-code the Incompleteness theorems, in order to achieve "machines with originality". I'm not using a metaphor when I say that I went mad in the process, and barely rescued myself. It's known that Gödel went mad himself, and I still can't shake off the thought that the suicidal deaths of AI researchers Chris McKinstry and Push Singh are related to this sort of madness: the madness that comes from staring too long and hard into the abyss that is "mathematical infinity", and trying to compute the provably uncomputable, because it just can't be impossible! "Dread" . . . that's the word.
To exemplify, I'll just take up your "trope/word memory" idea. It'll start innocently enough: I'll figure that what I'll have to do is include any combination of words from the source that was used in an output in a "used" list, then match any new output candidates against this list, drop them if they're already on it, and try the next one instead. Pretty soon, I'll find that memory usage and processing time go through the roof (I'll compress out the idea of compression here, as it makes no difference in principle). Also, if I have any significant amount of users, and am strict about my "no repetition" policy (i.e. not just "no repetition with regards to a particular user", but "no repetition, period"), I'll consume my source text with an amazing speed. But what's probably worst, from an aesthetic POV, is that repetition is a very valuable tool writers can use (one simple example: the Rule of Threes).
OK, so I vow to augment my algorithm so that it uses the RoT. My results become different, but after a while, just as boring as those I got when I started. So I code more rules about how to use the RoT, and rules about how to use the rules that govern the use of the RoT, and on and on. Sho'nuff, some of my outputs are excellent now. OTOH, a seemingly equal number of my outputs are worse than ever! Like an untalented barber who tries to give his client a properly symmetrical haircut, I'm working here, and there, and everywhere, and every time I look at what I've done, the offending bulge is at a different place.
The barber has the relative advantage that there's a natural limit to what he can do - at one time, he'll decide to just shave it all off, and live with the consequences. If I'm trying to generate an infinite amount of meaningful text from a finite amount of coded information, I'm not going to end up at somebody's scalp. Rather, every single time that I think I've finally found The Rule That Rules All Rules, it turns out that there's at least one exception, just as Gödel predicted. And if I can't pull outta there in time, I'm doomed. I mean it.
Fortunately for me, I did manage to pull out in time, and it just became one of the most craziest experiences I ever had, and certainly the most dreadful one. It also made me delve into the relevant math afterwards, in order to find out whether I'm just too stupid, or whether the phenomenon is generalizable. After understanding Algorithmic Information Theory, I'm now convinced it's the latter: an axiom-based system cannot increase its own complexity, and thus, cannot generate information (in the sense of Shannon's definition of "information", that is, "new stuff" which goes beyond the system's axioms). All of the "originality" that a machine-based text generator might display has to be hand-coded into the system by human writers, whose operations, as I'm convinced now, are not based on doing discrete math, i.e. computation.
This has shown to be a very unpopular, or even taboo-violating, thing to say in today's AI research as well as e-lit circles, and since I don't have anything riding on being right (besides my own mental health), I'm not trying to do much evangelizing on it. But I dig your project, and I dig what you have to say about the use of goto, and about the code being the programmer, so I hope that a note about my view of "machine originality" might at least be interesting to you. I'm not trying to discourage you in your efforts (I certainly didn't discourage myself by going through the process I described), just to point out that it might be more efficient to heed the relative limitations of digital computers (as compared with those of people), and that those limitations are probably more limiting than we would have expected, or would have liked.
Dirk
Post a Comment
To exemplify, I'll just take up your "trope/word memory" idea. It'll start innocently enough: I'll figure that what I'll have to do is include any combination of words from the source that was used in an output in a "used" list, then match any new output candidates against this list, drop them if they're already on it, and try the next one instead. Pretty soon, I'll find that memory usage and processing time go through the roof (I'll compress out the idea of compression here, as it makes no difference in principle). Also, if I have any significant amount of users, and am strict about my "no repetition" policy (i.e. not just "no repetition with regards to a particular user", but "no repetition, period"), I'll consume my source text with an amazing speed. But what's probably worst, from an aesthetic POV, is that repetition is a very valuable tool writers can use (one simple example: the Rule of Threes).
OK, so I vow to augment my algorithm so that it uses the RoT. My results become different, but after a while, just as boring as those I got when I started. So I code more rules about how to use the RoT, and rules about how to use the rules that govern the use of the RoT, and on and on. Sho'nuff, some of my outputs are excellent now. OTOH, a seemingly equal number of my outputs are worse than ever! Like an untalented barber who tries to give his client a properly symmetrical haircut, I'm working here, and there, and everywhere, and every time I look at what I've done, the offending bulge is at a different place.
The barber has the relative advantage that there's a natural limit to what he can do - at one time, he'll decide to just shave it all off, and live with the consequences. If I'm trying to generate an infinite amount of meaningful text from a finite amount of coded information, I'm not going to end up at somebody's scalp. Rather, every single time that I think I've finally found The Rule That Rules All Rules, it turns out that there's at least one exception, just as Gödel predicted. And if I can't pull outta there in time, I'm doomed. I mean it.
Fortunately for me, I did manage to pull out in time, and it just became one of the most craziest experiences I ever had, and certainly the most dreadful one. It also made me delve into the relevant math afterwards, in order to find out whether I'm just too stupid, or whether the phenomenon is generalizable. After understanding Algorithmic Information Theory, I'm now convinced it's the latter: an axiom-based system cannot increase its own complexity, and thus, cannot generate information (in the sense of Shannon's definition of "information", that is, "new stuff" which goes beyond the system's axioms). All of the "originality" that a machine-based text generator might display has to be hand-coded into the system by human writers, whose operations, as I'm convinced now, are not based on doing discrete math, i.e. computation.
This has shown to be a very unpopular, or even taboo-violating, thing to say in today's AI research as well as e-lit circles, and since I don't have anything riding on being right (besides my own mental health), I'm not trying to do much evangelizing on it. But I dig your project, and I dig what you have to say about the use of goto, and about the code being the programmer, so I hope that a note about my view of "machine originality" might at least be interesting to you. I'm not trying to discourage you in your efforts (I certainly didn't discourage myself by going through the process I described), just to point out that it might be more efficient to heed the relative limitations of digital computers (as compared with those of people), and that those limitations are probably more limiting than we would have expected, or would have liked.
Dirk
# posted by 
: 8:10 AM
: 8:10 AM << Home
