Other people are making the same point I'm about to make, but I'm going to try to clarify it anyway because, y'know, besides being a programmer, I'm also a technical writer, and I just have to scratch that itch.
Common Lisp source code (and the source code of its immediate ancestors) is not made of text strings. It's made of S-expressions, which are made of cons cells, symbols, numbers, and so on.
A text file of "Lisp source code" does not actually contain Lisp source code. It contains a text-based serialization of Lisp source code. Other serializations are possible (and there are things you can do in a Common Lisp repl to see some of them).
The "read" in "read-eval-print" means "deserialize the text into the source data that it's meant to represent".
This point is not trivial pedantry because the full power of the Lisp language is available to the read process, and can be brought to bear on how reading is done and what happens when you do it. Compilers for other languages certainly do read text strings and convert them into tree structures and so forth, but the difference is that those data structures are private to the compiler; the data structures that Lisp reads into are standard parts of Lisp's public API, as are the read function, the compile function, the eval function, the print function, and so on. It's all on the table for you to work with.
The same is true of the disposition of the s-expressions produced by the read process; you have an opportunity to bring the whole of the Lisp language to bear on those s-expressions before they are ever passed to (compile or) eval. Then, once again, what eval produces is S-expressions, and those, not strings, are passed to the print function. You once again have the opportunity to intervene in the process that produces the text serialization.
It so happens that I've spent the past six months working on an AI machine-control system written in Common Lisp, and every one of these capabilities was an important part of the work we were doing.
You are missing that by "reading" the code in Lisp you don't also "execute it", you can just read it and have a nice tree data structure. If you have any reason to eval it, go ahead, but that would be a bad idea for a log or a config file (unless you want a Turing complete configuration or logging DSL... hopefully you know this is a bad idea!)
Most other languages don't have an obvious way to just parse a file of source code without actually executing it: `#include <stdio.h>` or `import mycoolpythonlib` or `require __DIR__ . '/goolib.php'` both parse and execute/compile/link the file, so yes, any config/log file becomes a code file and "arbitrary code" gets executed, but it doesn't have to be this way.
(Now, you have things like "import hooks" and access to the AST in Python for example if you want a tree of the imported file, but nothing "simple" and obvious. And this is the thing actually: as much as people consider Lisps too complex and powerful, the nice thing is how "obvious" and "simple" they make things like this - by just parsing sexp tree and manioulating nested lists instead of inventing a file format like XML with its transformation languages, query languages and so on, and all the tools to support them, and all the man-years (or is it decades? ...centuries?) spend to code and maintain them ...just think how much needless effort could have been spared.)
Python and Ruby use flat strings as input to eval. That's an entirely different thing. To do ANY useful code transformations one needs to parse it. Lisp code already comes in a hierarchical token tree.
Ruby and Python represent code as text. That's what Lisp does, too. One has a file of code. But Lisp has something which neither Python and Ruby do: the text is actually a data structure: lists, strings, numbers, vectors, symbols. Lisp provides a function READ.
Or use infix conversion during reading. Write your code with sections of a more conventional syntax:
CL-USER 80 > (defun hypot (a b)
"Compute the length of the hypotenuse of a right triangle
with sides A and B."
#I( sqrt(a^^2 + b^^2) ))
HYPOT
CL-USER 81 > (hypot 1 2)
2.236068
Just put a quote in front of the expression and enter it at the repl -> the result is the transformed code.
CL-USER 84 > '#I( sqrt(a^^2 + b^^2) )
(SQRT (+ (EXPT A 2) (EXPT B 2)))
There is nothing like this in Ruby or Python.
If you look at a Lisp interpreter, it actually executes Lisp data as code. Macros transform Lisp data as code. This is far far away from what Ruby or Python do. Python provides some access to its AST - that's also far away from what Lisp does.
Lisp macros don't define irregularities. They do code transformations. Stuff where the language designer/implementor of, say, Python refuses to provide a control structure, you can do it yourself in Lisp in a few minutes or you can spend years to implement really cool stuff. The Common Lisp object system started as a large portable library (with only very little system dependent code), which code be loaded into Lisp and which the provided a very elaborate object system. This also included some clever macro code.
I think titles like 'The problem of Lisp' may need a bit more reflection and patience. What is new for you is old news for the Lisp community. The exact same problem report can be already read in Lisp books from the 1960s. It's a bit like blogging 'The world isn't flat!.;-)
> I'm aware of that, but text is a fairly convenient representation of Lisp data.
Some Lisp data does not have a textual representation, it might have a graphical representation or the textual representation may not be very helpful (Lisp data being a graph might be better displayed as a 2d or even 3d graph, than as a textual representation). The Symbolics UI of the REPL/Listener would allow you to interact with the 2d/3d graph as it were Lisp data, which it actually is underneath.
> In fact, I'm unsure what you mean by directly interacting with the data, as you can't have bytes fly from your fingertips, AFAIK.
GNU Emacs pushes characters around.
S-Edit manipulates S-expressions.
GNU Emacs lets you pretend that you edit s-expressions, by pushing characters in a buffer around. But you don't. All you do is push text around.
S-Edit directly manipulates the data. It's a structure editor.
Symbolics Genera uses 'presentations' to record for every output (graphical or not) the original Lisp data. When you interact with the interface, you interact with these data objects. For example in a Listener (the REPL) programs and the Listener itself display presentations, which then can be acted on. It also parses text into data objects and then runs the commands on the data objects and not on text.
SLIME provides a very simple and partial reconstruction of that - which is a nice feature.
> In addition, files are a pretty good metaphor as well: If you want to store your code as something textual, they're indispensible. And sure, you can use image storage and navigate in other ways, but Lisp isn't Smalltalk: The way Lisp is written isn't as unified, so that wouldn't work as well, AFAIK.
Smalltalk does not use data as representation for source code. It uses text as representation for source code.
Interlisp-D is more radical than Smalltalk 80. The Smalltalk editor edits text. The Interlisp-D editor S-Edit edits s-expressions as data.
Interlisp-D treats the files as a code database similar to Smalltalk, but the source code loaded remains data and you can run the program from that data directly via the Lisp interpreter. Smalltalk execution does not provide something like that. The so-called interpreter in Smalltalk executes bytecode. This is different from a Lisp interpreter, which works over the Lisp source data.
The combination of a s-expression editor with an s-expression interpreter (plus optional compilation) is very different, from what Smalltalk 80 did.
When you mark an expression in Smalltalk 80 and execute it via a menu command, then the expression gets compiled to bytecode and the bytecode interpreter then runs it.
If you mark an expression in S-Edit, the s-expression data is extracted from the data structure you edit and you can run that s-expression data with an interpreter, walking directly over that s-expression data.
I hate that. The half page of code is not Lisp. Not even close, and I'm not talking about a standard library. There is no parser in there. Eval is operating on Lisp data, not lisp source code. There is no support for efficient math, or strings, or anything really.
If that is lisp, then a byte code interpreter loop is every interpreted language: read_opcode, inc_pc, call a function indexed from a list indexed by opcode. It can be written in one line of C. This BTW translates to hardware a lot easier than the half page of Lisp. That one line of code by itself is also just as useless as the half page of Lisp.
It's still interesting, but people need to stop claiming it's Lisp defined in this tiny little block of code.
One of the inventions of McCarthy (and team) was that EVAL can be defined in the core language itself (!) and thus serve as a relatively simple model of computation. Thus the particular EVAL is important, not that source code can be executed at runtime by some api call to execute code.
That's part of the core of the language.
Sure: Many other aspects from Lisp can be found also in Lisp-derived languages and other languages. Like all or some of runtime code loading, symbolic expressions, using symbolic expressions to encode source code, saving and starting heap images, garbage collection, managed memory, source level interpreters/debuggers, self-hosted compilers, read-eval-print-loops, macros, fexprs, integrated interactive development environments, ...
In general I like this essay, but I do see how it left the impression that Lisp is about making all your data into executable programs. Most data would be read and written with the same functions that read and write programs, but the data would not be evaluated as a program. Among his examples, only configuration files seem suitable for that.
That being said, even if you did make log files into programs, there's never an issue of escaping text properly. The function that writes data always escapes strings so they'll read back in as the same string, not as any kind of structure.
Even for lisp, you would have to define an extended syntax for it and modify the parser to additionally read and store whitespaces, comments and newlines. This way you will be able to serialize your source to plaintext without loosing the original format.
Non text sourcecode also allows for way smarter tooling, tree diffs on the AST, graphical repls and improved rendering of sourcecode.
Compiling will not change READ into PARSE. READ is still there.
> Most CL implementations compile before calling eval
Many or a lot, but I'm not sure actually most.
There are quite a lot which use an Interpreter in the REPL, but where one can compile on demand. SBCL by default compiles. LispWorks for example does not.
> After it's reader is done with it, the code is represented with CL data structures.
Right, but the data structures carry no syntactic information. If you write (1 2 +) the reader accepts it and does not know that this is invalid syntax.
> AST and IR are all CL data structures.
True, but not necessarily like s-expressions...
See for example data structures the SBCL compiler uses:
2. How is a string read from the terminal fundamentally different in Lisp than in Python? How is
(eval (read-from-string "(anything here)"))
different from Python's
eval('anything(here)')
? Just because it's possible to build the arguments of eval without using strings in Lisp does not mean that they are somehow less arbitrary.
3. I don't see how Python's import is different, the SBCL runtime, for example, locates and reads the source file when you import it, compiles it to native code (if a cached copy is not available), and runs it.
4. I missed the (apparently) implicit "that easily fits into GCC's present compilation model" in your earlier comment. I'm have no arguments with that statement, and indeed, I don't think GCC would be a good place to work on static compilation for Python. Rather, my argument is with a claim (intended or not) that Python is just too dynamic to ever be compiled, for which I think Lisp compilers are a living counter-example. From the sbcl man page:
SBCL compiles by default: even functions entered in the read-eval-print
loop are compiled to native code, unless the evaluator has been explicitly
turned on. (Even today, some 30 years after the MacLisp compiler, people
will tell you that Lisp is an interpreted language. Ignore them.)
Evidently you are not familiar with how many Lisp compilers work. You don't need an interpreter anywhere, just a compiler.
Note that eval(), as much as it makes sense in the C language, can be implemented in C too (inject the text into a stub, compile it into a shared library, dlopen(), dlsym(), call it).
Which is why you're not allowed to use a Lisp interpreter or use any method of evaluating data as code. In this model the only thing that data can do is change which code paths run, not what they do.
Lisp is a fun language to program in. I learned Lisp after already being familiar with Java / Python and some other languages, which maybe made it even more beautiful.
One of my favourite pieces of code is a Lisp REPL in Lisp:
Classically, "code is data" is used upfront, in creating macros that manipulate Lisp code as data, before sending the result to the interpreter or compiler like normal functions and data. Ideally you develop your own Domain Specific Language that provides power in expression and execution for what you're trying to do.
As implied in other answers in the subthread, nowhere in this paradigm does random input data get treated as something that can safely be executed. READ in Lisp Machines is mentioned because reader macros like #. allow data in the form being read to be evaluated, which in this domain is an obvious big no-no.
As early as 1983, I think earlier, it was recognized that things like eval servers were a bad idea if accessible by the outside world.
1. I was replying to your statement that eval() required an interpreter. My antagonism was mostly in jest since I vaguely remembered your name, but I also believe in discussions without arguments from authority.
2. Lisp can eval a string that has been constructed at runtime. That's kind of the point of a repl, for example, which can be implemented by compiling each unit of input to native code before executing it.
3. The Python community shuns the use of eval. Are you really claiming that Python's eval is used in fundamentally more dynamic ways than Lisp's? Or is it that Lisp's eval usually operates on input that is constructed differently? Or that Lispers tend not to put eval in an inner loop, but some important Python programs do? I understand that there are attributes of Python that make compilation hard compared to a Lisp, but I'm not aware of something that makes it impossible or guarantees that compilation cannot be performant. Maybe I need to read chapter 6 of your thesis.
Common Lisp source code (and the source code of its immediate ancestors) is not made of text strings. It's made of S-expressions, which are made of cons cells, symbols, numbers, and so on.
A text file of "Lisp source code" does not actually contain Lisp source code. It contains a text-based serialization of Lisp source code. Other serializations are possible (and there are things you can do in a Common Lisp repl to see some of them).
The "read" in "read-eval-print" means "deserialize the text into the source data that it's meant to represent".
This point is not trivial pedantry because the full power of the Lisp language is available to the read process, and can be brought to bear on how reading is done and what happens when you do it. Compilers for other languages certainly do read text strings and convert them into tree structures and so forth, but the difference is that those data structures are private to the compiler; the data structures that Lisp reads into are standard parts of Lisp's public API, as are the read function, the compile function, the eval function, the print function, and so on. It's all on the table for you to work with.
The same is true of the disposition of the s-expressions produced by the read process; you have an opportunity to bring the whole of the Lisp language to bear on those s-expressions before they are ever passed to (compile or) eval. Then, once again, what eval produces is S-expressions, and those, not strings, are passed to the print function. You once again have the opportunity to intervene in the process that produces the text serialization.
It so happens that I've spent the past six months working on an AI machine-control system written in Common Lisp, and every one of these capabilities was an important part of the work we were doing.
reply