newfangle
Sam Liddicott
August 2009
Introduction
Newfangle is a tool for newfangled literate programming. Newfangled is defined as New and often needlessly novel by TheFreeDictionary.com.
In this case, newfangled means yet another new and improved method for literate programming.
Literate Programming has a long history starting with the great Donald Knuth himself, whose literate programming tools seem to make use of as many escape sequences for semantic markup as TeX itself.
Norman Ramsey wrote the noweb set of tools (notangle, noweave and noroots) and helpfully reduced the amount of magic character sequences to pretty much just
<<
and
@
, and in doing so brought the wonders of literate programming within my reach.
While using the LyX editor for LaTeX editing I had various troubles with the noweb tools, some of which were my fault, some of which were noweb's fault and some of which were LyX's fault.
Noweb generally brought literate programming to the masses through removing some of the complexity of the original literate programming, but this would be of no advantage to me if the LyX / LaTeX combination brought more complications in their place.
Newfangle was thus born --- as an awk replacement for notangle, adding some important features, like better integration with LyX and LaTeX, multiple output format conversions, and fixing notangle bugs like indentation when using -L for line numbers.
Significantly, newfangle is just one program which replaces various programs in Noweb. Noweave is done away with and implemented directly as LaTeX macros, and noroots is implemented as a function of the untangler newfangle.
Newfangle is written in awk for portability reasons, awk being available for most platforms. A python conversion will probably be attempted for the benefit of LyX. (Hasn't anyone implemented awk in python yet?)
As an extension to many literate-programming styles, Newfangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates.
Todo
-
^^ is always going to be a problem, see texbytopic 1.2.2 (Work out what I meant by this).
-
copy over up to date Makefile guide from noweb-lyx document
-
Make chunk-name settings only apply to chunks with that name
-
indent of multi-line chunks may be mode dependant (i.e. not in string literals)
-
add support for other commands in =<...>, starting with \label which takes the line-number within the chunk, and maybe should also take the chunk name/page
-
cant have listing inside a ruled box
-
when a parameterized chunk is included as well as the #line emission, say what the paremeters were for that invocation.
-
\chunkref[3]{preamble} to include a certain chunk needs to work in newfangle.awk instead of failing to be recognized at all
-
make in-listins labels track the chunk ref too, and make \chunref{[2],thing}> resolve to 41c (or d, or whatever chunk the 2nd chunk of thing is
-
\chunkref in text needs a trailing space maybe, it keeps butting up to the next word
-
because the white-space indent is output by the parent chunk, the #line is that of the parent chunk. White space indents must be passed to the child chunk
License
Newfangle is licensed under the GPL 3
1 (or later). This doesn't mean that you can't use or distribute newfangle with sources of an incompatible license, but it means you must make the source of newfangle available too.
gpl3-copyright,language=
newfangle - fully featured notangle replacement in awk
Copyright (C) Sam Liddicott 2009
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Part 1. Using Newfangle
1 Running Newfangle
Newfangle is a replacement for noweb, which consists of
notangle
,
noroots
and
noweave
.
Like
notangle
and
noroots
it can read multiple named files, or from stdin.
1.1 Listing roots
The -r option causes newfangle to behave like noroots.
newfangle -r filename.tex
will print out the newfangle roots of a tex file.
Unlike the
noroots
command, the roots are not enclosed in
<<name>>
, unless at least one of the roots is defined using the
notangle
style
<<name>>=
.
Also, unlike noroots, it prints out all roots --- not just those that are not used elsewhere. I find that a root not being used, doesn't make it particularly top level. My convention is that top level roots to be extracted begin with
./
and have the form of a filename.
1.2 Extracting roots
notangle's
-R
and
-L
options are supported.
The standard way to extract a file would be:
newfangle -R./Makefile.inc newfangle.tex > ./Makefile.inc
Unlike the
noroots
command, the
-L
option does not break indenting; also the
-L
option does not interrupt (and break) multi-line C macros --- or indeed any line ending with a backslash. This does mean that sometimes the compiler might calculate the source line wrongly when generating error messages in such cases, but there isn't any other way around if multi-line macros include other chunks.
1.3 Formatting source in LaTeX
The noweave replacement is a set of LaTeX macros dependant upon noweb.sty, and which can be included with:
\usepackage{newfangle.sty}
The LaTeX macros are shown in section
13.2?, and are part of a LyX module file
newfangle.module
, which automatically includes the macros in the document pre-amble when the newfangle LyX module is used.
Because the noweave replacement is impemented in LaTeX, there is no processing stage required before running the
latex
command. LaTeX may need running two or more times, so that the code chunk references can be fully calculated.
The
noweb.sty
package is required as it is used for formatting the code chunk captions
The
listings.sty
package is also required, as it is used for formatting the code chunks themselves.
The
xargs.sty
package is also required.
2 Literate Programming with Newfangle
Todo. Should really follow on from a part-0 explanation of what literate programming is.
3 Using Newfangle with LyX
3.1 Setting up Lyx
3.1.1 Installing the LyX module
Copy
newfangle.module
to your LyX layouts directory, which for unix users will be
~/.lyx/layouts
You will need to reconfigure LyX by clicking Tools ? Reconfigure, and then re-start LyX.
3.1.2 Configuring the build script
Make sure you don't have a conversion defined for Lyx -> Program
From the menu Tools ? Preferences, add a conversion from Latex(Plain) -> Program as:
set -x ; newfangle -Rlyx-build $$i |
env LYX_b=$$b LYX_i=$$i LYX_o=$$o LYX_p=$$p LYX_r=$$r bash
(But don't cut-n-paste it from this document or you'll be pasting a multi-line string which will break your lyx preferences file).
I hope that one day, LyX will set these into the environment when calling the build script.
You may also want to consider adding options to this conversion...
parselog=/usr/share/lyx/scripts/listerrors
...but if you do you will lose your stderr
.
Now, a shell script chunk called
lyx-build
will be extracted and run whenever you choose the Document ? Build Program menu item.
The lyx-build script for this document is in section
? and on a unix system will extract
newfangle.module
and the
newfangle
awk script.
3.1.3 Preparing your Lyx document
It is not necessary to base your literate document on any of the original LyX literate classes; so select a regular class for your document type.
Add the new module Newfangle Literate Listings and possibly also Logical Markup.
In the drop-down style listbox you should notice a new style defined, called Chunk.
When you wish to insert a literate chunk, you enter it's plain name in the Chunk style, instead of the older method that used
<<name>>=
type tags. Following the chunk name, you insert a listing with: Insert ? Program Listing.
Inside the white listing box you can type (or paste using shift+ctrl+V) your listing. There is not need to use ctrl+enter at the end of lines as with some older LyX literate techniques --- just press enter as normal.
3.1.3.1 Customising the listing appearance
In the final document, the code is formatted using the lstlistings package. The chunk style doesn't just define the chunk name, but can also define any other chunk options supported by the lstlistings package
\lstset
command. In fact, what you type in the chunk style is raw latex. If you want to set the chunk language without having to right-click the listing, just add
,lanuage=C
after the chunk name.
Of course you can do this by editing the listings box advanced properties by right-clicking on the listings box, but that takes longer, and you can't see at-a-glance what the advanced settings are while editing the document; also advanced settings apply only to that box --- the chunk settings apply through the rest of the document
.
3.1.3.2 Global customisations
As lstlistings is used to set the code chunks, it's
\lstset
command can be used in the pre-amble to set some document wide settings.
If your source has many words with long sequences of capital letters, then
columns=fullflexible
may be a good idea, or the capital letters will get crowded. (I think lstlistings ought to use a slightly smaller font for captial letters so that they still fit).
The font family
\ttfamily
looks more normal for code, but has no bold (unless luximono is used, but it doesn't work for me); so I use
\color{darkgreen}
for my keywords. With
\ttfamily
,
columns=fullflexible
is used or the wrong letter spacing is used.
In my LeTeX pre-amble I usually specialise my code format with:
document-preamble,language=tex
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0,0.5,0}
\lstset{numbers=left, stepnumber=5, numbersep=5pt, breaklines=false,
basicstyle=\footnotesize\ttfamily,
keywordstyle=\color{darkgreen},
numberstyle=\tiny,language=C,columns=fullflexible,
numberfirstline=true
}
4 Newfangle with Makefiles
This chapter needs revising
Here we describe a Makefile.inc that you can include in your own Makefiles, or glue as a recursive make to other projects.
The Makefile.inc described here was put together for a Samba4 vfs module, but can be used in any Make project, including automake projects.
4.1 A word about makefiles formats
Whitespace formatting is very important in a Makefile. The first character of each command line must be a TAB.
target: pre-requisite
-> action
-> action
But a TAB is pretty hard to enter into most of the Lyx formats and insets I've been using. An alternative is to use a semi-colon after the pre-requisite, and a backslash at the end of each line (except the last). Then any whitespace (or none) can prefix each action.
target: pre-requisite ; \
__action \
__action
This is the style that we use and it works pretty well for GNU make at least.
We also adopt a convention that code chunks whose names beginning with ./ should always be automatically extracted from the document. Code chunks whose names do not begin with ./ are for internal reference. (This doesn't prevent such chunks from being extracted directly).
4.2 Boot-strapping the extraction
4.2.1 Using a Makefile
It seems convenient to have the makefile extract or update the C source files as part of it's operation. It also seems convenient to have the makefile itself extracted from this document.
It would also be convenient to have the code to extract the makefile from this document to also be part of this document, however we have to start somewhere and this unfortunately requires us to type at least a few words by hand to start things off.
Therefore we will have a minimal root fragment, which, when extracted, can cope with extracting the rest of the source. perhaps with this shell script, which could be called autoboot.
*
#! /bin/sh
MAKE_SRC="${1:-${NW_LYX:-../../noweb-lyx/noweb-lyx3.lyx}}"
MAKE_SRC='dirname "$MAKE_SRC"'/'basename "$MAKE_SRC" .lyx'
NOWEB_SRC="${2:-${NOWEB_SRC:-$MAKE_SRC.lyx}}"
lyx -e latex $MAKE_SRC
newfangle -R./Makefile.inc ${MAKE_SRC}.tex \
| sed "/NEWFANGLE_SOURCE=/s/^/#/;T;aNOWEB_SOURCE=$NEWFANGLE_SRC" \
| cpif ./Makefile.inc
make -f ./Makefile.inc newfangle_sources
The general Makefile can be invoked with ./autoboot and can also be included into any automake file to automatically re-generate the source files.
The autoboot can be extracted with this command:
lyx -e latex newfangle.lyx && \
newfangle newfangle.lyx > ./autoboot
This looks simple enough, but as mentioned, newfangle has to be had from somewhere before it can be extracted.
When the lyx-build chunk is executed, the current directory will be a temporary directory, and
LYX_SOURCE
will refer to the tex file in this temporary directory. This is unfortunate as our makefile wants to run from the project directory where the Lyx file is kept.
We can extract the project directory from $$r, and derive the probable Lyx filename from the noweb file that Lyx generated.
lyx-build-helper
PROJECT_DIR="$LYX_r"
LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
TEX_DIR="$LYX_p"
TEX_SRC="$TEX_DIR/$LYX_i"
And then we can define a lyx-build fragment similar to the autoboot fragment
lyx-build
#! /bin/sh
=<\chunkref{lyx-build-helper}>
cd $PROJECT_DIR || exit 1
#/usr/bin/newfangle -filter ./notanglefix-filter \
# -R./Makefile.inc "../../noweb-lyx/noweb-lyx3.lyx" \
# | sed '/NOWEB_SOURCE=/s/=.*/=samba4-dfs.lyx/' \
# > ./Makefile.inc
#
#make -f ./Makefile.inc newfangle_sources
4.3 Extracting Sources
4.3.1 Including Makefile.inc
Makefile.inc will cope with extracting all the other source files from this document and keeping them up to date.
It may also be included by a Makefile or Makefile.am defined in a Lyx document to automatically deal with the extraction of source files and documents.
A makefile has two parts; variables must be defined before the targets that use them.
./Makefile.inc
=<\chunkref{Makefile.inc-vars}>
=<\chunkref{Makefile.inc-targets}>
We first define
NOWEB_SOURCE
to hold the name of this Lyx file.
Makefile.inc-vars
LYX_SOURCE=
LITERATE_SOURCE=$(LYX_SOURCE)
4.3.2 Recursive use of Makefile.inc
The makefile glue described here is used when building Samba4 vfs modules.
If you are defining a module of an existing program you may find it easier to use a slight recursive make instead of including the makefile directly. This way there is less chance of definitions in Makefile.inc interfering with definitions in the main makefile, or with definitions in other Makefile.inc from other noweb modules.
The glue works by adding a .PHONY target to call the recursive make, and adding this target as an additional pre-requisite to the existing targets.
In this example, the existing build system already has a build target for
example.o
, so we just add another pre-requisite to that. In this case we use
example.tex.stamp
as a pre-requisite, the stamp file's modified time indicating when all sources were extracted.
makefile-glue
$(example_srcdir)/example.o: $(example_srcdir)/example.tex.stamp
The target for this new pre-requisite is generated by a recursive make using Makefile.inc which will make sure that the source is up to date, before it is built by the main projects makefile.
makefile-glue
$(example_srcdir)/example.tex.stamp: $(example_srcdir)/example.tex ; \
cd $(example_srcdir) && \
$(MAKE) -f Makefile.inc newfangle_sources
We can do similar glue for the docs, clean and distclean targets. In this example our build system is using a double colon for these targets, so we use the same in our glue.
makefile-glue
docs:: docs_example
.PHONY: docs_example
docs_example:: ; cd $(example_srcdir) && \
$(MAKE) -f Makefile.inc docs
clean:: clean_example
.PHONEY: clean_example
clean_example: ; cd $(example_srcdir) && \
$(MAKE) -f Makefile.inc clean
distclean:: distclean_example
.PHONY: distclean_example
distclean_example: ; cd $(example_srcdir) && \
$(MAKE) -f Makefile.inc distclean
We could do similarly for install targets to install the generated docs.
4.3.3 Converting from Lyx to LaTeX
The first stage will always be to convert the Lyx file to a LaTeX file; this must be so not only because newfangle needs to to run on a TeX file, but also because the Lyx command server-goto-file-line
insists that the line number provided is a line in the TeX file, and always reverse maps this to derive the line in the Lyx docment.
The command [[lyx -e literate noweb-lyx.lyx]] will produce [[noweb-lyx.nw]] a tex file, so we define the noweb target to be the same as the Lyx file but with the .nw extension.
Makefile.inc-vars
TEX_SOURCE=$(LYX_SOURCE:.lyx=.tex)
Makefile.inc-targets
$(TEX_SOURCE): $(LYX_SOURCE) ;\
lyx -e latex $<
clean_tex: ; rm -f -- $(TEX_SOURCE)
4.3.4 Extracting Program Source
The program source is extracted using newfangle, which is designed to operate on a LaTeX document.
Makefile.inc-vars
NEWFANGLE_SOURCE=$(TEX_SOURCE)
The Lyx document can result in any number of source documents, but not all of these will be changed each time the Lyx document is updated. We certainly don't want to update the timestamps of these files and cause the whole source tree to be recompiled just because the Lyx document was edited.
To solve this problem we use a stamp file which is always updated each time the sources are extracted from the LaTeX document. If the stamp file is older than the LaTeX document, then we can make an attempt to re-extract the sources.
Makefile.inc-vars
NEWFANGLE_SOURCE_STAMP=$(NEWFANGLE_SOURCE).stamp
Makefile.inc-targets
$(NEWFANGLE_SOURCE_STAMP): $(NEWFANGLE_SOURCE) \
$(NEWFANGLE_SOURCES) ; \
echo > $(NEWFANGLE_SOURCE_STAMP)
clean_stamp: ; rm -f $(NEWFANGLE_SOURCE_STAMP)
clean: clean_stamp
4.3.5 Extracting C sources
We compute
NEWFANGLE_SOURCES
to hold the names of all the C source files defined in this document. We compute this only once, by means of := in assignent. The sed deletes the any << and >> which may surround the roots names (for noroots compatibility).
As we use chunk names beginning with ./ to denote top level fragments that should be extracted, we filter out all fragments that do not begin with ./
Makefile.inc-vars
NEWFANGLE_PREFIX:=\.\/
NEWFANGLE_SOURCES:=$(shell \
newfangle -r $(NEWFANGLE_SOURCE) |\
sed -e 's/^[<][<]//;s/[>][>]$$//;/^$(NEWFANGLE_PREFIX)/!d' \
-e 's/^$(NEWFANGLE_PREFIX)/\.\//' )
@
Makefile.inc-targets
.PHONY: echo_newfangle_sources
echo_newfangle_sources: ; @echo $(NEWFANGLE_SOURCES)
We define a convenient target called
newfangle_sources
to re-extract the source if the LaTeX file has been updated.
Makefile.inc-targets
.PHONY: newfangle_sources
newfangle_sources: $(NEWFANGLE_SOURCE_STAMP)
And also a convenient target to remove extracted sources.
Makefile.inc-targets
.PHONY: clean_newfangle_sources
clean_newfangle_sources: ; \
rm -f -- $(NEWFANGLE_SOURCE_STAMP) $(NEWFANGLE_SOURCES)
This
if_extension
macro takes 4 arguments: the filename (1), some extensions to match (2) and a some shell command to return if the filename matches the exentions (3), or not (4).
Makefile.inc-vars
if_extension=$(if $(findstring $(suffix $(1)),$(2)),$(3),$(4))
For some source files like C files, we want to output the line number and filename of the original LaTeX document from which the source came.
To make this easier we define the file extensions for which we want to do this.
Makefile.inc-vars
We can then use the if_extensions macro to define a macro which expands out to the
-L
option if newfangle is being invoked in a C source file, so that C compile errors will refer to the line number in the Lyx document.
Makefile.inc-vars
TABS=8
nf_line=-L -T$(TABS)
newfangle=newfangle \
$(call if_extension,$(2),$(C_EXTENSIONS),$(nf_line)) \
-R"$(2)" $(1)
We can use a similar trick to define an indent macro which takes just the filename as an argument and can return a pipeline stage calling the indent command. Indent can be turned off with
make newfangle_sources indent=
Makefile.inc-vars
indent_options=-npro -kr -i8 -ts8 -sob -l80 -ss -ncs
indent=$(call if_extension,$(1),$(C_EXTENSIONS),\
| indent $(indent_options))
We now define the pattern for extracting a file. The files are written using noweb's cpif
so that the file timestamp will not be touched if the contents haven't changed. This avoids the need to rebuild the entire project because of a typographical change in the documentation, or if only a few C source files have changed.
Makefile.inc-vars
newfangle_extract=@mkdir -p $(dir $(1)) && \
$(call newfangle,$(2),$(1)) > "$(1).tmp" && \
cat "$(1).tmp" $(indent) | cpif "$(1)" \
&& rm -- "$(1).tmp" || \
(echo error newfangling $(1) from $(2) ; exit 1)
We define a target which will extract or update all sources. To do this we first defined a makefile template that can do this for any source file in the LaTeX document.
Makefile.inc-vars
define NEWFANGLE_template
$(1): $(2); \
$$(call newfangle_extract,$(1),$(2))
NEWFANGLE_TARGETS+=$(1)
endef
We then enumerate the discovered
NEWTANGLE_SOURCES
to generate a makefile rule for each one using the makefile template we defined above.
Makefile.inc-targets
$(foreach source,$(NEWFANGLE_SOURCES),\
$(eval $(call NEWFANGLE_template,$(source),$(NEWFANGLE_SOURCE))) \
)
These will all be built with NEWFANGLE_SOURCE_STAMP.
We also remove the generated sources on a make distclean.
Makefile.inc-targets
_distclean: clean_newfangle_sources
4.3.6 Extracting Documentation
We then identify the intermediate stages of the documentation and their build and clean targets.
4.3.6.1 Running pdflatex
We produce a pdf file from the tex file.
Makefile.inc-vars
NEWFANGLE_PDF=$(TEX_SOURCE:.tex=.pdf)
We run pdflatex twice to be sure that the contents and aux files are up to date. We certainly are required to run pdflatex twice if these files do not exist!
Makefile.inc-targets
$(NEWFANGLE_PDF): $(TEX_SOURCE); pdflatex $< && pdflatex $<
clean_pdf: ; rm -f -- $(NEWFANGLE_PDF) \
$(TEX_SOURCE:.tex=.toc) \
$(TEX_SOURCE:.tex=.log) \
$(TEX_SOURCE:.tex=.aux)
4.3.6.2 The docs as a whole
Currently we only build pdf as a final format, but NEWFANGLE_DOCS may later hold other output formats.
Makefile.inc-vars
NEWFANGLE_DOCS=$(NEWFANGLE_PDF)
We also define newfangle_docs as a convenient phony target<
Makefile.inc-targets
.PHONY: newfangle_docs
newfangle_docs: $(NEWFANGLE_DOCS)
docs: newfangle_docs
And define a convenient clean_noweb_docs which we add to the regular clean target
Makefile.inc-targets
.PHONEY: clean_newfangle_docs
clean_newfangle_docs: clean_tex clean_pdf
clean: clean_newfangle_docs
distclean_newfangle_docs: clean_tex clean_newfangle_docs
distclean: clean distclean_newfangle_docs
4.3.7 Other helpers
If Makefile.inc is included into Makefile, then extracted files can be updated with this command:
make newfangle_sources
otherwise, with:
make -f Makefile.inc newfangle_sources
5 Newfangle awk source code
We use the copyright notice from chapter
?.
./newfangle,language=awk,morestring=[b]{/},morekeywords=else
#! /usr/bin/awk -f
# =<\chunkref{gpl3-copyright}>
We also use code from Arnold Robbins public domain getopt (1993 revision) defined in chapter
12?, and naturally want to attribute this appropriately.
# NOTE: Arnold Robbins public domain getopt for awk is also used:
=<\chunkref{getopt.awk-header}>
=<\chunkref{getopt.awk-getopt()}>
And include the following chunks
./newfangle
=<\chunkref{helper-functions}>
=<\chunkref{mode-tracker}>
=<\chunkref{parse_chunk_args}>
=<\chunkref{chunk-storage-functions}>
=<\chunkref{output_chunk_names()}>
=<\chunkref{output_chunks()}>
=<\chunkref{write_chunk()}>
=<\chunkref{expand_chunk_args()}>
=<\chunkref{begin}>
=<\chunkref{recognize-chunk}>
=<\chunkref{end}>
5.1 AWK tricks
The portable way to erase an array in awk is to split the empty string, like this:
awk-delete-array,params=ARRAY
dump-array,params=ARRAY
print "\nDump: ${ARRAY}\n--------\n" > "/dev/stderr";
for (_x in ${ARRAY}) {
print _x "=" ${ARRAY}[_x] "\n" > "/dev/stderr";
}
print "========\n" > "/dev/stderr";
,params=
5.2 Catching errors
Fatal errors are issued with the error function:
error(),append=helper-functions
function error(message)
{
print "ERROR: " FILENAME ":" FNR " " message > "/dev/stderr";
exit 1;
}
function warning(message)
{
print "WARNING: " FILENAME ":" FNR " " message > "/dev/stderr";
warnings++;
}
6 lstlistings
LaTeX arguments to lstlistings macros are a comma seperated list of key-value pairs. Values containing commas are enclosed in
{
braces
}
, which is to be expected for LaTeX.
A sample expressions is:
name=thomas, params={a, b}, something, something-else
but we see that this is just a simpler form of this expression:
name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
We may consider that we need a function that can parse such LaTeX expressions and assign the values to an AWK associated array, perhaps using a recursive parser into a multi-dimensional hash
, resulting in:
key
|
value
|
a[name]
|
freddie
|
a[foo, bar]
|
baz
|
a[foo, quux, quirk]
|
|
a[foo, quux, a]
|
fleeg
|
a[etc]
|
|
Oon reflection it seems that sometimes such nesting is not desirable, as the braces are also used to delimit values that contain commas --- we may consider that
name={williamson, freddie}
should assign
williamson, freddie
to
name
.
In fact we are not so interested in the detail so as to be bothered by this, which turns out to be a good thing for two reasons. Firstly LaTeX has a malleable parser with no strict syntax, and secondly whether or not
williamson
and
freddie
should count as two items will be context dependant anyway.
We need to parse this latex for only one reason; which is that we are extending lstlistings to add some additional arguments which will be used to express chunk parameters and other chunk options.
6.1 Additional lstlstings parameters
Further on we define a
\Chunk
LaTeX macro whose arguments will consist of a the chunk name, optionally followed by a comma and then a comma separated list of arguments. In fact we will just need to prefix
name=
to the arguments to in order to create valid lstlistings arguments.
There will be other arguments supported too;
params As an extension to many literate-programming styles, newfangle permits code chunks to take parameters and thus operate somewhat like C pre-processor macros, or like C++ templates. Chunk parameters are declared with a chunk argument called
params
, which holds a semi-colon separated list of parameters, like this:
achunk,language=C,params=name;address
addto a named chunk that this chunk is to be included into. This saves the effort of having to declare another listing of the named chunk merely to include this one.
Function
get_chunk_args()
will accept two paramters,
text
being the text to parse, and
values
being an array to receive the parsed values as described above. The optional parameter
path
is used during recursion to build up the multi-dimensional array path.
./newfangle
=<\chunkref{get_chunk_args()}>
get_chunk_args()
function get_chunk_args(text, values,
# optional parameters
path, # hierarchical precursors
# local vars
a, name)
The strategy is to parse the name, and then look for a value. If the value begins with a brace
{
, then we recurse and consume as much of the text as necessary, returning the remaining text when we encounter a leading close-brace
}
. This being the strategy --- and executed in a loop --- we realise that we must first look for the closing brace (perhaps preceded by white space) in order to terminate the recursion, and returning remaining text.
{
split("", next_chunk_args);
while(length(text)) {
if (match(text, "^ *}(.*)", a)) {
return a[1];
}
=<\chunkref{parse-chunk-args}>
}
return text;
}
We can see that the text could be inspected with this regex:
parse-chunk-args
if (! match(text, " *([^,=]*[^,= ]) *(([,=]) *(([^,}]*) *,* *(.*))|)$", a)) {
return text;
}
and that
a
will have the following values:
a[n]
|
assigned text
|
1
|
freddie
|
2
|
=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
|
3
|
=
|
4
|
freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc
|
5
|
freddie
|
6
|
, foo={bar=baz, quux={quirk, a=fleeg}}, etc
|
a[3] will be either
=
or
,
and signify whether the option named in
a[1]
has a value or not (respectively).
If the option does have a value, then if the expression
substr(a[4],1,1)
returns a brace
{
it will signify that we need to recurse:
name=a[1];
if (a[3] == "=") {
if (substr(a[4],1,1) == "{") {
text = get_chunk_args(substr(a[4],2), values, path name SUBSEP);
} else {
values[path name]=a[5];
text = a[6];
}
} else {
values[path name]="";
text = a[2];
}
We can test this function like this:
gca-test.awk
=<\chunkref{get_chunk_args()}>
BEGIN {
SUBSEP=".";
print get_chunk_args("name=freddie, foo={bar=baz, quux={quirk, a=fleeg}}, etc", a);
for (b in a) {
print "a[" b "] => " a[b];
}
}
which should give this output:
gca-test.awk-results
a[foo.quux.quirk] =>
a[foo.quux.a] => fleeg
a[foo.bar] => baz
a[etc] =>
a[name] => freddie
6.2 Parsing chunk arguments
Arguments to paramterized chunks are expressed in round brackets as a comma separated list of optional arguments. For example, a chunk that is defined with:
\Chunk{achunk, params=name ; address}
could be invoked as:
\chunkref{achunk}(John Jones, jones@example.com)
An argument list may be as simple as in
\chunkref{pull}(thing, otherthing)
or as complex as:
\chunkref{pull}(things[x, y], get_other_things(a, "(all)"))
--- which for all it's commas and quotes and parenthesis represents only two parameters:
things[x, y]
and
get_other_things(a, "(all)")
.
If we simply split parameter list on commas, then the comma in
things[x,y]
would split into two seperate arguments:
things[x
and
y]
--- neither of which make sense on their own.
One way to prevent this would be by refusing to split text between matching delimiters, such as
[
,
]
,
(
,
)
,
{
,
}
and most likely also
"
,
"
and
'
,
'
. Of course this also makes it impossible to pass such mis-matched code fragments as parameters, but I think that it would be hard for readers to cope with authors who would pass such code unbalanced fragments as chunk parameters
.
Unfortunately, the full set of matching delimiters may vary from language to language. In certain C++ template contexts,
<
and
>
would count as delimiters, and yet in other contexts they would not.
This puts me in the unfortunate position of having to parse-somewhat all programming languages without knowing what they are!
However, if this universal mode-tracking is possible, then parsing the arguments would be trivial. Such a mode tracker is described in chapter
? and used here with simplicity.
parse_chunk_args
function parse_chunk_args(language, text, values, mode,
# local vars
c, context, rest)
{
=<\chunkref{new-mode-tracker}(context, language, mode)>
rest = mode_tracker(context, text, values);
# extract values
for(c=1; c <= context[0, "values"]; c++) {
values[c] = context[0, "values", c];
}
return rest;
}
6.3 Expanding parameters in the text
Within the body of the chunk, the parameters are referred to with:
${name}
and
${address}
. There is a strong case that a LaTeX style notation should be used, like \param{name} which would be expressed in the listing as =<\param{name}> and be rendered as
. Such notation would make me go blind, but I do intend to adopt it
.
We therefore need a function
expand_chunk_args
which will take a block of text, a list of permitted parameters, and the arguments which must substitute for the parameters.
Here we split the text on
${
which means that all parts except the first will begin with a parameter name which will be terminated by
}
. The split function will consume the literal
${
in each case.
expand_chunk_args()
function expand_chunk_args(text, params, args,
p, text_array, next_text, v, t, l)
{
if (split(text, text_array, "\\${")) {
=<\chunkref{substitute-chunk-args}>
}
return text;
}
First, we produce an associative array of substitution values indexed by parameter names. This will serve as a cache, allowing us to look up the replacement values as we extract each name.
substitute-chunk-args
for(p in params) {
v[params[p]]=args[p];
}
We accumulate substituted text in the variable
text
. As the first part of the split function is the part before the delimiter --- which is
${
in our case --- this part will never contain a parameter reference, so we assign this directly to the result kept in
$text
.
We then iterate over the remaining values in the array
, and substitute each reference for it's argument.
for(t=2; t in text_array; t++) {
=<\chunkref{substitute-chunk-arg}>
}
After the split on
${
a valid parameter reference will consist of valid parameter name terminated by a close-brace
}
. A valid character name begins with the underscore or a letter, and may contain letters, digits or underscores.
A valid looking reference that is not actually the name of a parameter will be and not substituted. This is good because there is nothing to substitute anyway, and it avoids clashes when writing code for languages where ${...} is a valid construct --- such constructs will not be interfered with unless the parameter name also matches.
substitute-chunk-arg
if (match(text_array[t], "^([a-zA-Z_][a-zA-Z0-9_]*)}", l) &&
l[1] in v)
{
text = text v[l[1]] substr(text_array[t], length(l[1])+2);
} else {
text = text "${" text_array[t];
}
7 Language Modes
lstlistings and newfangle both recognize source languages, and perform some basic parsing. lstlistings can detect strings and comments within a language definition and perform suitable rendering, such as italics for comments, and visible-spaces within strings.
Newfangle similarly can recognize strings, and comments, etc, within a language, so that any chunks included with
\chunkref
can be suitably escape or quoted.
7.1.1 Modes to keep code together
As an example, in the C language there are a few parse modes, affecting the interpretation of characters.
One parse mode is the strings mode. The string mode is commenced by an un-escaped quotation mark
"
and terminated by the same. Within the string mode, only one additional mode can be commenced, it is the backslash mode
\
, which is always terminated by the folloing character.
Other modes are
[
which is terminated by a
]
(unless it occurs in a string).
Consider this line of C code:
things\underbrace[x, \ y]1.\ [\ mode, \ get_other_things\underbrace(a, \overbrace"(all)")2.\ (\ mode3.\ "\ mode
Mode nesting prevents the close parenthesis in quote mode (part 3) from terminating the parenthesis mode (part 2).
Each language has a set of modes, the default mode being the null mode. Each mode can lead to other modes.
7.1.2 Modes to included chunks
For instance, consider this chunk with
language=perl
:
example-perl,language=perl
If it were included in a chunk with
language=sh
, like this:
example-sh,language=sh
perl -pe "=<\chunkref{example-perl}>"
newfangle would need to generate output like this if it were to work:
perl -pe "s/\"\$/'/;"
See that the double quote
"
, and
$
in the regex have been quoted with a back-slash to protect them from shell interpretation.
If that were then included in a chunk with
language=make
, like this:
example-makefile,language=make
target: pre-req
=<\chunkref{example-sh}>
We would need the output to look like this --- note the $$:
target: pre-req
perl -pe "s/\"\$$/'/;"
In order to make this work, we need to define a mode-tracker supporting each language, that can detect the various quoting modes, and provide a transformation that must be applied to any included text so that included text will be interpreted correctly after any interpolation that it may be subject to at run-time.
For example, the sed transformation for text to be inserted into sh double-quoted strings would be something like:
s/\\/\\\\/g;s/$/\\$/g;s/"/\\"/g;
which protects
\ $ "
.
The mode tracker must also track nested mode-changes, as in this
sh
example.
echo "hello 'id ...'"
?
Any characters inserted at the point marked ? would need to be escaped, including
' | *
among others. First it would need escaping for the back-ticks ', and then for the double-quotes ".
Escaping need not occur if the format and mode of the included chunk matches that of the including chunk.
As each chunk is output a new mode tracker for that language is initialized in it's normal state. As text is output for that chunk the output mode is tracked. When a new chunk is included, a transformation appropriate to that mode is selected and pushed onto a stack of transformations. Any text to be output is first passed through this stack of transformations.
It remains to consider if the chunk-include function should return it's generated text so that the caller can apply any transformations (and formatting), or if it should apply the stack of transformations itself.
Note that the transformed text should have the property of not being able to change the mode in the current chunk.
7.2 Language Mode Definitions
All modes are stored in a single multi-dimensional hash. The first index is the language, and the second index is the mode-identifier. The third indexes are terminators, and optionally, submodes, and delimiters.
A useful set of mode definitions for a nameless general C-type language is shown here. (Don't be confused by the double backslash escaping needed in awk. One set of escaping is for the string, and the second set of escaping is for the regex).
Submodes are entered by the characters
\
"
'
{
(
[
/*
common-mode-definitions,params=language
modes[${language}, "", "submodes" ]="\\\\|\"|'|{|\\(|\\[";
In the default mode, a comma surrounded by un-important white space is a delimiter of language items.
modes[${language}, "", "delimiters"]=" *, *";
and should pass this test:
test:mode-definitions
parse_chunk_args("c-like", "1,2,3", a, "");
if (a[1] != "1") e++;
if (a[2] != "2") e++;
if (a[3] != "3") e++;
if (length(a) != 3) e++;
=<\chunkref{pca-test.awk:summary}>
parse_chunk_args("c-like", "joe, red", a, "");
if (a[1] != "joe") e++;
if (a[2] != "red") e++;
if (length(a) != 2) e++;
=<\chunkref{pca-test.awk:summary}>
parse_chunk_args("c-like", "${colour}", a, "");
if (a[1] != "${colour}") e++;
if (length(a) != 1) e++;
=<\chunkref{pca-test.awk:summary}>
Nested modes are identified by a backslash, a double or single quote, various bracket styles or a /* comment.
For each of these sub-modes modes we must also identify at a mode terminator, and any sub-modes or delimiters that may be entered
.
The backslash mode has no submodes or delimiters, and is terminated by any character. Note that we are not so much interested in evaluating or interpolating content as we are in delineating content. It is no matter that a double backslash (
\\
) may represent a single backslash while a backslash-newline may represent white space, but it does matter that the newline in a backslash newline should not be able to terminate a C pre-processor statement; and so the newline will be consumed by the backslash however it is to be interpreted.
common-mode-definitions
modes[${language}, "\\", "terminators"]=".";
Common languages support two kinds of strings quoting, double quotes and single quotes.
mode:common-string,params=language;quote
In a string we have one special mode, which is the backslash. This may escape an embedded quote and prevent us thinking that it should terminate the string.
modes[${language}, ${quote}, "submodes" ]="\\\\";
Otherwise, the string will be terminated by the same character that commenced it.
modes[${language}, ${quote}, "terminators"]=${quote};
In C type languages, certain escape sequences exist in strings. We need to define mechanism to enclode any chunks included in this mode using those escape sequences. These are expressed in two parts, s meaning search, and r meaning replace.
The first substitution is to replace a backslash with a double backslash. We do this first as other substitutions may introduce a backslash which we would not then want to escape again here.
escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\\";
escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\\\";
If the quote character occurs in the text, it should be preceded by a backslash, otherwise it would terminate the string unexpectedly.
escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]=${quote};
escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\" ${quote};
Any newlines in the string, must be replaced by
\n
escapes[${language}, ${quote}, ++escapes[${language}, ${quote}], "s"]="\n";
escapes[${language}, ${quote}, escapes[${language}, ${quote}], "r"]="\\n";
For the common modes, we define this string handling for double and single quotes.
common-mode-definitions,params=language
=<\chunkref{mode:common-string}(${language}, "\textbackslash{}"")>
=<\chunkref{mode:common-string}(${language}, "'")>
Working strings should pass this test:
test:mode-definitions
parse_chunk_args("c-like", "say \"I said, \\\"Hello, how are you\\\".\", for me", a, "");
if (a[1] != "say \"I said, \\\"Hello, how are you\\\".\"") e++;
if (a[2] != "for me") e++;
if (length(a) != 2) e++;
=<\chunkref{pca-test.awk:summary}>
7.2.3 Parentheses, Braces and Brackets
Where quotes are closed by the same character, parentheses, brackets and braces are closed by an alternate character.
mode:common-brackets,params=language;open;close
modes[${language}, ${open}, "submodes" ]="\\\\|\"|{|\\(|\\[|'|/\\*";
modes[${language}, ${open}, "delimiters"]=" *, *";
modes[${language}, ${open}, "terminators"]=${close};
Note that the open is NOT a regex but the close token IS.
common-mode-definitions,params=language
=<\chunkref{mode:common-brackets}(${language}, "{", "}")>
=<\chunkref{mode:common-brackets}(${language}, "[", "\textbackslash{}\textbackslash{}]")>
=<\chunkref{mode:common-brackets}(${language}, "(", "\textbackslash{}\textbackslash{})")>
7.2.4 Customizing Standard Modes
mode:add-submode,params=language;mode;submode
modes[${language}, ${mode}, "submodes"] = modes[${language}, ${mode}, "submodes"] "|" ${submode};
mode:add-escapes,params=language;mode;search;replace
escapes[${language}, ${mode}, ++escapes[${language}, ${mode}], "s"]=${search};
escapes[${language}, ${mode}, escapes[${language}, ${mode}], "r"]=${replace};
We can define
/* comment */
style comments and
//comment
style comments to be added to any language:
mode:multi-line-comments,params=language
=<\chunkref{mode:add-submode}(${language}, "", "/\textbackslash{}\textbackslash{}*")>
modes[${language}, "/*", "terminators"]="\\*/";
mode:single-line-slash-comments,params=language
=<\chunkref{mode:add-submode}(${language}, "", "//")>
modes[${language}, "//", "terminators"]="\n";
=<\chunkref{mode:add-escapes}(${language}, "//", "\textbackslash{}n", "\textbackslash{}n//")>
We can also define
# comment
style comments (as used in awk and shell scripts) in a similar manner.
mode:add-hash-comments,params=language
=<\chunkref{mode:add-submode}(${language}, "", "\#")>
modes[${language}, "#", "terminators"]="\n";
=<\chunkref{mode:add-escapes}(${language}, "\#", "\textbackslash{}n", "\textbackslash{}n\#")>
In C, the
#
denotes pre-processor directives which can be multi-line
mode:add-hash-defines,params=language
=<\chunkref{mode:add-submode}(${language}, "", "\#")>
modes[${language}, "#", "submodes" ]="\\\\";
modes[${language}, "#", "terminators"]="\n";
=<\chunkref{mode:add-escapes}(${language}, "\#", "\textbackslash{}n", "\textbackslash{}\textbackslash{}\textbackslash{}\textbackslash{}\textbackslash{}n")>
We can add these definitions to various languages
mode-definitions
=<\chunkref{common-mode-definitions}("c-like")>
=<\chunkref{common-mode-definitions}("c")>
=<\chunkref{mode:multi-line-comments}("c")>
=<\chunkref{mode:single-line-slash-comments}("c")>
=<\chunkref{mode:add-hash-defines}("c")>
=<\chunkref{common-mode-definitions}("awk")>
=<\chunkref{mode:add-hash-comments}("awk")>
The awk definitions should allow a comment block like this:
test:comment-quote,language=awk
# Comment: =<\chunkref{test:comment-text}>
test:comment-text,language=
Now is the time for
the quick brown fox to bring lemonade
to the party
to come out like this:
test:comment-quote:result
# Comment: Now is the time for
#the quick brown fox to bring lemonade
#to the party
The C definition for such a block should have it come out like this:
test:comment-quote:C-result
# Comment: Now is the time for\
the quick brown fox to bring lemonade\
to the party
7.3 Some tests
Also, the parser must return any spare text at the end that has not been processed due to a mode terminator being found.
test:mode-definitions
rest = parse_chunk_args("c-like", "1, 2, 3) spare", a, "(");
if (a[1] != 1) e++;
if (a[2] != 2) e++;
if (a[3] != 3) e++;
if (length(a) != 3) e++;
if (rest != " spare") e++;
=<\chunkref{pca-test.awk:summary}>
We must also be able to parse the example given earlier.
test:mode-definitions
parse_chunk_args("c-like", "things[x, y], get_other_things(a, \"(all)\"), 99", a, "(");
if (a[1] != "things[x, y]") e++;
if (a[2] != "get_other_things(a, \"(all)\")") e++;
if (a[3] != "99") e++;
if (length(a) != 3) e++;
=<\chunkref{pca-test.awk:summary}>
7.4 A non-recursive mode tracker
7.4.1 Constructor
The mode tracker holds its state in a stack based on a hash. This function, when passed an empty hash will intialize it.
new_mode_tracker()
function new_mode_tracker(context, language, mode) {
context[""] = 0;
context[0, "language"] = language;
context[0, "mode"] = mode;
}
Because awk functions cannot return an array, we must create the array first and pass it in, so we have a newfangle macro to do this:
new-mode-tracker,language=awk,params=context;language;mode
=<\chunkref{awk-delete-array}(${context})>
new_mode_tracker(${context}, ${language}, ${mode});
7.4.2 Management
And for tracking modes, we dispatch to a mode-tracker action based on the current language
mode_tracker,language=awk
function push_mode_tracker(context, language, mode,
# local vars
top)
{
if (! ("" in context)) {
=<\chunkref{new-mode-tracker}(context, language, mode)>
} else {
top = context[""];
if (context[top, "language"] == language && mode=="") mode = context[top, "mode"];
top++;
context[top, "language"] = language;
context[top, "mode"] = mode;
context[""] = top;
}
}
function finalize_mode_tracker(context)
{
if ( ("" in context) && context[""] != 0) return 0;
return 1;
}
This implies that any chunk must be syntactically whole; for instance, this is fine:
test:whole-chunk
if (1) {
=<\chunkref{test:say-hello}>
}
test:say-hello
But this is not fine; the chunk
test:hidden-else
is not properly cromulent.
test:partial-chunk
if (1) {
=<\chunkref{test:hidden-else}>
}
test:hidden-else
print "I'm fine";
} else {
print "I'm not";
These tests will check for correct behaviour:
test:cromulence
echo Cromulence test
passtest $NEWFANGLE -Rtest:whole-chunk $TEX_SRC &>/dev/null || ( echo "Whole chunk failed" && exit 1 )
failtest $NEWFANGLE -Rtest:partial-chunk $TEX_SRC &>/dev/null || ( echo "Partial chunk failed" && exit 1 )
We must avoid recursion as a language construct because we intend to employ mode-tracking to track language mode of emitted code, and the code is emitted from a function which is itself recursive, so instead we implement psuedo-recursion using our own stack based on a hash.
mode_tracker()
function mode_tracker(context, text, values,
# optional parameters
# local vars
mode, submodes, language,
cindex, c, a, part, item, name, result, new_values, new_mode,
delimiters, terminators)
{
We could be re-commencing with a valid context, so we need to setup the state according to the last context.
cindex = context[""] + 0;
mode = context[cindex, "mode"];
language = context[cindex, "language" ];
First we construct a single large regex combining the possible sub-modes for the current mode along with the terminators for the current mode.
parse_chunk_args-reset-modes
submodes=modes[language, mode, "submodes"];
if ((language, mode, "delimiters") in modes) {
delimiters = modes[language, mode, "delimiters"];
if (length(submodes)>0) submodes = submodes "|";
submodes=submodes delimiters;
} else delimiters="";
if ((language, mode, "terminators") in modes) {
terminators = modes[language, mode, "terminators"];
if (length(submodes)>0) submodes = submodes "|";
submodes=submodes terminators;
} else terminators="";
If we don't find anything to match on --- probably because the language is not supported --- then we return the entire text without matching anything.
if (! length(submodes)) return text;
mode_tracker()
=<\chunkref{parse_chunk_args-reset-modes}>
We then iterate the text (until there is none left) looking for sub-modes or terminators in the regex.
while((cindex >= 0) && length(text)) {
if (match(text, "(" submodes ")", a)) {
A bug that creeps in regularly during development is bad regexes of zero length which result in an infinite loop (as no text is consumed), so I catch that right away with this test.
if (RLENGTH<1) {
error(sprintf("Internal error, matched zero length submode, should be impossible - likely regex computation error\n" \
"Language=%s\nmode=%s\nmatch=%s\n", language, mode, submodes));
}
part
is defined as the text up to the sub-mode or terminator, and this is appended to
item
--- which is the current text being gathered. If a mode has a delimiter, then item is reset each time a delimiter is found.
\overbrace"\overbracehelloitem, \ \overbracethereitem"item, \ \overbracehe\ said.item
part = substr(text, 1, RSTART -1);
item = item part;
We must now determine what was matched. If it was a terminator, then we must restore the previous mode.
if (match(a[1], "^" terminators "$")) {
#printf("%2d EXIT MODE [%s] by [%s] [%s]\n", cindex, mode, a[1], text) > "/dev/stderr"
context[cindex, "values", ++context[cindex, "values"]] = item;
delete context[cindex];
context[""] = --cindex;
if (cindex>=0) {
mode = context[cindex, "mode"];
language = context[cindex, "language"];
=<\chunkref{parse_chunk_args-reset-modes}>
}
item = item a[1];
text = substr(text, 1 + length(part) + length(a[1]));
}
If a delimiter was matched, then we must store the current item in the parsed values array, and reset the item.
else if (match(a[1], "^" delimiters "$")) {
if (cindex==0) {
context[cindex, "values", ++context[cindex, "values"]] = item;
item = "";
} else {
item = item a[1];
}
text = substr(text, 1 + length(part) + length(a[1]));
}
otherwise, if a new submode is detected (all submodes have terminators), we must create a nested parse context until we find the terminator for this mode.
else if ((language, a[1], "terminators") in modes) {
#check if new_mode is defined
item = item a[1];
#printf("%2d ENTER MODE [%s] in [%s]\n", cindex, a[1], text) > "/dev/stderr"
text = substr(text, 1 + length(part) + length(a[1]));
context[""] = ++cindex;
context[cindex, "mode"] = a[1];
context[cindex, "language"] = language;
mode = a[1];
=<\chunkref{parse_chunk_args-reset-modes}>
} else {
error(sprintf("Submode '%s' set unknown mode in text: %s\nLanguage %s Mode %s\n", a[1], text, language, mode));
text = substr(text, 1 + length(part) + length(a[1]));
}
}
In the final case, we parsed to the end of the string. If the string was entire, then we should have no nested mode context, but if the string was just a fragment we may have a mode context which must be preserved for the next fragment. Todo: Consideration ought to be given if sub-mode strings are split over two fragments.
else {
context[cindex, "values", ++context[cindex, "values"]] = item text;
text = "";
item = "";
}
}
context["item"] = item;
if (length(item)) context[cindex, "values", ++context[cindex, "values"]] = item;
return text;
}
7.4.3.1 One happy chunk
All the mode tracker chunks are referred to here:
mode-tracker
=<\chunkref{new_mode_tracker()}>
=<\chunkref{mode_tracker()}>
We can test this function like this:
pca-test.awk,language=awk
=<\chunkref{error()}>
=<\chunkref{mode-tracker}>
=<\chunkref{parse_chunk_args()}>
BEGIN {
SUBSEP=".";
=<\chunkref{mode-definitions}>
=<\chunkref{test:mode-definitions}>
}
pca-test.awk:summary,language=awk
if (e) {
printf "Failed " e
for (b in a) {
print "a[" b "] => " a[b];
}
} else {
print "Passed"
}
split("", a);
e=0;
which should give this output:
pca-test.awk-results,language=
a[foo.quux.quirk] =>
a[foo.quux.a] => fleeg
a[foo.bar] => baz
a[etc] =>
a[name] => freddie
7.5 Escaping and Quoting
Each nested mode can optionally define a set of transforms to be applied to any text that is included from another language.
This code can perform transforms
mode_tracker,language=awk
function transform_escape(s, r, text,
# optional
max,
# local vars
c)
{
for(c=1; c <= max && (c in s); c++) {
gsub(s[c], r[c], text);
}
return text;
}
This function must append from index
c
onwards, and escape transforms from the supplied context, and return c + number of new transforms.
function mode_escaper(context, s, r, src,
c, cp, cpl)
{
for(c = context[""]; c >= 0; c--) {
if ( (context[c, "language"], context[c, "mode"]) in escapes) {
cpl = escapes[context[c, "language"], context[c, "mode"]];
for (cp = 1; cp <= cpl; cp ++) {
++src;
s[src] = escapes[context[c, "language"], context[c, "mode"], c, "s"];
r[src] = escapes[context[c, "language"], context[c, "mode"], c, "r"];
}
}
}
return src;
}
test:escapes,language=sh
echo escapes test
passtest $NEWFANGLE -Rtest:comment-quote $TEX_SRC &>/dev/null || ( echo "Comment-quote failed" && exit 1 )
8 Recognizing Chunks
Newfangle recognizes noweb chunks, but as we also want better LaTeX integration we will recognize any of these:
-
notangle chunks matching the pattern
^<<.*?>>=
-
a chunks beginning with
\begin{lstlistings}
, possibly with \Chunk{...} on the previous line
-
an older form I have used, beginning with
\begin{Chunk}[options]
--- also more suitable for plain LaTeX users
.
8.1 Chunk start
The variable
chunking
is used to signify that we are processing a code chunk and not document. In such a state, input lines will be assigned to the current chunk; otherwise they are ignored.
8.1.1 lstlistings
Our current scheme is to recognize the new lstlisting chunks, but these may be preceded by a
\Chunk
command which in LyX is a more convenient way to pass the chunk name to the
\begin{lstlistings}
command, and a more visible way to specify other
lstset
settings.
The arguments to the
\Chunk
command are a name, and then a comma-seperated list of key-value pairs after the manner of
\lstset
. (In fact within the LaTeX
\Chunk
macro (section
13.2.1?) the text
name=
is prefixed to the argument which is then literally passed to
\lstset
).
recognize-chunk,language=awk
/^\\Chunk{/ {
if (match($0, "^\\\\Chunk{ *([^ ,}]*),?(.*)}", line)) {
next_chunk_name = line[1];
get_chunk_args(line[2], next_chunk_args);
}
next;
}
We also make a basic attempt to parse the name out of the
\lstlistings[name= chunk-name]
text, otherwise we fall back to the name found in the previous chunk command. This attempt is very basic and doesn't support commas or spaces or square brackets as part of the chunkname. We also recognize
\begin{Chunk}
which is convenient for some users
.
/^\\begin{lstlisting}|^\\begin{Chunk}/ {
if (match($0, "}.*[[,] *name= *{? *([^], }]*)", line)) {
new_chunk(line[1]);
} else {
new_chunk(next_chunk_name, next_chunk_args);
}
chunking=1;
next;
}
We recognize notangle style chunks too:
recognize-chunk,language=awk
/^[<]<.*[>]>=/ {
if (match($0, "^[<]<(.*)[>]>= *$", line)) {
chunking=1;
notangle_mode=1;
new_chunk(line[1]);
next;
}
}
8.2 Chunk end
Likewise, we need to recognize when a chunk ends.
8.2.1 lstlistings
The
e
in
[e]end{lislisting}
is surrounded by square brackets so that when this document is processed, this chunk doesn't terminate early when the lstlistings package recognizes it's own end-string!
This doesn't make sense as the regex is anchored with ^, which this line does not begin with!
recognize-chunk
/^\\[e]nd{lstlisting}|^\\[e]nd{Chunk}/ {
chunking=0;
active_chunk="";
next;
}
recognize-chunk
/^@ *$/ {
chunking=0;
active_chunk="";
}
All other recognizers are only of effect if we are chunking; there's no point in looking at lines if they aren't part of a chunk, so we just ignore them as efficiently as we can.
recognize-chunk
8.3 Chunk contents
Chunk contents are any lines read while
chunking
is true. Some chunk contents are special in that they refer to other chunks, and will be replaced by the contents of these chunks when the file is generated.
We add the output record separator
ORS
to the line now, because we will set
ORS
to the empty string when we generate the output
.
recognize-chunk
length(active_chunk) {
=<\chunkref{process-chunk-tabs}>
=<\chunkref{process-chunk}>
}
If a chunk just consisted of plain text, we could handle the chunk like this:
process-chunk-simple
chunk_line(active_chunk, $0 ORS);
but in fact a chunk can include references to other chunks. Chunk includes are traditionally written as
<<chunk-name>>
, but we support other variations.
However, we also process tabs at this point, a tab at input can be replaced by a number of spaces defined by the
tabs
variable, set by the
-T
option. Of course this is poor tab behaviour, we should probably have the option to use proper counted tab-stops and process this on output.
process-chunk-tabs
if (length(tabs)) {
gsub("\t", tabs);
}
8.3.1 lstlistings
If
\lstset{escapeinside={=<}{>}}
is set, then we can use
=<\chunkref{ chunk-name}>
in listings. The sequence
=<
was chosen because:
-
it is a better mnemonic than
<<chunk-name>>
in that the = sign signifies equivalent or substitutability,
-
and because =< is not valid in C or in any language I can think of
-
and also because lstlistings doesn't like
>>
as an end delimiter for the texcl escape, so we must make do with a single
>
, which is better matched by
=<
than
<<
.
Unfortunately the
=<...>
that we use re-enters a LaTeX parsing mode in which some characters are special, e.g.
# \
, and so these cause trouble if used in arguments to
\chunkref
. At some point I must fix the LaTeX command
\chunkref
so that it can accept these literally, but until then, when writing chunkref argumemts that need these characters, I must use the forms
\textbackslash{}
and
\#
; so I also define a hacky chunk
delatex
whose purpose it is to remove these from any arguments parsed by newfangle, and used further on.
delatex,params=text
# FILTHY HACK
gsub("\\\\#", "#", ${text});
gsub("\\\\textbackslash{}", "\\", ${text});
As each chunk line may contain more than one chunk include, we will split out chunk includes in an iterative fashion
.
First, as long as the chunk contains a
\chunkref
command we take as much as we can up to the first
\chunkref
command.
process-chunk
chunk = $0;
indent = 0;
while(match(chunk,
"([=]<\\\\chunkref{([^}>]*)}(\\(.*\\)|)>|<<([a-zA-Z_][-a-zA-Z0-9_]*)>>)",
line)\
) {
chunklet = substr(chunk, 1, RSTART - 1);
We keep track of the indent count, by counting the number of literal characters found. We can then preserve this indent on each output line when multi-line chunks are expanded.
We then process this first part literal text, and set the chunk which is still to be processed to be the text after the
\chunkref
command, which we will process next as we continue around the loop.
indent += length(chunklet);
chunk_line(active_chunk, chunklet);
chunk = substr(chunk, RSTART + RLENGTH);
We then consider the type of chunk command we have found, whether it is the newfangle style command beginning with
=<
or the older notangle style beginning with
<<
.
Newfangle chunks may have parameters contained within square brackets. These will be matched in
line[3]
and are considered at this stage of processing to be part of the name of the chunk to be included.
if (substr(line[1], 1, 1) == "=") {
# chunk name up to }
=<\chunkref{delatex}(line[3])>
chunk_include(active_chunk, line[2] line[3], indent);
} else if (substr(line[1], 1, 1) == "<") {
chunk_include(active_chunk, line[4], indent);
} else {
error("Unknown chunk fragment: " line[1]);
}
The loop will continue until there are no more chunkref statements in the text, at which point we process the final part of the chunk.
}
chunk_line(active_chunk, chunk);
We add the newline character as a chunklet on it's own, to make it easier to detect new lines and thus manage indentation when processing the output.
chunk_line(active_chunk, "\n");
We will also permit a chunk-part number to follow in square brackets, so that
=<\chunkref{chunk-name[1]}>
will refer to the first part only. This can make it easy to include a C function prototype in a header file, if the first part of the chunk is just the function prototype without the trailing semi-colon. The header file would include the prototype with the trailing semi-colon, like this:
=<\chunkref{chunk-name[1]}>;
We should perhaps introduce a notion of language specific chunk options; so that perhaps we could specify:
=<\chunkref{chunk-name[function-declaration]}>;
which applies a transform
function-declaration
to the chunk --- which in this case would extract a function prototype from a function.
9 Processing Options
At the start, first we set the default options.
default-options
debug=0;
linenos=0;
notangle_mode=0;
root="*";
tabs = "";
Then we use getopt the standard way, and null out ARGV afterwards in the normal AWK fashion.
read-options
Optind = 1 # skip ARGV[0]
while(getopt(ARGC, ARGV, "R:LdT:hr")!=-1) {
=<\chunkref{handle-options}>
}
for (i=1; i<Optind; i++) { ARGV[i]=""; }
This is how we handle our options:
handle-options
if (Optopt == "R") root = Optarg;
else if (Optopt == "r") root="";
else if (Optopt == "L") linenos = 1;
else if (Optopt == "d") debug = 1;
else if (Optopt == "T") tabs = indent_string(Optarg+0);
else if (Optopt == "h") help();
else if (Optopt == "?") help();
We do all of this at the beginning of the program
begin
BEGIN {
=<\chunkref{constants}>
=<\chunkref{mode-definitions}>
=<\chunkref{default-options}>
=<\chunkref{read-options}>
}
And have a simple help function
help()
function help() {
print "Usage:"
print " newfangle [-L] -R<rootname> [source.tex ...]"
print " newfangle -r [source.tex ...]"
print " If the filename, source.tex is not specified then stdin is used"
print
print "-L causes the C statement: #line <lineno> \"filename\"" to be issued"
print "-R causes the named root to be written to stdout"
print "-r lists all roots in the file (even those used elsewhere)"
exit 1;
}
10 Generating the output
We generate output by calling output_chunk, or listing the chunk names.
generate-output
if (length(root)) output_chunk(root);
else output_chunk_names();
We also have some other output debugging:
debug-output
if (debug) {
print "------ chunk names "
output_chunk_names();
print "====== chunks"
output_chunks();
print "++++++ debug"
for (a in chunks) {
print a "=" chunks[a];
}
}
We do both of these at the end. We also set
ORS=""
because each chunklet is not necessarily a complete line, and we already added
ORS
to each input line in section
?.
end
END {
=<\chunkref{debug-output}>
ORS="";
=<\chunkref{generate-output}>
}
We write chunk names like this. If we seem to be running in notangle compatibility mode, then we enclose the name like this
<<name>>
the same way notangle does:
output_chunk_names()
function output_chunk_names( c, prefix, suffix)
{
if (notangle_mode) {
prefix="<<";
suffix=">>";
}
for (c in chunk_names) {
print prefix c suffix "\n";
}
}
This function would write out all chunks
output_chunks()
function output_chunks( a)
{
for (a in chunk_names) {
output_chunk(chunk_names[a]);
}
}
function output_chunk(chunk) {
newline = 1;
lineno_needed = linenos;
write_chunk(chunk);
}
10.1 Assembling the chunks
chunk_path
holds a string consisting of the names of all the chunks that resulted in this chunk being output. It should probably also contain the source line numbers at which each inclusion also occured.
write_chunk()
We first initialize the mode tracker for this chunk.
function write_chunk(chunk_name) {
=<\chunkref{awk-delete-array}(context)>
return write_chunk_r(chunk_name, context);
}
write_chunk(),emph={chunk_path}
function write_chunk_r(chunk_name, context, indent, tail,
# optional vars
chunk_path, chunk_args,
s, r, src, new_src,
# local vars
chunk_params, part, max_part, part_line, frag, max_frag, text,
chunklet, only_part, call_chunk_args)
{
As mentioned in section
8.3.1?, a chunk name may contain a part specifier in square brackets, limiting the parts that should be emitted.
if (match(chunk_name, "^(.*)\\[([0-9]*)\\]$", chunk_name_parts)) {
chunk_name = chunk_name_parts[1];
only_part = chunk_name_parts[2];
}
We then create a mode tracker
=<\chunkref{new-mode-tracker}(context, chunks[chunk_name, "language"], "")>
We extract into
chunk_params
the names of the parameters that this chunk accepts, whose values were (optionally) passed in
chunk_args
.
split(chunks[chunk_name, "params"], chunk_params, " *; *");
To assemble a chunk, we write out each part.
write_chunk()
if (! (chunk_name in chunk_names)) {
error(sprintf(_"The root module <<%s>> was not defined.\nUsed by: %s",\
chunk_name, chunk_path));
}
max_part = chunks[chunk_name, "part"];
for(part = 1; part <= max_part; part++) {
if (! only_part || part == only_part) {
=<\chunkref{write-part}>
}
}
if (! finalize_mode_tracker(context)) {
error(sprintf(_"Module %s did not close context properly.\nUsed by: %s\n", chunk_name, chunk_path));
}
}
A part can either be a chunklet of lines, or an include of another chunk.
Chunks may also have parameters, specified in LaTeX style with braces after the chunk name --- looking like this in the document:
chunkname{param1, param2}
. Arguments are passed in square brackets:
\chunkref{chunkname}[arg1, arg2]
.
Before we process each part, we check that the source position hasn't changed unexpectedly, so that we can know if we need to output a new file-line directive.
write-part
=<\chunkref{check-source-jump}>
chunklet = chunks[chunk_name, "part", part];
if (chunks[chunk_name, "part", part, "type"] == part_type_chunk) {
=<\chunkref{write-included-chunk}>
} else if (chunklet SUBSEP "line" in chunks) {
=<\chunkref{write-chunklets}>
} else {
# empty last chunklet
}
To write an included chunk, we must detect any optional chunk arguments in parenthesis. Then we recurse calling
write_chunk()
.
write-included-chunk
if (match(chunklet, "^([^\\[\\(]*)\\((.*)\\)$", chunklet_parts)) {
chunklet = chunklet_parts[1];
parse_chunk_args("c-like", chunklet_parts[2], call_chunk_args, "(");
for (c in call_chunk_args) {
call_chunk_args[c] = expand_chunk_args(call_chunk_args[c], chunk_params, chunk_args);
}
} else {
split("", call_chunk_args);
}
# update the transforms arrays
new_src = mode_escaper(context, s, r, src);
write_chunk_r(chunklet, context,
chunks[chunk_name, "part", part, "indent"] indent,
chunks[chunk_name, "part", part, "tail"],
chunk_path "\n " chunk_name,
call_chunk_args,
s, r, new_src);
Before we output a chunklet of lines, we first emit the file and line number if we have one, and if it is safe to do so.
Chunklets are generally broken up by includes, so the start of a chunklet is a good place to do this. Then we output each line of the chunklet.
When it is not safe, such as in the middle of a multi-line macro definition,
lineno_suppressed
is set to true, and in such a case we note that we want to emit the line statement when it is next safe.
write-chunklets
max_frag = chunks[chunklet, "line"];
for(frag = 1; frag <= max_frag; frag++) {
=<\chunkref{write-file-line}>
We then extract the chunklet text and expand any arguments.
text = chunks[chunklet, frag];
/* check params */
text = expand_chunk_args(text, chunk_params, chunk_args);
If the text is a single newline (which we keep separate - see
?) then we increment the line number. In the case where this is the last line of a chunk and it is not a top-level chunk we replace the newline with an empty string --- because the chunk that included this chunk will have the newline at the end of the line that included this chunk.
We also note by
newline = 1
that we have started a new line, so that indentation can be managed with the following piece of text.
if (text == "\n") {
lineno++;
if (part == max_part && frag == max_frag && length(chunk_path)) {
text = "";
break;
} else {
newline = 1;
}
If this text does not represent a newline, but we see that we are the first piece of text on a newline, then we prefix our text with the current indent. NOTE:
newline
is a global output-state variable, but the
indent
is not.
} else if (length(text) || length(tail)) {
if (newline) text = indent text;
newline = 0;
}
Tail will soon no longer be relevant once mode-detection is in place.
text = text tail;
mode_tracker(context, text);
print transform_escape(s, r, text, src);
If a line ends in a backslash --- suggesting continuation --- then we supress outputting file-line as it would probably break the continued lines.
if (linenos) {
lineno_suppressed = substr(lastline, length(lastline)) == "\\";
}
}
Of course there is no point in actually outputting the source filename and line number (file-line) if they don't say anything new! We only need to emit them if they aren't what is expected, or if we we not able to emit one when they had changed.
write-file-line
if (newline && lineno_needed && ! lineno_suppressed) {
filename = a_filename;
lineno = a_lineno;
print "#line " lineno " \"" filename "\"\n"
lineno_needed = 0;
}
We check if a new file-line is needed by checking if the source line matches what we (or a compiler) would expect.
check-source-jump
if (linenos && (chunk_name SUBSEP "part" SUBSEP part SUBSEP "FILENAME" in chunks)) {
a_filename = chunks[chunk_name, "part", part, "FILENAME"];
a_lineno = chunks[chunk_name, "part", part, "LINENO"];
if (a_filename != filename || a_lineno != lineno) {
lineno_needed++;
}
}
11 Storing chunks
Awk has pretty limited data structures, so we will use two main hashes. Uninterrupted sequences of a chunk will be stored in
chunklets
and the chunklets used in a chunk will be stored in
chunks
.
constants
part_type_chunk=1;
SUBSEP=",";
The
params
mentioned are not chunk parameters for parameterized chunks, as mentioned in
?, but the lstlistings style parameters used in the
\Chunk
command
.
chunk-storage-functions
function new_chunk(chunk_name, params,
# local vars
p, append )
{
# HACK WHILE WE CHANGE TO ( ) for PARAM CHUNKS
gsub("\\(\\)$", "", chunk_name);
if (! (chunk_name in chunk_names)) {
if (debug) print "New chunk " chunk_name;
chunk_names[chunk_name];
for (p in params) {
chunks[chunk_name, p] = params[p];
}
if ("append" in params) {
append=params["append"];
if (! (append in chunk_names)) {
warning("Chunk " chunk_name " is appended to chunk " append " which is not defined yet");
new_chunk(append);
}
chunk_include(append, chunk_name);
chunk_line(append, ORS);
}
}
active_chunk = chunk_name;
prime_chunk(chunk_name);
}
function prime_chunk(chunk_name)
{
chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = \
chunk_name SUBSEP "chunklet" SUBSEP "" ++chunks[chunk_name, "chunklet"];
chunks[chunk_name, "part", chunks[chunk_name, "part"], "FILENAME"] = FILENAME;
chunks[chunk_name, "part", chunks[chunk_name, "part"], "LINENO"] = FNR + 1;
}
function chunk_line(chunk_name, line){
chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"],
++chunks[chunk_name, "chunklet", chunks[chunk_name, "chunklet"], "line"] ] = line;
}
Chunk include represents a chunkref statement, and stores the requirement to include another chunk. The parameter indent represents the quanity of literal text characters that preceded this chunkref statement and therefore by how much additional lines of the included chunk should be indented.
function chunk_include(chunk_name, chunk_ref, indent, tail)
{
chunks[chunk_name, "part", ++chunks[chunk_name, "part"] ] = chunk_ref;
chunks[chunk_name, "part", chunks[chunk_name, "part"], "type" ] = part_type_chunk;
chunks[chunk_name, "part", chunks[chunk_name, "part"], "indent" ] = indent_string(indent);
chunks[chunk_name, "part", chunks[chunk_name, "part"], "tail" ] = tail;
prime_chunk(chunk_name);
}
The indent is calculated by indent_string, which may in future convert some spaces into tab characters. This function works by generating a printf padded format string, like
%22s
for an indent of 22, and then printing an empty string using that format.
function indent_string(indent) {
return sprintf("%" indent "s", "");
}
12 getopt
I use Arnold Robbins public domain getopt (1993 revision). This is probably the same one that is covered in chapter 12 of "Edition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU Awk" but as that is licensed under the GNU Free Documentation License, Version 1.3, which conflicts with the GPL3, I can't use it from there (or it's accompanying explanations), so I do my best to explain how it works here.
The getopt.awk header is:
getopt.awk-header,language=awk,morestring=[b]{/},morekeywords=else
# getopt.awk --- do C library getopt(3) function in awk
#
# Arnold Robbins, arnold@skeeve.com, Public Domain
#
# Initial version: March, 1991
# Revised: May, 1993
The provided explanation is:
getopt.awk-notes
# External variables:
# Optind -- index in ARGV of first nonoption argument
# Optarg -- string value of argument to current option
# Opterr -- if nonzero, print our own diagnostic
# Optopt -- current option letter
# Returns:
# -1 at end of options
# ? for unrecognized option
# <c> a character representing the current option
# Private Data:
# _opti -- index in multi-flag option, e.g., -abc
The function follows. The final two parameters,
thisopt
and
i
are local variables and not parameters --- as indicated by the multiple spaces preceding them. Awk doesn't care, the multiple spaces are a convention to help us humans.
getopt.awk-getopt()
function getopt(argc, argv, options, thisopt, i)
{
if (length(options) == 0) # no options given
return -1
if (argv[Optind] == "--") { # all done
Optind++
_opti = 0
return -1
} else if (argv[Optind] !~ /^-[^: \t\n\f\r\v\b]/) {
_opti = 0
return -1
}
if (_opti == 0)
_opti = 2
thisopt = substr(argv[Optind], _opti, 1)
Optopt = thisopt
i = index(options, thisopt)
if (i == 0) {
if (Opterr)
printf("%c -- invalid option\n",
thisopt) > "/dev/stderr"
if (_opti >= length(argv[Optind])) {
Optind++
_opti = 0
} else
_opti++
return "?"
}
At this point, the option has been found and we need to know if it takes any arguments.
if (substr(options, i + 1, 1) == ":") {
# get option argument
if (length(substr(argv[Optind], _opti + 1)) > 0)
Optarg = substr(argv[Optind], _opti + 1)
else
Optarg = argv[++Optind]
_opti = 0
} else
Optarg = ""
if (_opti == 0 || _opti >= length(argv[Optind])) {
Optind++
_opti = 0
} else
_opti++
return thisopt
}
A test program is built in, too
getopt.awk-begin
BEGIN {
Opterr = 1 # default is to diagnose
Optind = 1 # skip ARGV[0]
# test program
if (_getopt_test) {
while ((_go_c = getopt(ARGC, ARGV, "ab:cd")) != -1)
printf("c = <%c>, optarg = <%s>\n",
_go_c, Optarg)
printf("non-option arguments:\n")
for (; Optind < ARGC; Optind++)
printf("\tARGV[%d] = <%s>\n",
Optind, ARGV[Optind])
}
}
The entire getopt.awk is made out of these chunks in order
getopt.awk
=<\chunkref{getopt.awk-header}>
=<\chunkref{getopt.awk-notes}>
=<\chunkref{getopt.awk-getopt()}>
=<\chunkref{getopt.awk-begin}>
Although we only want the header and function:
getopt
# try: locate getopt.awk for the full original file
# as part of your standard awk installation
=<\chunkref{getopt.awk-header}>
=<\chunkref{getopt.awk-getopt()}>
13 Newfangle LaTeX source code
13.1 newfangle module
Here we define a Lyx .module file that makes it convenient to use LyX for writing such literate programs.
This file
./newfangle.module
can be installed in your personal
.lyx/layouts folder
. You will need to Tools Reconfigure so that LyX notices it. It adds a new format Chunk, which should precede every listing and contain the chunk name.
./newfangle.module,language=
#\DeclareLyXModule{Newfangle Literate Listings}
#DescriptionBegin
# Newfangle literate listings allow one to write
# literate programs after the fashion of noweb, but without having
# to use noweave to generate the documentation. Instead the listings
# package is extended in conjunction with the noweb package to implement
# to code formating directly as latex.
# The newfangle awk script
#DescriptionEnd
Format 11
AddToPreamble
=<\chunkref{./newfangle.sty}>
EndPreamble
=<\chunkref{chunkstyle}>
=<\chunkref{chunkref}>
13.1.1 The Chunk style
The purpose of the chunk style is to make it easier for LyX users to provide the name to
\lstlistings
. Normally this requires right-clicking on the listing, choosing settings, advanced, and then typing
name=chunk-name
. This has the further disadvantage that the name (and other options) are not generally visible during document editing.
The chunk style is defined as a LaTeX command, so that all text on the same line is passed to the LaTeX command
Chunk
. This makes it easy to parse using
newfangle
, and easy to pass these options on to the listings package. The first word in a chunk section should be the chunk name, and will have
name=
prepended to it. Any other words are accepted arguments to
\lstset
.
We set PassThru to 1 because the user is actually entering raw latex.
chunkstyle
Style Chunk
LatexType Command
LatexName Chunk
Margin First_Dynamic
LeftMargin Chunk:xxx
LabelSep xx
LabelType Static
LabelString "Chunk:"
Align Left
PassThru 1
To make the label very visible we choose a larger font coloured red.
LabelFont
Family Sans
Size Large
Series Bold
Shape Italic
Color red
EndFont
End
13.1.2 The chunkref style
We also define the Chunkref style which can be used to express cross references to chunks.
chunkref
InsetLayout Chunkref
LyxType charstyle
LatexType Command
LatexName chunkref
PassThru 1
LabelFont
Shape Italic
Color red
EndFont
End
13.2 Latex Macros
We require the listings, noweb and xargs packages. As noweb defines it's own
\code
environment, we re-define the one that LyX logical markup module expects here.
./newfangle.sty,language=tex,basicstyle=\ttfamily
\usepackage{listings}%
\usepackage{noweb}%
\usepackage{xargs}%
\renewcommand{\code}[1]{\texttt{#1}}%
We also define a
CChunk
macro, for use as:
\begin{CChunk}
which will need renaming to
\begin{Chunk}
when I can do this without clashing with
\Chunk
.
\lstnewenvironment{Chunk}{\relax}{\relax}%
We also define a suitable
\lstset
of parameters that suit the literate programming style after the fashion of noweave.
\lstset{numbers=left, stepnumber=5, numbersep=5pt,
breaklines=false,basicstyle=\ttfamily,
numberstyle=\tiny, language=C}%
We also define a notangle-like mechanism for escaping to LaTeX from the listing, and by which we can refer to other listings. We declare the
=<...>
sequence to contain LaTeX code, and include another like this chunk:
=<\chunkref{chunkname}>
. However, because
=<...>
is already defined to contain LaTeX code for this document --- this is a newfangle document after all --- the code fragment below effectively contains the LaTeX code:
}{
. To avoid problems with document generation, I had to declare an lstlistings property:
escapeinside={}
for this listing only; which in LyX was done by right-clicking the listings inset, choosing
settings
?
advanced
.
\lstset{escapeinside={=<}{>}}%
Although our macros will contain the @ symbol, they will be included in a
\makeatletter
section by LyX; however we keep the commented out
\makeatletter
as a reminder. The listings package likes to centre the titles, but noweb titles are specially formatted and must be left aligned. The simplest way to do this turned out to be by removing the definition of
\lst@maketitle
. This may interact badly if other listings want a regular title or caption. We remember the old maketitle in case we need it.
%\makeatletter
%somehow re-defining maketitle gives us a left-aligned title
%which is extactly what our specially formatted title needs!
\global\let\newfangle@lst@maketitle\lst@maketitle%
\global\def\lst@maketitle{}%
13.2.1 The chunk command
Our chunk command accepts one argument, and calls
\ltset
. Although
\ltset
will note the name, this is erased when the next
\lstlisting
starts, so we make a note of this in
\lst@chunkname
and restore in in lstlistings Init hook.
\def\Chunk#1{%
\lstset{title={\newfanglecaption},name=#1}%
\global\edef\lst@chunkname{\lst@intname}%
}%
\def\lst@chunkname{\empty}%
13.2.1.1 Chunk parameters
Newfangle permits parameterized chunks, and requires the paramters to be specified as listings options. The newfangle script uses this, and although we don't do anything with these in the LaTeX code right now, we need to stop the listings package complaining.
\lst@Key{params}\relax{\def\newfangle@chunk@params{#1}}%
As it is common to define a chunk which then needs appending to another chunk, and annoying to have to declare a single line chunk to manage the include, we support an
append=
option.
\lst@Key{append}\relax{\def\newfangle@chunk@append{#1}}%
13.2.2 The noweb styled caption
We define a public macro
\newfanglecaption
which can be set as a regular title. By means of
\protect
, It expands to
\newfangle@caption
at the appriate time when the caption is emitted.
\def\newfanglecaption{\protect\newfangle@caption}%
The general noweb output format compactly identifies the current chunk, and references to the first chunk, and the previous and next chunks that have the same name.
This means that we need to keep a counter for each chunk-name, that we use to count chunks of the same name.
13.2.3 The chunk counter
It would be natural to have a counter for each chunk name, but TeX would soon run out of counters
, so we have one counter which we save at the end of a chunk and restore at the beginning of a chunk.
\newcounter{newfangle@chunkcounter}%
We construct the name of this variable to store the counter to be the text
lst-chunk-
prefixed onto the chunks own name, and store it in
\chunkcount
.
We save the counter like this:
save-counter
\global\expandafter\edef\csname \chunkcount\endcsname{\arabic{newfangle@chunkcounter}}%
and restore the counter like this:
restore-counter
\setcounter{newfangle@chunkcounter}{\csname \chunkcount\endcsname}%
./newfangle.sty
If there does not already exist a variable whose name is stored in
\chunkcount
, then we know we are the first chunk with this name, and then define a counter.
Although chunks of the same name share a common counter, they must still be distinguished. We use is the internal name of the listing, suffixed by the counter value. So the first chunk might be
something-1
and the second chunk be
something-2
, etc.
We also calculate the name of the previous chunk if we can (before we increment the chunk counter). If this is the first chunk of that name, then
\prevchunkname
is set to
\relax
which the noweb package will interpret as not existing.
\def\newfangle@caption{%
\edef\chunkcount{lst-chunk-\lst@intname}%
\@ifundefined{\chunkcount}{%
\expandafter\gdef\csname \chunkcount\endcsname{0}%
\setcounter{newfangle@chunkcounter}{\csname \chunkcount\endcsname}%
\let\prevchunkname\relax%
}{%
\setcounter{newfangle@chunkcounter}{\csname \chunkcount\endcsname}%
\edef\prevchunkname{\lst@intname-\arabic{newfangle@chunkcounter}}%
}%
After incrementing the chunk counter, we then define the name of this chunk, as well as the name of the first chunk.
\addtocounter{newfangle@chunkcounter}{1}%
\global\expandafter\edef\csname \chunkcount\endcsname{\arabic{newfangle@chunkcounter}}%
\edef\chunkname{\lst@intname-\arabic{newfangle@chunkcounter}}%
\edef\firstchunkname{\lst@intname-1}%
We now need to calculate the name of the next chunk. We do this by temporarily skipping the counter on by one; however there may not actually be another chunk with this name! We detect this by also defining a label for each chunk based on the chunkname. If there is a next chunkname then it will define a label with that name. As labels are persistent, we can at least tell the second time LaTeX is run. If we don't find such a defined label then we define
\nextchunkname
to
\relax
.
\addtocounter{newfangle@chunkcounter}{1}%
\edef\nextchunkname{\lst@intname-\arabic{newfangle@chunkcounter}}%
\@ifundefined{r@label-\nextchunkname}{\let\nextchunkname\relax}{}%
The noweb package requires that we define a
\sublabel
for every chunk, with a unique name, which is then used to print out it's navigation hints.
We also define a regular label for this chunk, as was mentioned above when we calculated
\nextchunkname
. This requires LaTeX to be run at least twice after new chunk sections are added --- but noweb requried that anyway.
\sublabel{\chunkname}%
% define this label for every chunk instance, so we
% can tell when we are the last chunk of this name
\label{label-\chunkname}%
We also try and add the chunk to the list of listings, but I'm afraid we don't do very well. We want each chunk name listing once, with all of it's references.
\addcontentsline{lol}{lstlisting}{\lst@name~[\protect\subpageref{\chunkname}]}%
We then call the noweb output macros in the same way that noweave generates them, except that we don't need to call
\nwstartdeflinemarkup
or
\nwenddeflinemarkup
-- and if we do it messes up the output somewhat.
\nwmargintag{%
{%
\nwtagstyle{}%
\subpageref{\chunkname}%
}%
}%
%
\moddef{%
{\lst@name}%
{%
\nwtagstyle{}\/%
\@ifundefined{newfangle@chunk@params}{}{%
(\newfangle@chunk@params)%
}%
[\csname \chunkcount\endcsname]~%
\subpageref{\firstchunkname}%
}%
\@ifundefined{newfangle@chunk@append}{}{%
\ifx{}\newfangle@chunk@append{x}\else%
,~add~to~\newfangle@chunk@append%
\fi%
}%
\global\def\newfangle@chunk@append{}%
\lstset{append=x}%
}%
%
\ifx\relax\prevchunkname\endmoddef\else\plusendmoddef\fi%
% \nwstartdeflinemarkup%
\nwprevnextdefs{\prevchunkname}{\nextchunkname}%
% \nwenddeflinemarkup%
}%
Originally this was developed as a
listings
aspect, in the Init hook, but it was found easier to affect the title without using a hook ---
\lst@AddToHookExe{PreSet}
is still required to set the listings name to the name passed to the
\Chunk
command, though.
%\lst@BeginAspect{newfangle}
%\lst@Key{newfangle}{true}[t]{\lstKV@SetIf{#1}{true}}
\lst@AddToHookExe{PreSet}{\global\let\lst@intname\lst@chunkname}
\lst@AddToHook{Init}{}%\newfangle@caption}
%\lst@EndAspect
13.2.4 Cross references
We define the \chunkref command which makes it easy to generate visual references to different code chunks, e.g.
Macro
|
Appearance
|
\chunkref{preamble}
|
|
\chunkref[3]{preamble}
|
|
\chunkref{preamble}[arg1, arg2]
|
|
Chunkref can also be used within a code chunk to include another code chunk. The third optional parameter to chunkref is a comma sepatarated list of arguments, which will replace defined parameters in the chunkref.
\def\chunkref@args#1,{%
\def\arg{#1}%
\lst@ReplaceIn\arg\lst@filenamerpl%
\arg%
\@ifnextchar){\relax}{, \chunkref@args}%
}%
\newcommand\chunkref[2][0]{%
\@ifnextchar({\chunkref@i{#1}{#2}}{\chunkref@i{#1}{#2}()}%
}%
\def\chunkref@i#1#2(#3){%
\def\zero{0}%
\def\chunk{#2}%
\def\chunkno{#1}%
\def\chunkargs{#3}%
\ifx\chunkno\zero%
\def\chunkname{#2-1}%
\else%
\def\chunkname{#2-\chunkno}%
\fi%
\let\lst@arg\chunk%
\lst@ReplaceIn\chunk\lst@filenamerpl%
\LA{%\moddef{%
{\chunk}%
{%
\nwtagstyle{}\/%
\ifx\chunkno\zero%
\else%
[\chunkno]%
\fi%
\ifx\chunkargs\empty%
\else%
(\chunkref@args #3,)%
\fi%
~\subpageref{\chunkname}%
}%
}%
\RA%\endmoddef%
}%
14 Extracting newfangle
14.1 Extracting from Lyx
To extract from LyX, you will need to configure LyX as explained in section
3.1.2?.
And this lyx-build scrap will extract newfangle for me.
lyx-build,language=sh
#! /bin/sh
set -x
=<\chunkref{lyx-build-helper}>
cd $PROJECT_DIR || exit 1
=<\chunkref{generate-documentation}>
/usr/local/bin/newfangle -R./newfangle $TEX_SRC > ./newfangle
/usr/local/bin/newfangle -R./newfangle.module $TEX_SRC > ./newfangle.module
=<\chunkref{test:helpers}>
export NEWFANGLE=./newfangle
export TMP=${TMP:-/tmp}
=<\chunkref{test:run-tests}>
# Now check that we can extract a newfangle that also passes the tests!
$NEWFANGLE -R./newfangle $TEX_SRC > ./new-newfangle
export NEWFANGLE=./new-newfangle
=<\chunkref{test:run-tests}>
test:run-tests
# run tests
$NEWFANGLE -Rpca-test.awk $TEX_SRC | awk -f - || exit 1
=<\chunkref{test:cromulence}>
=<\chunkref{test:escapes}>
=<\chunkref{test:chunk-params}>
With a lyx-build-helper
lyx-build-helper,language=sh
PROJECT_DIR="$LYX_r"
LYX_SRC="$PROJECT_DIR/${LYX_i%.tex}.lyx"
TEX_DIR="$LYX_p"
TEX_SRC="$TEX_DIR/$LYX_i"
14.2 Extracting documentation
generate-documentation
python -m elyxer --css lyx.css $LYX_SRC | \
iconv -c -f utf-8 -t ISO-8859-1//TRANSLIT | \
sed 's/UTF-8"\(.\)>/ISO-8859-1"\1>/' > www/docs/newfangle.html
( mkdir -p www/docs/literate && cd www/docs/literate && \
lyx -e latex ../../../literate.lyx && htlatex ../../../literate.tex "xhtml,fn-in" )
14.3 Extracting from the command line
First you will need the tex output, then you can extract:
lyx-build-manual,language=sh
lyx -e latex newfangle.lyx
newfangle -R./newfangle newfangle.tex > ./newfangle
newfangle -R./newfangle.module newfangle.tex > ./newfangle.module
test:helpers
passtest() {
if "$@"
then echo "Passed"
else echo "Failed"
return 1
fi
}
failtest() {
if ! "$@"
then echo "Passed"
else echo "Failed"
return 1
fi
}
15 Chunk Parameters
test:chunk-params:sub,language=,params=THING;colour
I see a ${THING},
a ${THING} of colour ${colour},
and looking closer =<\chunkref{test:chunk-params:sub:sub}(${colour})>
test:chunk-params:sub:sub,params=colour,language=
a funny shade of ${colour}
test:chunk-params:text,language=
What do you see? "=<\chunkref{test:chunk-params:sub}(joe, red)>"
Well, fancy!
Should generate output:
test:chunk-params:result
What do you see? "I see a joe,
a joe of colour red,
and looking closer a funny shade of red"
Well, fancy!
And this chunk will perform the test:
test:chunk-params
$NEWFANGLE -Rtest:chunk-params:result $TEX_SRC > $TMP/answer || exit 1
$NEWFANGLE -Rtest:chunk-params:text $TEX_SRC > $TMP/result || exit 1
passtest diff $TMP/answer $TMP/result || (echo test:chunk-params:text failed ; exit 1)
Copyright (C) 2010 Sam Liddicott