1xsbuilder.osc2002(3)  User Contributed Perl Documentation xsbuilder.osc2002(3)
2
3
4

XSBuilder - Autogenerating XS-glue Code

6       O'Reilly OpenSource Convention 2002
7
8       Gerald Richter
9
10       ecos gmbh
11
12       http://www.ecos.de
13

XSBuilder - What is it?

15   It's purpose is to automaticly generate a Perl interface to C Code
16   Solves the problem, that the Perl interface is not always uptodate with the
17       C interface
18   Saves a lot of Copy&Paste work
19   Systematical changes have to be done only once
20       For example changes in the memory management of strings.
21
22   Is part of mod_perl 2.0 build system
23       Most code is developed by Doug MacEachern. Additionaly I have
24
25       abstracted the code from mod_perl so it's useable for any C code
26       added new features like callbacks and the ability to parse comments
27       Replaced C::Scan with a Parse::RecDescent to be platform and compiler
28       independend
29
30       Goal is to replace the current mod_perl XS generation code with
31       XSBuilder
32
33   Inline versus XSBuilder
34       Inline: embed C-Code into Perl
35       XSBuilder: Create interface for existing C-libraries/applicationen
36

XSBuilder - What does it do?

38   Create Perl functions/methods for every C function
39       The function can be assigned to different packages, also automaticly by
40       inspecting the first parameter
41
42   Create a Perl class for every C structure
43       Every element of structure becomes a Perl method to get/set it's value.
44       The object can be either a scalar reference (used by mod_perl) or a
45       reference to a hash (use by Embperl), which allows to store extra data
46       by the Perl code into this hash.
47
48   Create glue code to handle callbacks
49       There several sorts of callback, not all are implemented right now
50
51   Create Perl constant subs
52       Coming soon...
53

XSBuilder - How does it work?

55   Parse the C header files
56       Extract
57
58       Functions, their arguments and return types
59       Structures and it's members
60       Constants
61       Callbacks
62
63       and create four tables which contains the results
64
65   Create the XS code
66       Input is
67
68       The source tables
69       Mapfiles which contains the mapping from C to Perl
70       Addtional C and Perl code that can be used to customize the interface
71
72       Output is
73
74       The XS files (one form every generated class)
75       Makefile.PL for every class
76       pm files
77

Parse the source

79   Create your own ParseSource class and override methods...
80           package Apache::DAV::ParseSource;
81
82           use strict;
83           use vars qw{@ISA $VERSION} ;
84           use ExtUtils::XSBuilder::ParseSource  ;
85
86           @ISA = ('ExtUtils::XSBuilder::ParseSource') ;
87           $VERSION = '0.01';
88
89           my $dav_dir = 'C:\perl\msrc\cvs\mod_dav' ;
90           my $ap_dir  = 'c:\programme\apache group\apache' ;
91
92           # ============================================================================
93           sub find_includes {
94               my $self = shift;
95               return $self->{includes} if $self->{includes};
96               my @includes = ("$ap_dir/include/ap_alloc.h", "$dav_dir/mod_dav.h") ;
97               return $self->{includes} = $self -> sort_includes (\@includes) ;
98               }
99
100           # ============================================================================
101           sub package     { 'Apache::DAV' }
102
103           # ============================================================================
104           sub preprocess {
105               my $self     = shift ;
106               $_[0] =~ s/(?:API_EXPORT)(?:_NONSTD)?\s*\(\s*(.*?)\s*\)/$1/g ;
107           }
108
109
110           1;
111
112   ...run it
113           use FindBin ;
114           use lib ($FindBin::Bin) ;
115
116           require ParseSource ;
117
118           Apache::DAV::ParseSource -> run ;
119
120   ...and you get
121           C:\perl\msrc\davint>perl xsbuilder\source_scan.pl
122           Will use mod_dav in C:\perl\msrc\cvs\mod_dav
123           Will use Apache in c:\programme\apache group\apache
124           Initialize parser
125           scan c:\programme\apache group\apache/include/ap_alloc.h ...
126           constant: APACHE_ALLOC_H
127           func:     ap_init_alloc
128           func:     ap_cleanup_alloc
129           func:     ap_make_sub_pool
130           func:     ap_destroy_pool
131           constant: ap_pool_join
132           func:     ap_pool_join
133           func:     ap_find_pool
134           func:     ap_pool_is_ancestor
135           func:     ap_clear_pool
136           func:     ap_cleanup_for_exec
137           func:     ap_palloc
138           func:     ap_pcalloc
139           func:     ap_pstrdup
140           func:     ap_pstrndup
141           func:     ap_pstrcat
142           func:     ap_pvsprintf
143             valuefield: ap_pool * : pool
144             valuefield: int : elt_size
145             valuefield: int : nelts
146             valuefield: int : nalloc
147             valuefield: char * : elts
148           struct:    (type=array_header)
149           ...
150
151   The result is stored in four tables
152       xsbuilder/tables/Apache/DAV/FuntionTable.pm
153           Contains all function, it arguments and comments
154
155       xsbuilder/tables/Apache/DAV/ConstantTable.pm
156           Contains all constants
157
158       xsbuilder/tables/Apache/DAV/StructureTable.pm
159           Contains all structures, it's members and their comments
160
161       xsbuilder/tables/Apache/DAV/CallbackTable.pm
162           Contains all callback function definitions
163

Create the map files

165   Mapfiles are used to tell XSBuilder how C datatypes, structures and
166       function aruments should be mapped into Perl ones.
167   Create your own WrapXS class and override methods
168           package Apache::DAV::WrapXS ;
169           use strict;
170           use vars qw{@ISA $VERSION} ;
171
172           use ExtUtils::XSBuilder::WrapXS ;
173
174           @ISA = ('ExtUtils::XSBuilder::WrapXS') ;
175           $VERSION = '0.01';
176
177           # ============================================================================
178           sub new_parsesource  { [ Apache::DAV::ParseSource->new ] }
179
180           # ============================================================================
181           sub my_xs_prefix  { 'davxs_' }
182
183           # ============================================================================
184           sub h_filename_prefix  { 'moddav_xs_' }
185
186           # ============================================================================
187           sub xs_includes {
188               my $self = shift ;
189               my $i = $self -> SUPER::xs_includes ;
190               my @i = grep (!/ap_alloc/, @$i) ;
191               return \@i ;
192           }
193
194   XSBuilder can create/update initial maps for you
195           use FindBin ;
196           use lib ($FindBin::Bin) ;
197
198           require ParseSource ;
199           require WrapXS ;
200
201           Apache::DAV::WrapXS->checkmaps (' ');
202
203   run it
204           C:\perl\msrc\davint>perl xsbuilder\xs_check.pl
205           Will use mod_dav in C:\perl\msrc\cvs\mod_dav
206           Will use Apache in c:\programme\apache group\apache
207           Parse xsbuilder\maps/_types.map...
208           WARNING: No *_function.map file found in xsbuilder\maps
209           WARNING: No *_callback.map file found in xsbuilder\maps
210           WARNING: No *_structure.map file found in xsbuilder\maps
211           Write xsbuilder\maps/new_function.map...
212           Write xsbuilder\maps/new_callback.map...
213           Write xsbuilder\maps/new_structure.map...
214           Write xsbuilder\maps/new_type.map...
215
216   Now we have four map files
217       new_types.map
218           Contains the mapping from C type to Perl classes
219
220       new_functions.map
221           Contains the mapping form C functions to Perl functions. Can be
222           used to reorder arguments, tell XSBuilder which arguments are
223           actualy return values and in which Perl package the function will
224           be created.
225
226       new_structures.map
227           Contains the mapping from C structures to Perl classes and defines
228           for which members a access methods should be created. You can also
229           specify if you want a "new" method for the class.
230
231       new_callbacks.map
232           Contains the mapping form C callback functions to Perl callback
233           functions. Can be used to reorder arguments, tell XSBuilder which
234           arguments are actualy return values and in which Perl package the
235           function will be created.
236
237       It's a good idea to rename the prefix from "new_" to something unique,
238       here we use "dav"
239
240       Everytime you rerun checkmaps, XSBuilder will create new_* files with
241       the items that are not already part of the other maps.
242
243   Next step is to customize the maps...

type map

245   autogenerated dav_type.map
246           DIR |
247           FILE        |
248           HANDLE      |
249           array_header        |
250           dav_buffer  |
251           dav_dyn_context     |
252           dav_dyn_hooks       |
253           dav_dyn_module      |
254           dav_dyn_provider    |
255           dav_error   |
256           dav_get_props_result        |
257           dav_hooks_liveprop  |
258           dav_hooks_locks     |
259           dav_hooks_repository        |
260           dav_if_header       |
261           dav_if_state_type   |
262           ...
263
264   Add Perl classes
265           struct array_header         | Apache::Array
266           struct dav_buffer           |
267           struct dav_datum            | Apache::DAV::Datum
268           struct dav_dyn_context      | Apache::DAV::DynContext
269           struct dav_dyn_hooks        | Apache::DAV::DynHooks
270           struct dav_dyn_module       | Apache::DAV::DynModule
271           struct dav_dyn_provider     | Apache::DAV::DynProvider
272           struct dav_error            | Apache::DAV::Error
273           struct dav_get_props_result | Apache::DAV::PropsResult
274           struct dav_hooks_db         | Apache::DAV::HooksDb
275           struct dav_hooks_liveprop   | Apache::DAV::HooksLiveprop
276           struct dav_hooks_locks      | Apache::DAV::HooksLocks
277           struct dav_hooks_repository | Apache::DAV::HooksRepository
278           struct dav_hooks_vsn        |
279           struct dav_if_header        | Apache::DAV::IfHeader
280           struct dav_if_state_list    | Apache::DAV::StateList
281           ...
282           struct pool                 | Apache::Pool
283           struct request_rec          | Apache::
284           struct server_rec           | Apache::Server
285           ...
286
287       Defines the mapping from C datatypes to Perl datatypes and classes and
288       tells XSBuilder which datatype are (like) structures
289

function map

291       Function map defines the mapping from C functions arguments to Perl
292       arguments
293
294       Tell XSBuilder where to place functions and which prefix to strip
295               MODULE=Apache::DAV   PACKAGE=guess  PREFIX=dav_
296
297       Simple entries in the function map will be mapped 1:1 from C to Perl
298                dav_add_response
299                dav_buffer_append
300                dav_buffer_init
301                dav_buffer_place
302                dav_buffer_place_mem
303                dav_check_bufsize
304                dav_close_propdb
305                dav_collect_liveprop_uris
306                dav_dyn_module_add
307                dav_empty_elem
308               ...
309
310       The following map file entry tells XSBuilder that the value of "res_p"
311       should be returned
312                dav_get_resource           | | r, <res_p
313
314           The C function
315
316               int dav_get_resource(request_rec *r, dav_resource **res_p) ;
317
318           How to call it from Perl
319
320               # get a sub request of the actual document
321               $subr = $r -> lookup_uri($uri);
322
323               # get a mod_dav resource object
324               my ($err, $resource) =  $subr->get_resource;
325
326       You can let XSBuilder insert your custom code, for the interface
327           If you call "get_props" from Perl "glue_dav_get_props" will be
328           called, which can adjust the arguments and return types as
329           necessary. The actual code for "glue_dav_get_props" will be taken
330           from separate include file.
331
332                dav_get_props      | glue_
333                dav_get_allprops   | glue_
334
335           xsinclude\Apache\DAV\PropResult\Apache__DAV__PropResults.h
336
337               dav_get_props_result * dav_glue_get_props(dav_propdb * db, dav_xml_doc *doc)
338                   {
339                   dav_get_props_result * result = (dav_get_props_result *)ap_palloc (db -> p, sizeof (dav_get_props_result)) ;
340                   *result = dav_get_props(db, doc) ;
341                   return result ;
342                   }
343
344       Arguments can be replaced
345               MODULE=Apache::Array   PACKAGE=Apache::Array PREFIX=ap_
346
347                ap_make_array      | ap_make_array(r->pool, nelts, elt_size) | request_rec *:r, nelts, elt_size
348
349           ap_make_array requires a pool a it's first parameter, we pass the
350           request_rec from Perl and XSBuilder will take the pool from the
351           request_rec.
352

structure map

354            MALLOC=strdup:$dest = ($type)strdup($src)
355            FREE=strdup:free($src)
356
357            <dav_xml_elem>
358              name
359              ns
360              lang
361              first_cdata
362              following_cdata
363              parent
364              next
365              first_child
366              attr
367              last_child
368              ns_scope
369              propid
370              provider
371              ns_map
372              new
373            </dav_xml_elem>
374
375            MALLOC=strdup:$dest = ($type)ap_pstrdup(obj -> pool, $src)
376
377            <array_header>
378              pool
379              elt_size
380              nelts
381              nalloc
382              elts
383              private
384            </array_header>
385
386   Create a accessor functions for every element and, if requested, a new
387       method
388               $setprop = Apache::DAV::XMLElem -> new ({name => 'prop'}) ;
389               $elem    = Apache::DAV::XMLElem -> new ({name => $name, ns => $namespaces}) ;
390               $setprop -> first_child($elem) ;
391               $first = $setprop -> first_child ;
392
393   some datatypes, like strings, requires dynamic allocated memory
394       From _types.map,which conatins a set of standard types
395
396           int             | IV
397           int *           | UNDEFINED
398           unsigned int    | UV
399           signed int      | IV
400           long            | IV
401           long int        | IV
402           unsigned long   | UV
403           unsigned        | UV
404
405           char *                   | PV           | | | strdup
406           const char *             | PV           | | | strdup
407           const char **            | UNDEFINED
408           char const *             | PV           | | | strdup
409           unsigned char *          | PV           | | | strdup
410           const unsigned char *    | PV           | | | strdup
411           ...
412

callback maps

414       Callback maps have the same options a function maps
415
416           # dav_hooks_db -> open
417
418            dav_error *(*)(pool * p,const dav_resource * resource,int ro,dav_db * * pdb)  | p, resource, ro=0, <pdb
419
420
421           # dav_hooks_lock -> has_locks
422
423            dav_error *(*)(dav_lockdb * lockdb,const dav_resource * resource,int * locks_present)
424

Generate the XS files

426           use FindBin ;
427           use lib ($FindBin::Bin) ;
428
429           require ParseSource ;
430           require WrapXS ;
431
432           Apache::DAV::WrapXS->run;
433
434   ...and run...
435           C:\perl\msrc\davint>perl xsbuilder\xs_generate.pl
436           Will use mod_dav in C:\perl\msrc\cvs\mod_dav
437           Will use Apache in c:\programme\apache group\apache
438           Parse xsbuilder\maps/_types.map...
439           Parse xsbuilder\maps/dav_type.map...
440           mkdir xs
441           writing...xs//typemap
442           Parse xsbuilder\maps/dav_functions.map...
443           WARNING: Cannot map type int(*)(void * ,const char * ,const char * ) for function ap_table_do
444           WARNING: Cannot map type dav_buffer * for function dav_buffer_append
445           WARNING: Cannot map type dav_buffer * for function dav_buffer_init
446           WARNING: Cannot map type dav_buffer * for function dav_buffer_place
447           WARNING: Cannot map type dav_buffer * for function dav_buffer_place_mem
448           WARNING: Cannot map type dav_buffer * for function dav_check_bufsize
449           WARNING: Cannot map return type int * for function dav_collect_liveprop_uris
450           WARNING: Cannot map type dav_resource * * for function dav_ensure_resource_writable
451           WARNING: Cannot map type dav_buffer * for function dav_lock_get_activelock
452           WARNING: Cannot map type dav_buffer * for function dav_set_bufsize
453           WARNING: Cannot map type int * for function dav_xml2text
454           struct array_header...
455           Parse xsbuilder\maps/dav_structure.map...
456                elt_size...
457                nelts...
458                nalloc...
459                elts...
460           struct dav_buffer...
461           struct dav_datum...
462                dptr...
463                dsize...
464           struct dav_dyn_context...
465

Makefile.PL

467   We need create a top level Makefile.PL
468           use ExtUtils::MakeMaker ();
469
470           my $apdir  = '/path/to/apache';
471           my $davdir = '/path/to/moddav';
472
473           %MMARGS = (
474                      'INC'    => "-I\"$davdir\" -I\"$apdir/include\" -I\"$apdir/os/unix\" -I\"$dir/xs\"  -I\"$dir/xsinclude\"",
475                      ) ;
476
477
478           open FH, ">xs/mmargs.pl" or die "Cannot open xs/mmargs.pl ($!)" ;
479           print FH Data::Dumper -> Dump ([\%MMARGS], ['MMARGS']) ;
480           close FH ;
481
482
483           ExtUtils::MakeMaker::WriteMakefile(
484               'NAME'    => 'Apache::DAV',
485               'VERSION' => '0.13',
486               %MMARGS,
487           );
488
489   Makefile.PL's for all class are generated automaticly
490           C:\perl\msrc\davint>perl Makefile.PL
491           Will use Apache in c:\programme\apache group\apache
492           Will use mod_dav in C:\perl\msrc\cvs\mod_dav
493           Checking if your kit is complete...
494           Looks good
495           Writing Makefile for Apache::Array
496           Writing Makefile for Apache::DAV::Datum
497           Writing Makefile for Apache::DAV::DynContext
498           Writing Makefile for Apache::DAV::DynHooks
499           Writing Makefile for Apache::DAV::DynModule
500           Writing Makefile for Apache::DAV::DynProvider
501           Writing Makefile for Apache::DAV::Error
502           Writing Makefile for Apache::DAV::HooksDb
503           Writing Makefile for Apache::DAV::HooksLiveprop
504           Writing Makefile for Apache::DAV::HooksLocks
505           Writing Makefile for Apache::DAV::HooksRepository
506           Writing Makefile for Apache::DAV::IfHeader
507           Writing Makefile for Apache::DAV::Lock
508           Writing Makefile for Apache::DAV::LockDB
509           Writing Makefile for Apache::DAV::LockTokenList
510           Writing Makefile for Apache::DAV::LockupResult
511           Writing Makefile for Apache::DAV::PropCtx
512           Writing Makefile for Apache::DAV::PropsResult
513           Writing Makefile for Apache::DAV::Resource
514           Writing Makefile for Apache::DAV::Response
515           Writing Makefile for Apache::DAV::StateList
516           Writing Makefile for Apache::DAV::Text
517           Writing Makefile for Apache::DAV::TextHeader
518           Writing Makefile for Apache::DAV::WalkerCtx
519           Writing Makefile for Apache::DAV::XMLAttr
520           Writing Makefile for Apache::DAV::XMLDoc
521           Writing Makefile for Apache::DAV::XMLElem
522           Writing Makefile for Apache::DAV
523           Writing Makefile for Apache::TableEntry
524           Writing Makefile for Apache
525           Writing Makefile for WrapXS
526           Writing Makefile for Apache::DAV
527
528   and now compile...

How does it go on...

530   Generating documentation
531       XSBuilder already extracts source comments for functions and
532       structures.  It also parses doxygen comments, which are used in Apache
533       2.0.  Lyle Brooks has started on automaticly createing POD files from
534       this information.
535
536   Improving callbacks
537       Callbacks are the main area that needs improvement.
538
539   Bring it back to mod_perl 2.0
540   First version will be released just after the conference to CPAN
541   Any feedback and help appreciated
542   Questions?
543perl v5.28.0                      2002-08-05              xsbuilder.osc2002(3)
Impressum