1Roadmap(3) User Contributed Perl Documentation Roadmap(3)
2
6 DBI::Roadmap - Planned Enhancements for the DBI
7 Tim Bunce - 12th November 2004
9
11 This document gives a high level overview of the future of the Perl DBI
12 module.
13 The DBI module is the standard database interface for Perl applica‐
15 tions. It is used worldwide in countless applications, in every kind
16 of business, and on platforms from clustered super-computers to PDAs.
17 Database interface drivers are available for all common databases and
18 many not-so-common ones.
19 The planned enhancements cover testing, performance, high availability
21 and load balancing, batch statements, Unicode, database portability,
22 and more.
23 Addressing these issues together, in coordinated way, will help ensure
25 maximum future functionality with minimal disruptive (incompatible)
26 upgrades.
27
29 Broad categories of changes are outlined here along with some ratio‐
30 nale, but implementation details and minor planned enhancements are
31 omitted. More details can be found in: <http://svn.perl.org/mod‐
32 ules/dbi/trunk/ToDo>
33
35 These are grouped into categories and are not listed in any particular
36 order.
37 Performance
39 The DBI has always treated performance as a priority. Some parts of the
41 implementation, however, remain unoptimized, especially in relation to
42 threads.
43 * When the DBI is used with a Perl built with thread support enabled
45 (such as for Apache mod_perl 2, and some common Linux distributions) it
46 runs significantly slower. There are two reasons for this and both can
47 be fixed but require non-trivial changes to both the DBI and drivers.
48 * Connection pooling in a threaded application, such as mod_perl, is
50 difficult because DBI handles cannot be passed between threads. An
51 alternative mechanism for passing connections between threads has been
52 defined, and an experimental connection pool module implemented using
53 it, but development has stalled.
54 * The majority of DBI handle creation code is implemented in Perl.
56 Moving most of this to C will speed up handle creation significantly.
57 * The popular fetchrow_hashref() method is many times slower than
59 fetchrow_arrayref(). It has to get the names of the columns, then cre‐
60 ate and load a new hash each time. A $h->{FetchHashReuse} attribute
61 would allow the same hash to be reused each time making
62 fetchrow_hashref() about the same speed as fetchrow_arrayref().
63 * Support for asynchronous (non-blocking) DBI method calls would enable
65 applications to continue processing in parallel with database activity.
66 This is also relevant for GUI and other event-driven applications. The
67 DBI needs to define a standard interface for this so drivers can imple‐
68 ment it in a portable way, where possible.
69 These changes would significantly enhance the performance of the DBI
71 and many applications which use the DBI.
72 Testing
74 The DBI has a test suite. Every driver has a test suite. Each is lim‐
76 ited in its scope. The driver test suite is testing for behavior that
77 the driver author thinks the DBI specifies, but may be subtly incor‐
78 rect. These test suites are poorly maintained because the return on
79 investment for a single driver is too low to provide sufficient incen‐
80 tive.
81 A common test suite that can be reused by all the drivers is needed.
83 It would:
84 * Improve the quality of the DBI and drivers.
86 * Ensure all drivers conform to the DBI specification. Easing the
88 porting of applications between databases, and the implementation of
89 database independent modules layered over the DBI.
90 * Improve the DBI specification by clarifying unclear issues in order
92 to implement test cases.
93 * Encourage expansion of the test suite as driver authors and others
95 will be motivated by the greater benefits of their contributions.
96 * Detect and record optional functionality that a driver has not yet
98 implemented.
99 * Improve the testing of DBI subclassing, DBI::PurePerl and the various
101 "transparent" drivers, such as DBD::Proxy and DBD::Multiplex, by auto‐
102 matically running the test suite through them.
103 These changes would improve the quality of all applications using the
105 DBI.
106 High Availability and Load Balancing
108 * The DBD::Multiplex driver provides a framework to enable a wide range
110 of dynamic functionality, including support for high-availability,
111 failover, load-balancing, caching, and access to distributed data. It
112 is currently being enhanced but development has stalled.
113 * The DBD::Proxy module is complex and relatively inefficient because
115 it's trying to be a complete proxy for most DBI method calls. For many
116 applications a simpler proxy architecture that operates with a single
117 round-trip to the server would be simpler, faster, and more flexible.
118 New proxy client and server classes are needed, which could be sub‐
120 classed to support specific client to server transport mechanisms (such
121 as HTTP and Spread::Queue). Apart from the efficiency gains, this
122 would also enable the use of a load-balanced pool of stateless servers
123 for greater scalability and reliability.
124 * The DBI currently offers no support for distributed transactions.
126 The most useful elements of the standard XA distributed transaction
127 interface standard could be included in the DBI specification. Drivers
128 for databases which support distributed transactions could then be
129 extended to support it.
130 These changes would enable new kinds of DBI applications for critical
132 environments.
133 Unicode
135 Use of Unicode with the DBI is growing rapidly. The DBI should do more
137 to help drivers support Unicode and help applications work with drivers
138 that don't yet support Unicode directly.
139 * Define expected behavior for fetching data and binding parameters.
141 * Provide interfaces to support Unicode issues for XS and pure Perl
143 drivers and applications.
144 * Provide functions for applications to help diagnose inconsistencies
146 between byte string contents and setting of the SvUTF8 flag.
147 These changes would smooth the transition to Unicode for many applica‐
149 tions and drivers.
150 Batch Statements
152 Batch statements are a sequence of SQL statements, or a stored proce‐
154 dure containing a sequence of SQL statements, which can be executed as
155 a whole.
156 Currently the DBI has no standard interface for dealing with multiple
158 results from batch statements. After considerable discussion, an
159 interface design has been agreed upon with driver authors, but has not
160 yet been implemented.
161 These changes would enable greater application portability between
163 databases, and greater performance for databases that directly support
164 batch statements.
165 Introspection
167 * The methods of the DBI API are installed dynamically when the DBI is
169 loaded. The data structure used to define the methods and their dis‐
170 patch behavior should be made part of the DBI API. This would enable
171 more flexible and correct behavior by modules subclassing the DBI and
172 by dynamic drivers such as DBD::Proxy and DBD::Multiplex.
173 * Handle attribute information should also be made available, for the
175 same reasons.
176 * Currently is it not possible to discover all the child statement han‐
178 dles that belong to a database handle (or all database handles that
179 belong to a driver handle). This makes certain tasks more difficult,
180 especially some debugging scenarios. A cache of weak references to
181 child handles would solve the problem without creating reference loops.
182 * It is often useful to know which handle attributes have been changed
184 since the handle was created (e.g., in mod_perl where a handle needs to
185 be reset or cloned). This will become more important as developers
186 start exploring use of the newly added $h1->swap_inner_handle($h2)
187 method.
188 These changes would simplify and improve the stability of many advanced
190 uses of the DBI.
191 Extensibility
193 The DBI can be extended in three main dimensions: subclassing the DBI,
195 subclassing a driver, and callback hooks. Each has different pros and
196 cons, each is applicable in different situations, and all need enhanc‐
197 ing.
198 * Subclassing the DBI is functional but not well defined and some key
200 elements are incomplete, particularly the DbTypeSubclass mechanism
201 (that automatically subclasses to a class tree according to the type of
202 database being used). It also needs more thorough testing.
203 * Subclassing a driver is undocumented, poorly tested and very probably
205 incomplete. However it's a powerful way to embed certain kinds of func‐
206 tionality 'below' applications while avoiding some of the side-effects
207 of subclassing the DBI (especially in relation to error handling).
208 * Callbacks are currently limited to error handling (the HandleError
210 and HandleSetError attributes). Providing callback hooks for more
211 events, such as a row being fetched, would enable utility modules, for
212 example, to modify the behavior of a handle independent of any sub‐
213 classing in use.
214 These changes would enable cleaner and more powerful integration
216 between applications, layered modules, and the DBI.
217 Debugability
219 * Enabling DBI trace output at a high level of detail causes a large
221 volume of output, much of it probably unrelated to the problem being
222 investigated. Trace output should be controlled by the new named-topic
223 mechanism instead of just the trace level.
224 * Calls to XS functions (such as many DBI and driver methods) don't
226 normally appear in the call stack. Optionally enabling that would
227 enable more useful diagnostics to be produced.
228 * Integration with the Perl debugger would make it simpler to perform
230 actions on a per-handle basis (such as breakpoint on execute, break‐
231 point on error).
232 These changes would enable more rapid application development and fault
234 finding.
235 Database Portability
237 * The DBI has not yet addressed the issue of portability among SQL
239 dialects. This is the main hurdle limiting database portability for
240 DBI applications.
241 The goal is not to fully parse the SQL and rewrite it in a different
243 dialect. That's well beyond the scope of the DBI and should be left to
244 layered modules. A simple token rewriting mechanism for five comment
245 styles, two quoting styles, four placeholder styles, plus the ODBC
246 "{foo ...}" escape syntax, is sufficient to significantly raise the
247 level of SQL portability.
248 * Another problem area is date/time formatting. Since version 1.41 the
250 DBI has defined a way to express that dates should be fetched in SQL
251 standard date format (YYYY-MM-DD). This is one example of the more
252 general case where bind_col() needs to be called with particular
253 attributes on all columns of a particular type.
254 A mechanism is needed whereby an application can specify default
256 bind_col() attributes to be applied automatically for each column type.
257 With a single step, all DATE type columns, for example, can be set to
258 be returned in the standard format.
259 These changes would enable greater database portability for applica‐
261 tions and greater functionality for layered modules.
262 Intellectual Property
264 * Clarify current intellectual property status, including a review
266 of past contributions to ensure the DBI is unemcumbered.
267 * Establish a procedure for vetting future contributions for any
269 intellectual property issues.
270 These changes are important for companies taking a formal approach to
272 assessing their risks in using Open Source software.
273 Other Enhancements
275 * Reduce the work needed to create new database interface drivers.
277 * Definition of an interface to support scrollable cursors.
279 Parrot and Perl 6
281 The current DBI implementation in C code is unlikely to run on Perl 6.
283 Perl 6 will target the Parrot virtual machine and so the internal
284 architecture will be radically different from Perl 5.
285 One of the goals of the Parrot project is to be a platform for many
287 dynamic languages (including Python, PHP, Ruby, etc) and to enable
288 those languages to reuse each others modules. A database interface for
289 Parrot is also a database interface for any and all languages that run
290 on Parrot.
291 The Perl DBI would make an excellent base for a Parrot database inter‐
293 face because it has more functionality, and is more mature and extensi‐
294 ble, than the database interfaces of the other dynamic languages.
295 I plan to better define the API between the DBI and the drivers and use
297 that API as the primary API for the 'raw' Parrot database interface.
298 This project is known a Parrot DBDI (for "DataBase Driver Interface").
299 The announcement can be read in
300 <http://groups.google.com/groups?selm=20040127225639.GF38394@dansat.data-plan.com>
301 The bulk of the work will be translating the DBI C and Perl base class
303 code into Parrot PIR, or a suitable language that generates PIR. The
304 project stalled, due to Parrot not having key functionality at the
305 time. That has been resolved but the project has not yet restarted.
306 Each language targeting Parrot would implement their own small 'thin'
308 language-specific method dispatcher (a "Perl6 DBI", "Python DBI", "PHP
309 DBI" etc) layered over the common Parrot DBDI interface and drivers.
310 The major benefit of the DBDI project is that a much wider community of
312 developers share the same database drivers. There would be more devel‐
313 opers maintaining less code so the benefits of the Open Source model
314 are magnified.
315
317 Transition Drivers
318 The first priority is to make all the infrastructure changes that
320 impact drivers and make an alpha release available for driver authors.
321 As far as possible, the changes will be implemented in a way that
323 enables driver authors use the same code base for DBI v1 and DBI v2.
324 The main changes required by driver authors are:
326 * Code changes for PERL_NO_GET_CONTEXT, plus removing PERL_POLLUTE and
328 DBIS
329 * Code changes in DBI/DBD interface (new way to create handles, new
331 callbacks etc)
332 * Common test suite infrastructure (driver-specific test base class)
334 Transition Applications
336 A small set of incompatible changes that may impact some applications
338 will also be made in v2.0. See http://svn.perl.org/mod‐
339 ules/dbi/trunk/ToDo
340 Incremental Developments
342 Once DBI v2.0 is available, the other enhancements can be implemented
344 incrementally on the updated foundations. Priorities for those changes
345 have not been set.
346 DBI v1
348 DBI v1 will continue to be maintained on a separate branch for bug
350 fixes and any enhancements that ease the transition to DBI v2.
351
353 See <http://dbi.perl.org/contributing> for how you can help.
354 If your company has benefited from the DBI, please consider if it could
356 make a donation to The Perl Foundation "DBI Development" fund at
357 <http://dbi.perl.org/donate> to secure future development.
358 Alternatively, if your company would benefit from a specific new DBI
360 feature, please consider sponsoring its development through my consult‐
361 ing company, Data Plan Services. Work is performed rapidly on a fixed-
362 price payment-on-delivery basis. Contact me for details.
363 Using such targeted financing allows you to contribute to DBI develop‐
365 ment and rapidly get something specific and directly valuable to you in
366 return.
367 My company also offers annual support contracts for the DBI, which pro‐
369 vide another way to support the DBI and get something specific in
370 return. Contact me for details.
371 Thank you.
373perl v5.8.8 2006-02-07 Roadmap(3)