1Roadmap(3) User Contributed Perl Documentation Roadmap(3)
2
6 DBI::Roadmap - Planned Enhancements for the DBI
7 Tim Bunce - 12th November 2004
9
11 This document gives a high level overview of the future of the Perl DBI
12 module.
13 The DBI module is the standard database interface for Perl
15 applications. It is used worldwide in countless applications, in every
16 kind of business, and on platforms from clustered super-computers to
17 PDAs. Database interface drivers are available for all common
18 databases and many not-so-common ones.
19 The planned enhancements cover testing, performance, high availability
21 and load balancing, batch statements, Unicode, database portability,
22 and more.
23 Addressing these issues together, in coordinated way, will help ensure
25 maximum future functionality with minimal disruptive (incompatible)
26 upgrades.
27
29 Broad categories of changes are outlined here along with some
30 rationale, but implementation details and minor planned enhancements
31 are omitted. More details can be found in:
32 <http://svn.perl.org/modules/dbi/trunk/ToDo>
33
35 These are grouped into categories and are not listed in any particular
36 order.
37 Performance
39 The DBI has always treated performance as a priority. Some parts of the
40 implementation, however, remain unoptimized, especially in relation to
41 threads.
42 * When the DBI is used with a Perl built with thread support enabled
44 (such as for Apache mod_perl 2, and some common Linux distributions) it
45 runs significantly slower. There are two reasons for this and both can
46 be fixed but require non-trivial changes to both the DBI and drivers.
47 * Connection pooling in a threaded application, such as mod_perl, is
49 difficult because DBI handles cannot be passed between threads. An
50 alternative mechanism for passing connections between threads has been
51 defined, and an experimental connection pool module implemented using
52 it, but development has stalled.
53 * The majority of DBI handle creation code is implemented in Perl.
55 Moving most of this to C will speed up handle creation significantly.
56 * The popular fetchrow_hashref() method is many times slower than
58 fetchrow_arrayref(). It has to get the names of the columns, then
59 create and load a new hash each time. A $h->{FetchHashReuse} attribute
60 would allow the same hash to be reused each time making
61 fetchrow_hashref() about the same speed as fetchrow_arrayref().
62 * Support for asynchronous (non-blocking) DBI method calls would enable
64 applications to continue processing in parallel with database activity.
65 This is also relevant for GUI and other event-driven applications. The
66 DBI needs to define a standard interface for this so drivers can
67 implement it in a portable way, where possible.
68 These changes would significantly enhance the performance of the DBI
70 and many applications which use the DBI.
71 Testing
73 The DBI has a test suite. Every driver has a test suite. Each is
74 limited in its scope. The driver test suite is testing for behavior
75 that the driver author thinks the DBI specifies, but may be subtly
76 incorrect. These test suites are poorly maintained because the return
77 on investment for a single driver is too low to provide sufficient
78 incentive.
79 A common test suite that can be reused by all the drivers is needed.
81 It would:
82 * Improve the quality of the DBI and drivers.
84 * Ensure all drivers conform to the DBI specification. Easing the
86 porting of applications between databases, and the implementation of
87 database independent modules layered over the DBI.
88 * Improve the DBI specification by clarifying unclear issues in order
90 to implement test cases.
91 * Encourage expansion of the test suite as driver authors and others
93 will be motivated by the greater benefits of their contributions.
94 * Detect and record optional functionality that a driver has not yet
96 implemented.
97 * Improve the testing of DBI subclassing, DBI::PurePerl and the various
99 "transparent" drivers, such as DBD::Proxy and DBD::Multiplex, by
100 automatically running the test suite through them.
101 These changes would improve the quality of all applications using the
103 DBI.
104 High Availability and Load Balancing
106 * The DBD::Multiplex driver provides a framework to enable a wide range
107 of dynamic functionality, including support for high-availability,
108 failover, load-balancing, caching, and access to distributed data. It
109 is currently being enhanced but development has stalled.
110 * The DBD::Proxy module is complex and relatively inefficient because
112 it's trying to be a complete proxy for most DBI method calls. For many
113 applications a simpler proxy architecture that operates with a single
114 round-trip to the server would be simpler, faster, and more flexible.
115 New proxy client and server classes are needed, which could be
117 subclassed to support specific client to server transport mechanisms
118 (such as HTTP and Spread::Queue). Apart from the efficiency gains,
119 this would also enable the use of a load-balanced pool of stateless
120 servers for greater scalability and reliability.
121 * The DBI currently offers no support for distributed transactions.
123 The most useful elements of the standard XA distributed transaction
124 interface standard could be included in the DBI specification. Drivers
125 for databases which support distributed transactions could then be
126 extended to support it.
127 These changes would enable new kinds of DBI applications for critical
129 environments.
130 Unicode
132 Use of Unicode with the DBI is growing rapidly. The DBI should do more
133 to help drivers support Unicode and help applications work with drivers
134 that don't yet support Unicode directly.
135 * Define expected behavior for fetching data and binding parameters.
137 * Provide interfaces to support Unicode issues for XS and pure Perl
139 drivers and applications.
140 * Provide functions for applications to help diagnose inconsistencies
142 between byte string contents and setting of the SvUTF8 flag.
143 These changes would smooth the transition to Unicode for many
145 applications and drivers.
146 Batch Statements
148 Batch statements are a sequence of SQL statements, or a stored
149 procedure containing a sequence of SQL statements, which can be
150 executed as a whole.
151 Currently the DBI has no standard interface for dealing with multiple
153 results from batch statements. After considerable discussion, an
154 interface design has been agreed upon with driver authors, but has not
155 yet been implemented.
156 These changes would enable greater application portability between
158 databases, and greater performance for databases that directly support
159 batch statements.
160 Introspection
162 * The methods of the DBI API are installed dynamically when the DBI is
163 loaded. The data structure used to define the methods and their
164 dispatch behavior should be made part of the DBI API. This would enable
165 more flexible and correct behavior by modules subclassing the DBI and
166 by dynamic drivers such as DBD::Proxy and DBD::Multiplex.
167 * Handle attribute information should also be made available, for the
169 same reasons.
170 * Currently is it not possible to discover all the child statement
172 handles that belong to a database handle (or all database handles that
173 belong to a driver handle). This makes certain tasks more difficult,
174 especially some debugging scenarios. A cache of weak references to
175 child handles would solve the problem without creating reference loops.
176 * It is often useful to know which handle attributes have been changed
178 since the handle was created (e.g., in mod_perl where a handle needs to
179 be reset or cloned). This will become more important as developers
180 start exploring use of the newly added $h1->swap_inner_handle($h2)
181 method.
182 These changes would simplify and improve the stability of many advanced
184 uses of the DBI.
185 Extensibility
187 The DBI can be extended in three main dimensions: subclassing the DBI,
188 subclassing a driver, and callback hooks. Each has different pros and
189 cons, each is applicable in different situations, and all need
190 enhancing.
191 * Subclassing the DBI is functional but not well defined and some key
193 elements are incomplete, particularly the DbTypeSubclass mechanism
194 (that automatically subclasses to a class tree according to the type of
195 database being used). It also needs more thorough testing.
196 * Subclassing a driver is undocumented, poorly tested and very probably
198 incomplete. However it's a powerful way to embed certain kinds of
199 functionality 'below' applications while avoiding some of the side-
200 effects of subclassing the DBI (especially in relation to error
201 handling).
202 * Callbacks are currently limited to error handling (the HandleError
204 and HandleSetError attributes). Providing callback hooks for more
205 events, such as a row being fetched, would enable utility modules, for
206 example, to modify the behavior of a handle independent of any
207 subclassing in use.
208 These changes would enable cleaner and more powerful integration
210 between applications, layered modules, and the DBI.
211 Debugability
213 * Enabling DBI trace output at a high level of detail causes a large
214 volume of output, much of it probably unrelated to the problem being
215 investigated. Trace output should be controlled by the new named-topic
216 mechanism instead of just the trace level.
217 * Calls to XS functions (such as many DBI and driver methods) don't
219 normally appear in the call stack. Optionally enabling that would
220 enable more useful diagnostics to be produced.
221 * Integration with the Perl debugger would make it simpler to perform
223 actions on a per-handle basis (such as breakpoint on execute,
224 breakpoint on error).
225 These changes would enable more rapid application development and fault
227 finding.
228 Database Portability
230 * The DBI has not yet addressed the issue of portability among SQL
231 dialects. This is the main hurdle limiting database portability for
232 DBI applications.
233 The goal is not to fully parse the SQL and rewrite it in a different
235 dialect. That's well beyond the scope of the DBI and should be left to
236 layered modules. A simple token rewriting mechanism for five comment
237 styles, two quoting styles, four placeholder styles, plus the ODBC
238 "{foo ...}" escape syntax, is sufficient to significantly raise the
239 level of SQL portability.
240 * Another problem area is date/time formatting. Since version 1.41 the
242 DBI has defined a way to express that dates should be fetched in SQL
243 standard date format (YYYY-MM-DD). This is one example of the more
244 general case where bind_col() needs to be called with particular
245 attributes on all columns of a particular type.
246 A mechanism is needed whereby an application can specify default
248 bind_col() attributes to be applied automatically for each column type.
249 With a single step, all DATE type columns, for example, can be set to
250 be returned in the standard format.
251 These changes would enable greater database portability for
253 applications and greater functionality for layered modules.
254 Intellectual Property
256 * Clarify current intellectual property status, including a review
257 of past contributions to ensure the DBI is unemcumbered.
258 * Establish a procedure for vetting future contributions for any
260 intellectual property issues.
261 These changes are important for companies taking a formal approach to
263 assessing their risks in using Open Source software.
264 Other Enhancements
266 * Reduce the work needed to create new database interface drivers.
267 * Definition of an interface to support scrollable cursors.
269 Parrot and Perl 6
271 The current DBI implementation in C code is unlikely to run on Perl 6.
272 Perl 6 will target the Parrot virtual machine and so the internal
273 architecture will be radically different from Perl 5.
274 One of the goals of the Parrot project is to be a platform for many
276 dynamic languages (including Python, PHP, Ruby, etc) and to enable
277 those languages to reuse each others modules. A database interface for
278 Parrot is also a database interface for any and all languages that run
279 on Parrot.
280 The Perl DBI would make an excellent base for a Parrot database
282 interface because it has more functionality, and is more mature and
283 extensible, than the database interfaces of the other dynamic
284 languages.
285 I plan to better define the API between the DBI and the drivers and use
287 that API as the primary API for the 'raw' Parrot database interface.
288 This project is known a Parrot DBDI (for "DataBase Driver Interface").
289 The announcement can be read in
290 <http://groups.google.com/groups?selm=20040127225639.GF38394@dansat.data-plan.com>
291 The bulk of the work will be translating the DBI C and Perl base class
293 code into Parrot PIR, or a suitable language that generates PIR. The
294 project stalled, due to Parrot not having key functionality at the
295 time. That has been resolved but the project has not yet restarted.
296 Each language targeting Parrot would implement their own small 'thin'
298 language-specific method dispatcher (a "Perl6 DBI", "Python DBI", "PHP
299 DBI" etc) layered over the common Parrot DBDI interface and drivers.
300 The major benefit of the DBDI project is that a much wider community of
302 developers share the same database drivers. There would be more
303 developers maintaining less code so the benefits of the Open Source
304 model are magnified.
305
307 Transition Drivers
308 The first priority is to make all the infrastructure changes that
309 impact drivers and make an alpha release available for driver authors.
310 As far as possible, the changes will be implemented in a way that
312 enables driver authors use the same code base for DBI v1 and DBI v2.
313 The main changes required by driver authors are:
315 * Code changes for PERL_NO_GET_CONTEXT, plus removing PERL_POLLUTE and
317 DBIS
318 * Code changes in DBI/DBD interface (new way to create handles, new
320 callbacks etc)
321 * Common test suite infrastructure (driver-specific test base class)
323 Transition Applications
325 A small set of incompatible changes that may impact some applications
326 will also be made in v2.0. See
327 http://svn.perl.org/modules/dbi/trunk/ToDo
328 Incremental Developments
330 Once DBI v2.0 is available, the other enhancements can be implemented
331 incrementally on the updated foundations. Priorities for those changes
332 have not been set.
333 DBI v1
335 DBI v1 will continue to be maintained on a separate branch for bug
336 fixes and any enhancements that ease the transition to DBI v2.
337
339 See <http://dbi.perl.org/contributing> for how you can help.
340 If your company has benefited from the DBI, please consider if it could
342 make a donation to The Perl Foundation "DBI Development" fund at
343 <http://dbi.perl.org/donate> to secure future development.
344 Alternatively, if your company would benefit from a specific new DBI
346 feature, please consider sponsoring its development through my
347 consulting company, Data Plan Services. Work is performed rapidly on a
348 fixed-price payment-on-delivery basis. Contact me for details.
349 Using such targeted financing allows you to contribute to DBI
351 development and rapidly get something specific and directly valuable to
352 you in return.
353 My company also offers annual support contracts for the DBI, which
355 provide another way to support the DBI and get something specific in
356 return. Contact me for details.
357 Thank you.
359perl v5.10.1 2005-03-25 Roadmap(3)