About the Project
The Flash DataList
Developed in 2003 using ActionScript 1 and Flash 6, the Flash DataList is the easy way to provide an intuitive interface for managing the data in remote database tables. Speed optimised data transfer techniques mean that thousands of rows are loaded in seconds and the user can get on with editing their table-based data quickly using a familiar excel-style click-and-type interface, a great step forward over the traditional rather clunky click-and-wait user interfaces that HTML based editors typically offer for remote data editing. Installation is straightforward with just two HTML pages needed, one for embedding the client-side DataList SWF and the second for handling the data transfer between the DataList SWF and the database.
But why not just use the Macromedia DataGrid?
Despite appearances the DataList component isn't just a wrapper around one of the various Macromedia supplied DataGrid components, it's actually been built completely from scratch. One of the reasons for this is that the original Macromedia components simple weren't up to the job; although they look very pretty they just weren't built to handle real world data (with the components becoming internally inefficent when loaded with more than a few records). Also their one-size-fits-all approach to databinding leads to huge inefficiencies with the data transport routines, perhaps they simple didn't believe that databases with more than a hundred rows could actually exist?
So why is this different to the Macromedia Component?
The Flash DataList was written completely from scratch with all speed critical routines being maticulously tested and optimised for use with thousands of records. Scrolling, sorting, loading, rendering and data editing all operate quickly and flawlessly - in fact using the DataList doesn't feel like using a web-based interface, it feels more like using a desktop application.
Also the DataList has been designed to work as a standalone HTML website component, and not necessarily as part of a Flash based editing suite. This means that the DataList component can be just rolled out and reconfigured each time, it's just a case of editing some HTML and PHP or ASP code, no need to worry about using the Flash authoring environment.
What different methods were tried for the data transfer?
The key to the successful use of this component is the fast downloading of all the database data. Various techniques were tried for the data transfer, some traditional, some experimental. Here's a summary of the findings:
Flash Remoting: Using phpAMF, results were good but the downsides were the overhead of massaging the data between all the different formats and also the implementation complexities of the scripts required to generate the AMF packets. It may have been worth trying to generate the raw binary AMF packets straight from the database data as this would have reduced the processing steps and may have resulted in better performance.
Pure XML: Although this is the technical purist's solution, XML isn't known for being a size efficient data transport method. It's very speedy to get the server-side scripts to build and serve the XML data, but somewhat less efficient for ActionScript to work through the processed XML. The problem is due to a combination of the internal ActionScript XML handling and the actual structure of the XML document - typically each database row will be represented by an XML node with many sub-nodes, one for each database field, so for several thousand database rows there could be tens of thousands of nodes in the XML document. Considering the Flash player assigns each node an ActionScript object, it can quickly be seen that any process which relies on processing tens of thousands of objects in an interpreted environment isn't going to be fast.
Dynamically generated SWF files: A very interesting technique where the client-side DataList SWF actually loads in server-side dynamically generated SWF files containing the database data. The actual SWF bytecode is generated on the fly (similar to Generator) by PHP or ASP.NET scripts. Unlike the XML method outlined above the client-side processing of this format of data is almost instant thanks to the fact that the data is already in native ActionScript objects. This method also benefits further from the ZLib compression which is part of the SWF specification. Also, being a SWF file, it (and the data inside it) can be split into frames - some interesting experiments were conducted where the data was split across several frames, thus making data streaming and the display of accurate progress bars a possibility (although progress tracking is available with other data loading techniques, this is the only method where processing the data while it is still loading is a possibility).
The only downside with this method is the actual server-side overhead involved in generating the SWF files. Although SWF files can be streamed from the server, this is only in the 99% of situations where the SWF has been pre-compiled. When built dynamically the whole SWF needs to be in memory before being served to the client, this is because the SWF file specification designates many bytes in the file structure as length specifiers (both lengths for the whole file and also for each ActionScript statement) and these can only be filled in once the structures they refer to have been built in memory. This means that when generating a SWF file the server is constantly having to go back in memory and change bytes in what can be a very large string representing the SWF file's contents. Considering this string can be up to a 1Mb the processing can be a strain for interprested languages such as PHP (although the compiled .NET languages were able to cope with this memory handling rather better).
Despite these server-side complexities, this method's performance was excellent (as although much reduced, the actual data download was still the bottleneck).
Compressed and Optimised XML: As outlined above, the ordinary XML method had poor performance. However it only took a few tweeks and a bit of restructuring to drastically increase download and processing speeds for XML. The first step was to turn on ZLib output compression from the server-side script engine - although XML is a bytewise inefficent format it is very compressible and so with output compression the data transfer sizes are much reduced. The second problem with XML is the client-side processing, as mentioned above several thousand rows can lead to an XML document containing tens of thousands of ActionScript objects which in turn is complex for the Flash player to churn its way through. If instead the data in the fields are moved from sub-nodes of the row node into attributes of the row node then the number of XML nodes drops dramitically and so does the client-side processing time. Add in some simple A-Z XML attribute names for the fields and the ActionScript processing loop can be reduced to just a couple of statements. The efficiency gains all this leads to are remarkable although the XML purists may be looking at the resultant XML schema (or rather the lack of it) in disgust.
Although this method is slighlty slower than the dynamic SWF method, this is the preferred data transfer method as the speed differences are not too significanct and the code is a lot simpler to work with.
So where's the DataList used?
The DataList has been deployed as part of numerous web-based administration systems. It can be deployed as an all-on-one editing solution or as a read-only list driving a frame-based HTML form system.