Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2010-09

Any recommendations for a good in-memory storage type

10 messages · 6 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Graeme Geldenhuys

Hi,

While implementing drag-n-drop support in fpGUI, I created a TfpgMimeData class. This class will hold various data items, and a mime-type describing each of the data items. Obviously the TfpgMimeData class should be able to handle any data type. So my question is, what is a good universal type to store all kinds of data (strings, images (binary data), sound etc)?

The possible ideas I can think of is the following:

TByteArray
Variants
TMemoryStream

So which one would be more fitting for any mime types, or is there another option I didn't think of? Here is a simple usage example of the TfpgMimeData class - to help put this in perspective:

var
  m: TfpgMimeData;
  d: TfpgDrag;
  a: TfpgDropAction;
begin
      m := TfpgMimeData.Create;
      m.SetData('text/plain', 'My name is Earl');
      m.SetData('text/html', 'My name is <b>Earl</b>');  // text/plain
can actually be created from this automatically
      m.SetData('image/png', MyPNGImage);
      d := TfpgDrag.Create(self);
      d.MimeData := m;
      d.StartDrag([daCopy]);
      // d manages the lifespan of m
      // d will be freed automatically when drag action is complete or cancelled
end;
Regards,   - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net:8080/fpgui/
Marco van de Voort
to Graeme Geldenhuys: While implementing drag-n-drop support in fpGUI, I created a TfpgMimeData class. This…

In our previous episode, Graeme Geldenhuys said:

Show quoted text 12 lines
While implementing drag-n-drop support in fpGUI, I created a TfpgMimeData class. This class will hold various data items, and a mime-type describing each of the data items. Obviously the TfpgMimeData class should be able to handle any data type. So my question is, what is a good universal type to store all kinds of data (strings, images (binary data), sound etc)? The possible ideas I can think of is the following: TByteArray Variants TMemoryStream

Different choices. The question is if you want some typing (if only runtime) and easy conversions, or e.g. have the data in a directly writable format.

It depends on what you do most, operating on the data, or not.

e.g. You can easily assign a string to a variant, but for the rest you need to more or less stream it. Same for e.g. classes.

At the same time, you can very quickly dump bytearrays to disc (and tmemorystream is only a thin wrapper around one).

So what do you do more?

I would first try a variant.

(the example code doesn't perform any action on the data of the mimetype, so it doesn't really add anything IMHO)

Graeme Geldenhuys
to Marco van de Voort: In our previous episode, Graeme Geldenhuys said: Different choices. The question is if…
Marco van de Voort wrote
Different choices. The question is if you want some typing (if only runtime) and easy conversions, or e.g. have the data in a directly writable format.

Well, the data needs to be passed via Clipboard or Drag-n-Drop API's of the underlying system (when used from application to application), or can bypass those API's when dragging and dropping inside the same application (where a more efficient data format could be used).

Currently I'm only busy with Xlib (but GDI will follow soon). There are some conversion support, like I suggested. for example: a HTML data string can be converted to a Plain Text string, by simply striping all tags. Images, Sound etc will be passed as a byte array (at least under Xlib).

TfpgMimeData has the following convenience functions/properties to automatically search for the mime type and return the associated data, or if an exact mime type doesn't exist, try and do a conversion of a similar mime type (as in the case of HTML -> Plain Text). Or when setting data, use the property to determine what mime type must be associated with the data being stored.

eg: TfpgMimeData
property Urls (text with URI formatting
property Text (text)
property HTML (text with tags)

         function Data(mimetype)     (byte array)
I would first try a variant.

Just for getting the initial DND support to work under Xlib, I limited myself to text types only, so choose variant as my storage type, but now that I'm moving to the next phase (multiple data types), I am considering other storage types because I don't know if variant is suitable for binary data like sound, images etc.. I'm leaning towards TMemoryStream now. Just spotting the new incoming messages, Felipe and Michael seem to share my thoughts on that.

Regards,   - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net:8080/fpgui/
Marco van de Voort
to Graeme Geldenhuys

In our previous episode, Graeme Geldenhuys said:

Show quoted text 7 lines
> I would first try a variant. Just for getting the initial DND support to work under Xlib, I limited myself to text types only, so choose variant as my storage type, but now that I'm moving to the next phase (multiple data types), I am considering other storage types because I don't know if variant is suitable for binary data like sound, images etc..

No. But it allows you to keep text as text, not as streamed value. I don't know if assigning a string to a variant is copy-on-write though.

If you don't operate much on it, it doesn't matter. But the above sounds like you do most of the work on text, not binary blobs.

I'm leaning towards TMemoryStream now. Just spotting the new incoming messages, Felipe and Michael seem to share my thoughts on that.

They didn't ask what you wanted to do exactly either :-)

variants support most base Delphi types, afaik including class , interface and dyn array.

Michael Van Canneyt
to Marco van de Voort
Marco van de Voort wrote 23 lines
In our previous episode, Graeme Geldenhuys said: >> I would first try a variant. > > Just for getting the initial DND support to work under Xlib, I limited > myself to text types only, so choose variant as my storage type, but > now that I'm moving to the next phase (multiple data types), I am > considering other storage types because I don't know if variant is > suitable for binary data like sound, images etc.. No. But it allows you to keep text as text, not as streamed value. I don't know if assigning a string to a variant is copy-on-write though. If you don't operate much on it, it doesn't matter. But the above sounds like you do most of the work on text, not binary blobs. > I'm leaning towards > TMemoryStream now. Just spotting the new incoming messages, Felipe and > Michael seem to share my thoughts on that. They didn't ask what you wanted to do exactly either :-) variants support most base Delphi types, afaik including class , interface and dyn array.

For the kind of data he is most likely to receive, a stream still seems best.

For most data formats, maybe a TStringSTream is better suited. You'll have the text as data available at once, which is the most likely use anyway, and has all semantics of a stream.

Michael.

Graeme Geldenhuys
to Marco van de Voort: In our previous episode, Graeme Geldenhuys said: No. But it allows you to keep text as…
Marco van de Voort wrote
No. But it allows you to keep text as text, not as streamed value.

I'll see how it goes, maybe I could store text as text, and binary as binary - this will allow for the least amount of conversions. Plus the storage variable not being used will not be initialized - to reduce memory consumption. I'll experiment further.

If you don't operate much on it, it doesn't matter. But the above sounds like you do most of the work on text, not binary blobs.

Well, that will depend on the application the developer writes, not on what I do or use.

variants support most base Delphi types, afaik including class , interface and dyn array.

Supporting delphi data types is pointless when drag and drop occurs between two applications - hence the reason I am opting for the mime-type standards. This is why I can't believe the stupid drag-n-drop support implemented in Delphi and Kylix. There, the data enters the event handler as a TObject instance, so the VCL and CLX frameworks only support drag-n-drop inside the same application. For external DND, you must revert to external OS API's - using Win32 API under Windows, and no idea what under Kylix.

Regards,   - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net:8080/fpgui/
Matt Emson
to Graeme Geldenhuys: While implementing drag-n-drop support in fpGUI, I created a TfpgMimeData class. This…
Graeme Geldenhuys wrote
So which one would be more fitting for any mime types, or is there another option I didn't think of? Here is a simple usage example of the TfpgMimeData class - to help put this in perspective:

This seems a lot like the BMessage class used with in the BeAPI. There is an API called MUSCLE, which originates from a project created by Jeremey Freisner for BeOS, but is now very "cross platform". It uses a Message class to implement the format you mention below. It is designed to be used for comms, so it has the concept of "flattening" data to a binary format and then "unflattening" the data at the destination.

What does all this have to do with your class? I wrote an implementation for Delphi a good 5 years ago. It might be worth looking at? It was in the original MUSCLE distro, but if not, I can find you a copy.

https://public.msli.com/lcs/muscle/

Messages are not the "fastest" way of storing data, but the idea of an extensible generic data storage mechanism is extremely powerful, especially for distributed comms. MUSCLE implements a whole lot more and really is worth looking at if you want to create C/S over a WAN. The MUSCLE server becomes a middle man dispatcher on to which the clients publish resources and subscribe to events. The "server" becomes another node. Really love it, would love it more if it wasn't written in C++.

M

Show quoted text 22 lines
var m: TfpgMimeData; d: TfpgDrag; a: TfpgDropAction; begin m := TfpgMimeData.Create; m.SetData('text/plain', 'My name is Earl'); m.SetData('text/html', 'My name is <b>Earl</b>'); // text/plain can actually be created from this automatically m.SetData('image/png', MyPNGImage); d := TfpgDrag.Create(self); d.MimeData := m; d.StartDrag([daCopy]); // d manages the lifespan of m // d will be freed automatically when drag action is complete or cancelled end;
Zaher Dirkey
to Graeme Geldenhuys: While implementing drag-n-drop support in fpGUI, I created a TfpgMimeData class. This…
Graeme Geldenhuys wrote 41 lines
Hi, While implementing drag-n-drop support in fpGUI, I created a TfpgMimeData class. This class will hold various data items, and a mime-type describing each of the data items. Obviously the TfpgMimeData class should be able to handle any data type. So my question is, what is a good universal type to store all kinds of data (strings, images (binary data), sound etc)? The possible ideas I can think of is the following: TByteArray Variants TMemoryStream So which one would be more fitting for any mime types, or is there another option I didn't think of? Here is a simple usage example of the TfpgMimeData class - to help put this in perspective: var m: TfpgMimeData; d: TfpgDrag; a: TfpgDropAction; begin m := TfpgMimeData.Create; m.SetData('text/plain', 'My name is Earl'); m.SetData('text/html', 'My name is <b>Earl</b>'); // text/plain can actually be created from this automatically m.SetData('image/png', MyPNGImage); d := TfpgDrag.Create(self); d.MimeData := m; d.StartDrag([daCopy]); // d manages the lifespan of m // d will be freed automatically when drag action is complete or cancelled end;

Just initial idea.

TfpgMimeData class(TfpgDragData);
Picture:TPic ...
end;
var
 m: TfpgDragData;
 d: TfpgDrag;
 a: TfpgDropAction;
begin
     m := TfpgMimeData.Create;
     m.SetData('text/plain', 'My name is Earl');
     m.SetData('text/html', 'My name is <b>Earl</b>');  // text/plain
can actually be created from this automatically
   m.SetData('image/png');
     (m as TfpgMimeData).Picture.Assign(MyPNGImage);
     d := TfpgDrag.Create(self);
     d.DragData := m;
     d.StartDrag([daCopy]);
     // d manages the lifespan of m
     // d will be freed automatically when drag action is complete or
cancelled
end;
Zaher Dirkey