Hi,

I need to save some huge XML files in HD.
Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G.. 30G...

What the best way to save "line by line"?

Regards,
Marcos Douglas

Append chuncks of lines to a TFileStream until source XML file is exhausted

But I'm using TXMLDocument class to make the XML.

For each line (log) I do this:
(doesn't matter the information in Par variable)

procedure TXMLLogger.Log;
var
  I: Integer;
  No: TDOMElement;
  Par: TParam;
begin
  No := FDoc.CreateElement('line');
  for I := 0 to FParams.Count-1 do
  begin
    Par := FParams.Items[I];
    TDOMElement(No).SetAttribute(Par.Name, Par.AsString);
  end;
  FRootNode.AppendChild(No);
  FParams.Clear;
end;
At the end, I do some like:
procedure TXMLLogger.Finish;
var
  No: TDOMElement;
begin
  No := FDoc.CreateElement('finish');
  TDOMElement(No).SetAttribute('dh', FmtDateTime(Now));
  FRootNode.AppendChild(No);
end;
And the application call:
procedure TXMLLogger.SaveToFile(const AFileName: string);
begin
  WriteXMLFile(FDoc, AFileName);
end;

So, there is a way to use TXMLDocument but save line by line more faster?

Regards,
Marcos Douglas

No.
The writer already uses a fixed buffer. Your problem is the use of TXMLDocument.

Simply put: logging to XML (worse: using DOM) is a VERY bad idea.

Michael.

Ideas? Maybe do what Antonio Fortuny said before?

Simply put: logging to XML (worse: using DOM) is a VERY bad idea.

Well, this is a request of my client. Other process will read this XML.

Thanks,
Marcos Douglas

Under no circumstances should you use TXMLDocument for this.

Better write the XML manually then: 'fire and forget'.

Michael.

Yep, a text file: assign, reset, append, close. I use an own build logger facility (for historical reasons) but I wonder whether Lazarus (or FPC) does not have such a logger component.

So, where is recommended to use TXMLDocument class?

Better write the XML manually then: 'fire and forget'.

Ok, I'll do it.

Marcos Douglas

Ok, thank you.

Marcos Douglas

General XML processing. Small documents.

You should be aware that the DOM always has the whole XML document in memory, plus a substantial amount of overhead. This is not something FPC specific, but is inherent in the DOM model. (a W3 spec)

Regardles of XML or not: keeping logs in memory is simply a bad idea. You can buffer them for performance reasons, but after some time you must write to file.

The DOM model does not allow this.

Michael.

Keep the file open between writes.
Otherwise you'll notice a serious performance penalty. You can always call Flush() to make sure the content of a buffer was written to disc.

Thus, in case of a crash, the XML file will contain all log entries, but will not be well-formed XML.

Michael.

Yes, I see now.
I'll change my app.

Marcos Douglas

On Tue, Jun 18, 2013 at 5:01 PM, Michael Van Canneyt <

wrote: Regardles of XML or not: keeping logs in memory is simply a bad idea.

​Log it to a csv file, and at the end you can convert it to xml for your client, you can do that without loading it into memory, process it line by line.

Best Regards
Zaher Dirkey

But in that way I will have two process to run.
I'll write a XML using strings... but thank you.

Marcos Douglas

Marcos Douglas wrote

But in that way I will have two process to run. I'll write a XML using strings... but thank you.

one process, but in function like TXMLLogger.Finish you can convert it, ​using strings ​in memory, it will eat your memory.

Best Regards
Zaher Dirkey

But I continue to keep the XML in memory...
The easy and fast way as I can see is change the TXMLLogger class to write a simple file text but not using DOM.

Marcos Douglas

Yes.

Michael.

Marcos Douglas wrote

But I continue to keep the XML in memory... The easy and fast way as I can see is change the TXMLLogger class to write a simple file text but not using DOM.

Hmm, give us a small example of that xml.​

Best Regards
Zaher Dirkey

<?xml version="1.0" encoding="utf-8"?>
<root>
<connection login="uref" database="DB_REF"/>
<start dh="2013-06-18T13:54:57"/>
<line idDoc="912" status="ok"/>
<line idDoc="913" status="ok"/>
<line idDoc="914" status="ok"/>
<line...>
<finish dh="2013-06-18T13:54:58"/>
</root>

Marcos Douglas

Maybe we should implement a simple API for just writing an XML to a stream. Something like SAX only the other way round. E.g.

=== example code begin ===

xml.StartNode('outer');
xml.AddAttribute('foo', 42);
xml.StartNode('blubb');
xml.AddAttribute('bar', 'test');
xml.AddAttribute('bar2', 'xyz');
xml.SetValue('Hello');
xml.EndNode;
xml.StartNode('blubb');
xml.AddAttribute('bar2', 'alpha');
xml.AddAttribute('bar', 'beta');
xml.SetValue('World');
xml.EndNode;
xml.EndNode;

=== example code end ===

would the result in

=== xml begin ===

<outer foo="42">
<blubb bar="test" bar2="xyz">
Hello
</blubb>
<blubb bar2="alpha" bar="beta">
Hello
</blubb>
</outer>

=== xml end ===

Regards,
Sven

It exists for HTML already. So: yes, not a bad idea.

Michael.

Hmm... if we have something for HTML already it should be rather easy to adapt, right? Or we could at least pull the common parts out...

Regards,
Sven

​What about when opening first time the file, write head lines

<?xml version="1.0" encoding="utf-8"?>
<root>
<connection login="uref" database="DB_REF"/>
<start dh="2013-06-18T13:54:57"/>

then when every log you write (simple with format function) <line idDoc="xxx" status="xxx"/>

And when finish it write the footer

<finish dh="2013-06-18T13:54:58"/>
</root>

I mean writing directly to the file without put it in DOM or any kind of document in the memory.

​

Yes. That was Antonio Fortuny and Michael Van Canneyt said at the first time. ;-)

Marcos Douglas

Marcos Douglas wrote

Today I call WriteXMLFile(Doc, AFileName) but I have XML with 5G, 10G.. 30G... What the best way to save "line by line"?

Why not push it through a stream compressor ?

-Michael

You completely miss the point.

The stream has nothing to do with the problem.

The problem is that the source XML is completely in memory, which is why it takes so much memory after while.

Michael.

Michael Van Canneyt wrote

The problem is that the source XML is completely in memory, which is why it takes so much memory after while.

+1

And if "logging" is the use case, XML is by far the worst format anybody could choose.

Regards,
- Graeme -

Graeme Geldenhuys wrote

XML is by far the worst format anybody could choose

Yep. bus as it is "Standard" some need to ....

Thus IMHO the way to go is to have the basic XML component work on linked lists with binary data in memory and only when loading/saving use an ASCII representation.

(in fact, we did a "linked component list" handler / converter for HTML some years ago, same of course could be modified to use XML, as same is even more strictly defined than HTML. The unit is based in TComponentList and defines several components for the different tag types used).

-Michael

What do you think TXMLDocument and XMLWriter do ? Exactly this.

Michael.

Yes, but my client wants XML... another process will use it.

Marcos Douglas

Interesting.

Marcos Douglas

In cases like the it is best practice for the reporter change their implementation.

Stream Compression, and other things won't help. This is a limitation of strings and memory allocation w/r/t xml.

I would suggest that the reporter just implement function toXML() method(s) to construct XML data as a string and use T(File/Memory)Stream. This method has worked well for me.

In fact I did thinks this until you stated "the source XML is completely in memory" which misled me to the assumption that the XML text would be in memory (the binary list is not "XML", as it could be saves in different ASN.1 representations (or whatever) as well. )

-Michael

And that's why a TXMLDocument needs easily 10 times more memory than the actual xml file.

Mattias

Marcos Douglas wrote

Yes, but my client wants XML... another process will use it.

If the "another process" is also under your control, then maybe you can educate the client about not using XML. Yes, I know that is a tall order
- some client are idiots and just go with flashy words they heard elsewhere, and think it must be a requirement.

Your only [sane] alternative would then be to NOT use TXMLDocument, pray that your XML format isn't too complex, then use TFileStream (or something similar), and generate the XML lines one by one, using simple string concatenation or Format() statements etc.

Regards,
- Graeme -

yep... if it were myself, i would log in a regular more normal logging format... then i might create a tool to parse the log and output xml line by line as has been discussed... that way the logging is still as fast as it can be while those desiring xml can process the logs and get what they want... or they can create their own tool... but i guess $$$ talk, too ;)

The change was done.
I converted the sources to use simple files calling Assign(Handle, 'path') and Writeln('info'). Works and is faster.

Thank you all.

Marcos Douglas

No. Other software will read this file and is out of my control.

Your only [sane] alternative would then be to NOT use TXMLDocument, pray that your XML format isn't too complex, then use TFileStream (or something similar), and generate the XML lines one by one, using simple string concatenation or Format() statements etc.

Yes, I did like that.
The code is not pretty, but it works much faster.

Marcos Douglas

Options: Pretty code, Fast code, Working code

Choose any two options for final output. ;)

=)

Marcos Douglas