Despite documentation and manuals, I have not figured out how to use TCP sockets in Object Pascal, since there doesn't seem to be a primitive for it, or any easy way to instantiate sockets and manipulate incoming connections.

So my approach to sockets is to do it externally. A Bash shell script invokes another script, such as Ruby for binding the socket. Upon recieving a socket connection, it invokes an Object Pascal command line program, passing it socket traffic as standard input.

Another similar approach is to pre-parse serialized socket objects and then re-serialize them in a processed form, for the Object Pascal program to further process and manipulate from standard input. If I knew how to do more of this with pure Object Pascal, I'd do it that way.

But as it is, I'm thinking breaking up the program into components in a bunch of different languages. The way it's designed right now, a Bash Shell script is invoked by /etc/init.d to manage the program, including re-binding to the socket, rehashing, etc. A cron job periodically re-calls the program so that time-related functions can be invoked. this lets me get away with not using a core while-true loop. all components execute as a script and then exit until the next time the main Bash Shell script is invoked.

The /etc/init.d script is treated as an initializer, and is used to manage it, but really the main Bash Shell script in cron is what does the lion's share. A few of the persistent scripts are those that handle things like sockets and long-lived connections between components.

At the moment, all objects are stored on file in flat-file format, and serialized as an IRC-like text-line object notation, but I want to eventually switch to using MsgPack, ProtoBuf and SQLite for serialization and object storage, and libpurple for the many protocols libpurple supports.

The name of the application in question is named Tanis. Tanis is an automated personal assistant, or pseudo-AI. It does various things. At the moment, all it is is a protocol-agnostic bot that can retrieve weather, greet people with a random quote, and various IRC bot functions. It can also control chat management and such.

I'm working on an email management console, and from there a Usenet interface. It's going smoothly enough, I suppose. However, I really like Object Pascal, and the more I can do with it, and Common Lisp for the learning portions, the better.

I'd prefer for the sockets to be handled by Object Pascal, but I do not know how to do this. I'm not very good at using other peoples' code and libraries in general, so I tend to focus on either well-documented standard libraries that are easy to access and use, or using purely my own code.

I'm fairly proficient with Bash Shell, and I like Regular Expressions, and am fairly conversant with Extended Regular Expressions, so I tend to focus on using Bash Shell scripts as much as seems reasonable, while invoking other program components from within Bash while the Bash Shell script executes. There are three different Bash Shell scripts. One is an infinite loop, which can be started and stopped with /etc/init.d commands

Anyway, thank you for your time and patience.

Coyo Stormcaller wrote

Despite documentation and manuals, I have not figured out how to use TCP sockets in Object Pascal, since there doesn't seem to be a primitive for it, or any easy way to instantiate sockets and manipulate incoming connections.

There is source code for a simple serial and TCP/IP example here: http://ctrlterm.com/

Coyo Stormcaller wrote

Despite documentation and manuals, I have not figured out how to use TCP sockets in Object Pascal, since there doesn't seem to be a primitive for it, or any easy way to instantiate sockets and manipulate incoming connections.

There are at least 5 approaches that can do this for you.
- sockets unit - procedural, low-level approach to sockets.
- ssockets unit - object oriented layer for sockets.
- lnet - an async approach to sockets.
- synapse - a complete TCP/IP suite.
- Indy.

Regarding your overall approach:

If I may suggest, if you simply want to invoke a program with a socket as standard input, you can just use (x)inetd, it comes by default on all unix installations, and does just that: it is started by init, manages TCP/IP connections and spawns off a program to handle the connection with the socket handle as standard input/output.

Why reinvent the wheel ?

Michael.

If you have to do it that way then I strongly suggest using the existing netpipes (hose and faucet) utilities.

In our previous episode, Coyo Stormcaller said:

Despite documentation and manuals, I have not figured out how to use TCP sockets in Object Pascal, since there doesn't seem to be a primitive for it, or any easy way to instantiate sockets and manipulate incoming connections.

There are four socket suites and numerous code directly on top of unit sockets. That should be enough. Did you search the wiki?

Synapse sounds interesting. an entire TCP/IP stack written in Object Pascal. It sounds like fun to play with, and I've actually been looking around for a userspace TCP/IP stack for a VPN project of mine.

Thank you for the advice! You were very helpful! ^_^

I wonder if I could comment on a couple of misapprehensions. The first one is that the bulk of the TCP/IP stack is in the operating system, Synapse (or equivalent) isn't simply talking to a naked Ethernet device, instead it's using a group of system calls broadly referred to as "Berkeley Sockets".

While it is possible to write a complete stack from scratch in any reasonably-complete programming language, and while this has been done for e.g. embedded systems (or by idiots such as myself for demonstration/testing purposes :-), in practice it's avoided on account of (a) the amount of well-tested code that it would attempt to supplant and (b) consideration of the difficulty of implementing all required services in a single program.

Second, Pascal, in common with most modern languages, doesn't have primitives or constructs dedicated to things like sockets. Its primitives are generally restricted to features which are fairly general-purpose, with everything else being added as procedures/functions or (in modern implementations) classes... in either case loaded from libraries decoupled from the language itself.

It's interesting to compare this with some of the original ALGOL implementations or even with the original Turbo Pascal, which had a large number of keywords (e.g. to control file-open modes) which were defined as part of the language itself. Modern practice would be to make these into enumerations, which would be passed to a function as a parameter.

My interest in Userspace TCP/IP stacks is mostly due to being able to make dramatic modifications to the stack itself without modifying the kernelmode stack, and the ability to route without touching system calls to the socket system.

With a Userspace TCP/IP stack, I can send strange traffic with a raw UDP port. I'm developing a version of TCP/IP that uses 512-bit DHT hashes as addresses. When a packet is being routed to an unfamiliar destination IP, it uses Kademlia routing to reach the destination. The host itself has an internal router. All stacks in IPvCoyo have IP routing enabled. Only the first connection is routed in this inefficient way.

Along the way, a bidirectional label-switching tunnel is constructed. Subsequent packets are switched rapidly. This is efficient enough that no hardware acceleration is needed. This system is based on CJDNS, but rather than stopping at an IPv6 address, it goes further, altering TCP/IP itself to have full 512-bit DHT hashes as endpoint identifiers. Obviously, applications need to support the socket system.

That's where a userspace TCP/IP stack comes in. Applications can use pipes to send and receive traffic to another userspace application (the stack), or preferably, link to it as a dynamically-linked library and send and receive calls to that library.

Maybe, if people like the system enough, it can be ported to hardware-accelerated kernelmode code. But for now, modifying a userspace TCP/IP stack seems like the most reasonable approach. If you're wondering why I want to re-design TCP/IP it's because of a dare. I won't lose!

Interesting. I've seen at least one complete stack written in Pascal, although I can't remember what it was using as the low-level device: back in the DOS days when such things were common they'd usually have used Packet Driver. My partial implementations (embedded in test routines in a comms program) were on top of SLIP and PPP, my suggestion would be to start off with a simple conventional stack on top of SLIP and to test it against slirp running on Linux.

IP etc. are basically fairly simple protocols. The complexity comes from all the options that can be applied at the IP level, and the RFCs that are piled on top of it.

The real risk here, even for a research project, is the number of potential buffer overflows that you have to watch out for. It's tempting and comparatively simple to handle incoming messages using either pattern matching or overlaid records, but I think a more robust approach is to use state machines (Pascal doesn't, unfortunately, support coroutines). Things like unix kernels typically pass around messages as pointers, I'd avoid that and use bounds-checked arrays even if that meant that CPU time was wasted copying data around.

Coyo Stormcaller wrote

With a Userspace TCP/IP stack, I can send strange traffic with a raw UDP port.

In fact colleagues of mine once in Delphi implemented a TCP - like solution that uses UDP transfer to do secure, especially low overhead transfers via GPRS.

So I understand that such User-space IP stack extensions might make sense in certain cases.

But a User-Space "TCP/IP-Stack" implementation that sits on naked Ethernet does not seem to make any practical sense.

-Michael

It does if the objective is to experiment with variant protocols. Also remember that things like nmap already reimplement some parts of IP, so there's strong precedent.

I don't know what language implementation would be best for the job. The problem is that it's convenient to be able to overlay e.g. an IP packet with a predefined type e.g. a record describing the header followed by an array encompassing the data, however the actual length of the data is unknown at compilation time which means that hardware-supported bounds checking is less than useful.