FTP low-level working.

SMTP connection oriented working.
ltn12 improved.

15 files changed, 690 insertions, 648 deletions

diff --git a/TODO b/TODO
index 1e30a78..ca5b445 100644
--- a/TODO
+++ b/TODO
@@ -18,13 +18,17 @@
 * Padronizar os retornos de funccao
 * Separar as classes em arquivos
 * Retorno de sendto em datagram sockets pode ser refused
+* select sets are now associative
+
+colocar pump.all, pump.step e pump.simplify.
+mudar ltn12.html e usar o exemplo source.cat que está muito melhor.
 
 break smtp.send into c = smtp.open, c:send() c:close()
 
 falar sobre encodet/wrapt/decodet no manual sobre mime
 
 
-RECEIVE MUDOU!!! COLOCAR NO MANUAL.
+RECEIVE MUDOU!!! (partial stuff) COLOCAR NO MANUAL.
 HTTP.lua mudou bastante também.
 
 fazer com que a socket.source e socket.sink sejam "selectable".
diff --git a/ltn012.wiki b/ltn012.wiki
new file mode 100644
index 0000000..57561c5
--- /dev/null
+++ b/ltn012.wiki
@@ -0,0 +1,391 @@
+===Filters, sources and sinks: design, motivation and examples===
+==or Functional programming for the rest of us==
+by Diego Nehab
+
+{{{
+
+}}}
+
+===Abstract===
+Certain operations can be implemented in the form of filters. A filter is a function that processes data received in consecutive function calls, returning partial results chunk by chunk.  Examples of operations that can be implemented as filters include the end-of-line normalization for text, Base64 and Quoted-Printable transfer content encodings, the breaking of text into lines, SMTP byte stuffing, and there are many others.  Filters become even more powerful when we allow them to be chained together to create composite filters. Filters can be seen as middle nodes in a chain of data transformations. Sources an sinks are the corresponding end points of these chains. A source is a function that produces data, chunk by chunk, and a sink is a function that takes data, chunk by chunk. In this technical note, we define an elegant interface for filters, sources, sinks and chaining. We evolve our interface progressively, until we reach a high degree of generality. We discuss difficulties that arise during the implementation of this interface and we provide solutions and examples. 
+
+===Introduction===
+
+Applications sometimes have too much information to process to fit in memory and are thus forced to process data in smaller parts.  Even when there is enough memory, processing all the data atomically may take long enough to frustrate a user that wants to interact with the application.  Furthermore, complex transformations can often be defined as series of simpler operations. Several different complex transformations might share the same simpler operations, so that an uniform interface to combine them is desirable. The following concepts constitute our solution to these problems. 
+
+''Filters'' are functions that accept successive chunks of input, and produce successive chunks of output. Furthermore, the result of concatenating all the output data is the same as the result of applying the filter over the concatenation of the input data. As a consequence, boundaries are irrelevant: filters have to handle input data split arbitrarily by the user. 
+
+A ''chain'' is a function that combines the effect of two (or more) other functions, but whose interface is indistinguishable from the interface of one of its components.  Thus, a chained filter can be used wherever an atomic filter can be used.  However, its effect on data is the combined effect of its component filters. Note that, as a consequence, chains can be chained themselves to create arbitrarily complex operations that can be used just like atomic operations.
+
+Filters can be seen as internal nodes in a network through which data flows, potentially being transformed along its way.  Chains connect these nodes together. To complete the picture, we need ''sources'' and ''sinks'' as initial and final nodes of the network, respectively.  Less abstractly, a source is a function that produces new data every time it is called.  On the other hand, sinks are functions that give a final destination to the data they receive.  Naturally, sources and sinks can be chained with filters.
+
+Finally, filters, chains, sources, and sinks are all passive entities: they need to be repeatedly called in order for something to happen.  ''Pumps'' provide the driving force that pushes data through the network, from a source to a sink.
+
+ Hopefully, these concepts will become clear with examples. In the following sections, we start with simplified interfaces, which we improve several times until we can find no obvious shortcommings. The evolution we present is not contrived: it follows the steps we followed ourselves as we consolidated our understanding of these concepts. 
+
+== A concrete example ==
+
+Some data transformations are easier to implement as filters than others.  Examples of operations that can be implemented as filters include the end-of-line normalization for text, the Base64 and Quoted-Printable transfer content encodings, the breaking of text into lines, SMTP byte stuffing, and many others. Let's use the end-of-line normalization as an example to define our initial filter interface. We later discuss why the implementation might not be trivial.
+
+Assume we are given text in an unknown end-of-line convention (including possibly mixed conventions) out of the commonly found Unix (LF), Mac OS (CR), and DOS (CRLF) conventions. We would like to be able to write code like the following:
+        {{{
+input = source.chain(source.file(io.stdin), normalize("\r\n"))
+output = sink.file(io.stdout)
+pump(input, output)
+}}}
+
+This program should read data from the standard input stream and normalize the end-of-line markers to the canonic CRLF marker defined by the MIME standard, finally sending the results to the standard output stream.  For that, we use a ''file source'' to produce data from standard input, and chain it with a filter that normalizes the data. The pump then repeatedly gets data from the source, and moves it to the ''file sink'' that sends it to standard output.
+
+To make the discussion even more concrete, we start by discussing the implementation of the normalization filter. The {{normalize}} ''factory'' is a function that creates such a filter. Our initial filter interface is as follows: the filter receives a chunk of input data, and returns a chunk of processed data. When there is no more input data, the user notifies the filter by invoking it with a {{nil}} chunk. The filter then returns the final chunk of processed data.
+
+Although the interface is extremely simple, the implementation doesn't seem so obvious. Any filter respecting this interface needs to keep some kind of context between calls. This is because chunks can be broken between the CR and LF characters marking the end of a line.  This need for context storage is what motivates the use of factories: each time the factory is called, it returns a filter with its own context so that we can have several independent filters being used at the same time.  For the normalization filter, we know that the obvious solution (i.e.  concatenating all the input into the context before producing any output) is not good enough, so we will have to find another way.
+
+We will break the implementation in two parts: a low-level filter, and a factory of high-level filters. The low-level filter will be implemented in C and will not carry any context between function calls. The high-level filter factory, implemented in Lua, will create and return a high-level filter that keeps whatever context the low-level filter needs, but isolates the user from its internal details. That way, we take advantage of C's efficiency to perform the dirty work, and take advantage of Lua's simplicity for the bookkeeping.
+
+==The Lua part of the implementation==
+
+Below is the implementation of the factory of high-level end-of-line normalization filters:
+        {{{
+function filter.cycle(low, ctx, extra)
+    return function(chunk)
+        local ret
+        ret, ctx = low(ctx, chunk, extra)
+        return ret
+    end
+end
+
+function normalize(marker)
+    return cycle(eol, 0, marker)
+end
+}}}
+
+The {{normalize}} factory simply calls a more generic factory, the {{cycle}} factory. This factory receives a low-level filter, an initial context and some extra value and returns the corresponding high-level filter. Each time the high level filer is called with a new chunk, it calls the low-level filter passing the previous context, the new chunk and the extra argument. The low-level filter produces the chunk of processed data and a new context. Finally, the high-level filter updates its internal context and returns the processed chunk of data to the user.  It is the low-level filter that does all the work.  Notice that this implementation takes advantage of the Lua 5.0 lexical scoping rules to store the context locally, between function calls.  
+
+Moving to the low-level filter, we notice there is no perfect solution to the end-of-line marker normalization problem itself. The difficulty comes from an inherent ambiguity on the definition of empty lines within mixed input. However, the following solution works well for any consistent input, as well as for non-empty lines in mixed input. It also does a reasonable job with empty lines and serves as a good example of how to implement a low-level filter.
+
+Here is what we do: CR and LF are considered candidates for line break.  We issue ''one'' end-of-line line marker if one of the candidates is seen alone, or followed by a ''different'' candidate.  That is, CR CR and LF LF issue two end of line markers each, but CR LF and LF CR issue only one marker.  This idea takes care of Mac OS, Mac OS X, VMS and Unix, DOS and MIME, as well as probably other more obscure conventions.
+
+==The C part of the implementation==
+
+The low-level filter is divided into two simple functions. The inner function actually does the conversion. It takes each input character in turn, deciding what to output and how to modify the context. The context tells if the last character seen was a candidate and, if so, which candidate it was.
+        {{{
+#define candidate(c) (c == CR || c == LF)
+static int process(int c, int last, const char *marker, luaL_Buffer *buffer) {
+    if (candidate(c)) {
+        if (candidate(last)) {
+            if (c == last) luaL_addstring(buffer, marker);
+            return 0;
+        } else {
+            luaL_addstring(buffer, marker);
+            return c;
+        }
+    } else {
+        luaL_putchar(buffer, c);
+        return 0;
+    }
+}
+}}}
+
+The inner function makes use of Lua's auxiliary library's buffer interface for its efficiency and ease of use. The outer function simply interfaces with Lua.  It receives the context and the input chunk (as well as an optional end-of-line marker), and returns the transformed output and the new context.
+        {{{
+static int eol(lua_State *L) {
+    int ctx = luaL_checkint(L, 1);
+    size_t isize = 0;
+    const char *input = luaL_optlstring(L, 2, NULL, &isize);
+    const char *last = input + isize;
+    const char *marker = luaL_optstring(L, 3, CRLF);
+    luaL_Buffer buffer;
+    luaL_buffinit(L, &buffer);
+    if (!input) {
+       lua_pushnil(L);
+       lua_pushnumber(L, 0);
+       return 2;
+    }
+    while (input < last)
+        ctx = process(*input++, ctx, marker, &buffer);
+    luaL_pushresult(&buffer);
+    lua_pushnumber(L, ctx);
+    return 2;
+}
+}}}
+
+Notice that if the input chunk is {{nil}}, the operation is considered to be finished. In that case, the loop will not execute a single time and the context is reset to the initial state.  This allows the filter to be reused indefinitely. It is a good idea to write filters like this, when possible. 
+
+Besides the end-of-line normalization filter shown above, many other filters can be implemented with the same ideas. Examples include Base64 and Quoted-Printable transfer content encodings, the breaking of text into lines, SMTP byte stuffing etc. The challenging part is to decide what will be the context. For line breaking, for instance, it could be the number of bytes left in the current line.  For Base64 encoding, it could be the bytes that remain in the division of the input into 3-byte atoms. 
+
+===Chaining===
+
+Filters become more powerful when the concept of chaining is introduced. Suppose you have a filter for Quoted-Printable encoding and you want to encode some text. According to the standard, the text has to be normalized into its canonic form prior to encoding.  A nice interface that simplifies this task is a factory that creates a composite filter that passes data through multiple filters, but that can be used wherever a primitive filter is used.
+        {{{
+local function chain2(f1, f2)
+    return function(chunk)
+        local ret = f2(f1(chunk))
+        if chunk then return ret
+        else return ret .. f2() end
+    end
+end
+
+function filter.chain(...)
+    local f = arg[1]
+    for i = 2, table.getn(arg) do
+        f = chain2(f, arg[i])
+    end
+    return f
+end
+
+local chain = filter.chain(normalize("\r\n"), encode("quoted-printable"))
+while 1 do
+    local chunk = io.read(2048)
+    io.write(chain(chunk))
+    if not chunk then break end
+end
+}}}
+
+The chaining factory is very simple. All it does is return a function that passes data through all filters and returns the result to the user.  It uses the simpler auxiliar function that knows how to chain two filters together. In the auxiliar function, special care must be taken if the chunk is final. This is because the final chunk notification has to be pushed through both filters in turn. Thanks to the chain factory, it is easy to perform the Quoted-Printable conversion, as the above example shows.
+
+===Sources, sinks, and pumps===
+
+As we noted in the introduction, the filters we introduced so far act as the internal nodes in a network of transformations. Information flows from node to node (or rather from one filter to the next) and is transformed on its way out. Chaining filters together is the way we found to connect nodes in the network. But what about the end nodes?  In the beginning of the network, we need a node that provides the data, a source. In the end of the network, we need a node that takes in the data, a sink.
+
+==Sources==
+
+We start with two simple sources. The first is the {{empty}} source: It simply returns no data, possibly returning an error message. The second is the {{file}} source, which produces the contents of a file in a chunk by chunk fashion, closing the file handle when done.
+        {{{
+function source.empty(err)
+    return function()
+        return nil, err
+    end
+end
+
+function source.file(handle, io_err)
+    if handle then 
+        return function()
+            local chunk = handle:read(2048)
+            if not chunk then handle:close() end
+            return chunk
+        end
+    else return source.empty(io_err or "unable to open file") end
+end
+}}}
+
+A source returns the next chunk of data each time it is called. When there is no more data, it just returns {{nil}}.  If there is an error, the source can inform the caller by returning {{nil}} followed by an error message. Adrian Sietsma noticed that, although not on purpose, the interface for sources is compatible with the idea of iterators in Lua 5.0. That is, a data source can be nicely used in conjunction with {{for}} loops.  Using our file source as an iterator, we can rewrite our first example:
+        {{{
+local process = normalize("\r\n")
+for chunk in source.file(io.stdin) do
+    io.write(process(chunk))
+end
+io.write(process(nil))
+}}}
+
+Notice that the last call to the filter obtains the last chunk of processed data. The loop terminates when the source returns {{nil}} and therefore we need that final call outside of the loop.
+
+==Mantaining state between calls==
+
+It is often the case that a source needs to change its behaviour after some event. One simple example would be a file source that wants to make sure it returns {{nil}} regardless of how many times it is called after the end of file, avoiding attempts to read past the end of the file.  Another example would be a source that returns the contents of several files, as if they were concatenated, moving from one file to the next until the end of the last file is reached.
+
+One way to implement this kind of source is to have the factory declare extra state variables that the source can use via lexical scoping. Our file source could set the file handle itself to {{nil}} when it detects the end-of-file.  Then, every time the source is called, it could check if the handle is still valid and act accordingly:
+        {{{
+function source.file(handle, io_err)
+    if handle then 
+        return function()
+            if not handle then return nil end
+            local chunk = handle:read(2048)
+            if not chunk then 
+                handle:close() 
+                handle = nil
+            end
+            return chunk
+        end
+    else return source.empty(io_err or "unable to open file") end
+end
+}}}
+
+Another way to implement this behavior involves a change in the source interface to makes it more flexible. Let's allow a source to return a second value, besides the next chunk of data.  If the returned chunk is {{nil}}, the extra return value tells us what happened. A second {{nil}} means that there is just no more data and the source is empty.  Any other value is considered to be an error message.  On the other hand, if the chunk was ''not'' {{nil}}, the second return value tells us whether the source wants to be replaced. If it is {{nil}}, we should proceed using the same source.  Otherwise it has to be another source, which we have to use from then on, to get the remaining data. 
+
+This extra freedom is good for someone writing a source function, but it is a pain for those that have to use it.  Fortunately, given one of these ''fancy'' sources, we can transform it into a simple source that never needs to be replaced, using the following factory.
+        {{{
+function source.simplify(src)
+    return function()
+        local chunk, err_or_new = src()
+        src = err_or_new or src
+        if not chunk then return nil, err_or_new
+        else return chunk end
+    end
+end
+}}}
+
+The simplification factory allows us to write fancy sources and use them as if they were simple. Therefore, our next functions will only produce simple sources, and functions that take sources will assume they are simple.
+
+Going back to our file source, the extended interface allows for a more elegant implementation. The new source just asks to be replaced by an empty source as soon as there is no more data. There is no repeated checking of the handle. To make things simpler to the user, the factory itself simplifies the the fancy file source before returning it to the user:
+        {{{
+function source.file(handle, io_err)
+    if handle then 
+        return source.simplify(function()
+            local chunk = handle:read(2048)
+            if not chunk then 
+                handle:close()
+                return "", source.empty() 
+            end
+            return chunk
+        end)
+    else return source.empty(io_err or "unable to open file") end
+end
+}}}
+
+We can make these ideas even more powerful if we use a new feature of Lua 5.0: coroutines.  Coroutines suffer from a great lack of advertisement, and I am going to play my part here.  Just like lexical scoping, coroutines taste odd at first, but once you get used with the concept, it can save your day. I have to admit that using coroutines to implement our file source would be overkill, so let's implement a concatenated source factory instead.
+        {{{
+function source.cat(...)
+    local co = coroutine.create(function()
+        local i = 1
+        while i <= table.getn(arg) do
+            local chunk, err = arg[i]()
+            if chunk then coroutine.yield(chunk)
+            elseif err then return nil, err 
+            else i = i + 1 end 
+        end
+    end)
+    return function()
+        return shift(coroutine.resume(co))
+    end
+end
+}}}
+
+The factory creates two functions. The first is an auxiliar that does all the work, in the form of a coroutine. It reads a chunk from one of the sources. If the chunk is {{nil}}, it moves to the next source, otherwise it just yields returning the chunk. When it is resumed, it continues from where it stopped and tries to read the next chunk.  The second function is the source itself, and just resumes the execution of the auxiliar coroutine, returning to the user whatever chunks it returns (skipping the first result that tells us if the coroutine terminated). Imagine writing the same function without coroutines and you will notice the simplicity of this implementation. We will use coroutines again when we make the filter interface more powerful. 
+
+==Chaining Sources==
+
+What does it mean to chain a source with a filter?  The most useful interpretation is that the combined source-filter is a new source that produces data and passes it through the filter before returning it. Here is a factory that does it:
+        {{{
+function source.chain(src, f)
+    return source.simplify(function()
+        local chunk, err = src()
+        if not chunk then return f(nil), source.empty(err)
+        else return f(chunk) end
+    end)
+end
+}}}
+
+Our motivating example in the introduction chains a source with a filter. The idea of chaining a source with a filter is useful when one thinks about functions that might get their input data from a source. By chaining a simple source with one or more filters, the same function can be provided with filtered data even though it is unaware of the filtering that is happening behind its back.
+
+==Sinks==
+
+Just as we defined an interface for an initial source of data, we can also define an interface for a final destination of data. We call any function respecting that interface a ''sink''. Below are two simple factories that return sinks. The table factory creates a sink that stores all obtained data into a table. The data can later be efficiently concatenated into a single string with the {{table.concat}} library function. As another example, we introduce the {{null}} sink: A sink that simply discards the data it receives.
+        {{{
+function sink.table(t)
+    t = t or {}
+    local f = function(chunk, err)
+        if chunk then table.insert(t, chunk) end
+        return 1
+    end
+    return f, t
+end
+
+local function null()
+    return 1
+end
+
+function sink.null()
+    return null
+end
+}}}
+
+Sinks receive consecutive chunks of data, until the end of data is notified with a {{nil}} chunk. An error is notified by an extra argument giving an error message after the {{nil}} chunk.  If a sink detects an error itself and wishes not to be called again, it should return {{nil}}, optionally followed by an error message. A return value that is not {{nil}} means the source will accept more data. Finally, just as sources can choose to be replaced, so can sinks, following the same interface. Once again, it is easy to implement a {{sink.simplify}} factory that transforms a fancy sink into a simple sink.
+
+As an example, let's create a source that reads from the standard input, then chain it with a filter that normalizes the end-of-line convention and let's use a sink to place all data into a table, printing the result in the end.
+        {{{
+local load = source.chain(source.file(io.stdin), normalize("\r\n"))
+local store, t = sink.table()
+while 1 do
+    local chunk = load()
+    store(chunk)
+    if not chunk then break end
+end
+print(table.concat(t))
+}}}
+
+Again, just as we created a factory that produces a chained source-filter from a source and a filter, it is easy to create a factory that produces a new sink given a sink and a filter.  The new sink passes all data it receives through the filter before handing it in to the original sink. Here is the implementation: 
+        {{{
+function sink.chain(f, snk)
+    return function(chunk, err)
+        local r, e = snk(f(chunk))
+        if not r then return nil, e end
+        if not chunk then return snk(nil, err) end
+        return 1
+    end
+end
+}}}
+
+==Pumps==
+
+There is a while loop that has been around for too long in our examples.  It's always there because everything that we designed so far is passive.  Sources, sinks, filters: None of them will do anything on their own. The operation of pumping all data a source can provide into a sink is so common that we will provide a couple helper functions to do that for us.
+        {{{
+function pump.step(src, snk)
+    local chunk, src_err = src()
+    local ret, snk_err = snk(chunk, src_err)
+    return chunk and ret and not src_err and not snk_err, src_err or snk_err
+end
+
+function pump.all(src, snk, step)
+    step = step or pump.step
+    while true do
+        local ret, err = step(src, snk)
+        if not ret then return not err, err end
+    end
+end
+}}}
+
+The {{pump.step}} function moves one chunk of data from the source to the sink. The {{pump.all}} function takes an optional {{step}} function and uses it to pump all the data from the source to the sink. We can now use everything we have to write a program that reads a binary file from disk and stores it in another file, after encoding it to the Base64 transfer content encoding:
+        {{{
+local load = source.chain(
+    source.file(io.open("input.bin", "rb")), 
+    encode("base64")
+)
+local store = sink.chain(
+    wrap(76),
+    sink.file(io.open("output.b64", "w")), 
+)
+pump.all(load, store)
+}}}
+
+The way we split the filters here is not intuitive, on purpose.  Alternatively, we could have chained the Base64 encode filter and the line-wrap filter together, and then chain the resulting filter with either the file source or the file sink. It doesn't really matter.
+
+===One last important change===
+
+Turns out we still have a problem. When David Burgess was writing his gzip filter, he noticed that the decompression filter can explode a small input chunk into a huge amount of data. Although we wished we could ignore this problem, we soon agreed we couldn't. The only solution is to allow filters to return partial results, and that is what we chose to do.  After invoking the filter to pass input data, the user now has to loop invoking the filter to find out if it has more output data to return.  Note that these extra calls can't pass more data to the filter. 
+
+More specifically, after passing a chunk of input data to a filter and collecting the first chunk of output data, the user invokes the filter repeatedly, passing the empty string, to get extra output chunks. When the filter itself returns an empty string, the user knows there is no more output data, and can proceed to pass the next input chunk. In the end, after the user passes a {{nil}} notifying the filter that there is no more input data, the filter might still have produced too much output data to return in a single chunk. The user has to loop again, this time passing {{nil}} each time, until the filter itself returns {{nil}} to notify the user it is finaly done.
+
+Most filters won't need this extra freedom. Fortunately, the new filter interface is easy to implement. In fact, the end-of-line translation filter we created in the introduction already conforms to it. On the other hand, the chaining function becomes much more complicated. If it wasn't for coroutines, I wouldn't be happy to implement it. Let me know if you can find a simpler implementation that does not use coroutines!
+        {{{
+local function chain2(f1, f2)
+    local co = coroutine.create(function(chunk)
+        while true do
+            local filtered1 = f1(chunk)
+            local filtered2 = f2(filtered1)
+            local done2 = filtered1 and ""
+            while true do
+                if filtered2 == "" or filtered2 == nil then break end
+                coroutine.yield(filtered2)
+                filtered2 = f2(done2)
+            end
+            if filtered1 == "" then chunk = coroutine.yield(filtered1)
+            elseif filtered1 == nil then return nil
+            else chunk = chunk and "" end
+        end
+    end)
+    return function(chunk)
+        local _, res = coroutine.resume(co, chunk)
+        return res
+    end
+end
+}}}
+
+Chaining sources also becomes more complicated, but a similar solution is possible with coroutines. Chaining sinks is just as simple as it has always been. Interstingly, these modifications do not have a measurable negative impact in the the performance of filters that didn't need the added flexibility. They do severely improve the efficiency of filters like the gzip filter, though, and that is why we are keeping them.
+
+===Final considerations===
+
+These ideas were created during the development of {{LuaSocket}}[http://www.tecgraf.puc-rio.br/luasocket] 2.0, and are available as the LTN12 module.  As a result, {{LuaSocket}}[http://www.tecgraf.puc-rio.br/luasocket] implementation was greatly simplified and became much more powerful.  The MIME module is especially integrated to LTN12 and provides many other filters. We felt these concepts deserved to be made public even to those that don't care about {{LuaSocket}}[http://www.tecgraf.puc-rio.br/luasocket], hence the LTN. 
+
+One extra application that deserves mentioning makes use of an identity filter.  Suppose you want to provide some feedback to the user while a file is being downloaded into a sink. Chaining the sink with an identity filter (a filter that simply returns the received data unaltered), you can update a progress counter on the fly. The original sink doesn't have to be modified.  Another interesting idea is that of a T sink: A sink that sends data to two other sinks. In summary, there appears to be enough room for many other interesting ideas.
+
+In this technical note we introduced filters, sources, sinks, and pumps.  These are useful tools for data processing in general. Sources provide a simple abstraction for data acquisition. Sinks provide an abstraction for final data destinations. Filters define an interface for data transformations.  The chaining of filters, sources and sinks provides an elegant way to create arbitrarily complex data transformation from simpler transformations. Pumps just put the machinery to work.  
diff --git a/src/auxiliar.c b/src/auxiliar.c
index fe21d08..b1f9203 100644
--- a/src/auxiliar.c
+++ b/src/auxiliar.c
@@ -14,27 +14,6 @@
 * Exported functions
 \*=========================================================================*/
 /*-------------------------------------------------------------------------*\
-* Prints the value of a class in a nice way
-\*-------------------------------------------------------------------------*/
-int aux_meth_tostring(lua_State *L)
-{
-    char buf[32];
-    if (!lua_getmetatable(L, 1)) goto error;
-    lua_pushstring(L, "__index");
-    lua_gettable(L, -2);
-    if (!lua_istable(L, -1)) goto error;
-    lua_pushstring(L, "class");
-    lua_gettable(L, -2);
-    if (!lua_isstring(L, -1)) goto error;
-    sprintf(buf, "%p", lua_touserdata(L, 1));
-    lua_pushfstring(L, "socket: %s: %s", lua_tostring(L, -1), buf);
-    return 1;
-error:
-    lua_pushnil(L);
-    return 1;
-}
-
-/*-------------------------------------------------------------------------*\
 * Initializes the module
 \*-------------------------------------------------------------------------*/
 int aux_open(lua_State *L)
@@ -48,23 +27,20 @@ int aux_open(lua_State *L)
 void aux_newclass(lua_State *L, const char *classname, luaL_reg *func)
 {
     luaL_newmetatable(L, classname); /* mt */
-    /* set __tostring metamethod */
-    lua_pushstring(L, "__tostring"); 
-    lua_pushcfunction(L, aux_meth_tostring);
-    lua_rawset(L, -3);      
     /* create __index table to place methods */
     lua_pushstring(L, "__index");    /* mt,"__index" */
     lua_newtable(L);                 /* mt,"__index",it */ 
-    luaL_openlib(L, NULL, func, 0);
     /* put class name into class metatable */
     lua_pushstring(L, "class");      /* mt,"__index",it,"class" */
     lua_pushstring(L, classname);    /* mt,"__index",it,"class",classname */
     lua_rawset(L, -3);               /* mt,"__index",it */
-    /* get __gc method from class and use it for garbage collection */
-    lua_pushstring(L, "__gc");       /* mt,"__index",it,"__gc" */
-    lua_pushstring(L, "__gc");       /* mt,"__index",it,"__gc","__gc" */
-    lua_rawget(L, -3);               /* mt,"__index",it,"__gc",fn */
-    lua_rawset(L, -5);               /* mt,"__index",it */
+    /* pass all methods that start with _ to the metatable, and all others
+     * to the index table */
+    for (; func->name; func++) {     /* mt,"__index",it */
+        lua_pushstring(L, func->name);
+        lua_pushcfunction(L, func->func);
+        lua_rawset(L, func->name[0] == '_' ? -5: -3);
+    }
     lua_rawset(L, -3);               /* mt */
     lua_pop(L, 1);
 }
diff --git a/src/ftp.lua b/src/ftp.lua
index 18dab6d..6074623 100644
--- a/src/ftp.lua
+++ b/src/ftp.lua
@@ -30,474 +30,153 @@ EMAIL = "anonymous@anonymous.org"
 BLOCKSIZE = 2048
 
 -----------------------------------------------------------------------------
--- Gets ip and port for data connection from PASV answer
--- Input
---   pasv: PASV command answer
--- Returns
---   ip: string containing ip for data connection
---   port: port for data connection
+-- Low level FTP API
 -----------------------------------------------------------------------------
-local function get_pasv(pasv)
-        local a, b, c, d, p1, p2, _
-        local ip, port
-        _,_, a, b, c, d, p1, p2 =
-                string.find(pasv, "(%d*),(%d*),(%d*),(%d*),(%d*),(%d*)")
-        if not (a and b and c and d and p1 and p2) then return nil, nil end
-        ip = string.format("%d.%d.%d.%d", a, b, c, d)
-        port = tonumber(p1)*256 + tonumber(p2)
-        return ip, port
-end
-
------------------------------------------------------------------------------
--- Check server greeting
--- Input
---   control: control connection with server
--- Returns
---   code: nil if error
---   answer: server answer or error message
------------------------------------------------------------------------------
-local function greet(control)
-        local code, answer = check_answer(control, {120, 220})
-        if code == 120 then -- please try again, somewhat busy now...
-                return check_answer(control, {220})
-        end
-        return code, answer
-end
-
------------------------------------------------------------------------------
--- Log in on server
--- Input
---   control: control connection with server
---   user: user name
---   password: user password if any
--- Returns
---   code: nil if error
---   answer: server answer or error message
------------------------------------------------------------------------------
-local function login(control, user, password)
-        local code, answer = command(control, "user", user, {230, 331})
-        if code == 331 and password then -- need pass and we have pass
-                return command(control, "pass", password, {230, 202})
-        end
-        return code, answer
-end
-
------------------------------------------------------------------------------
--- Change to target directory
--- Input
---   control: socket for control connection with server
---   path: directory to change to
--- Returns
---   code: nil if error
---   answer: server answer or error message
------------------------------------------------------------------------------
-local function cwd(control, path)
-end
-
------------------------------------------------------------------------------
--- Change to target directory
--- Input
---   control: socket for control connection with server
--- Returns
---   server: server socket bound to local address, nil if error
---   answer: error message if any
------------------------------------------------------------------------------
-local function port(control)
-        local code, answer
-        local server, ctl_ip
-        ctl_ip, answer = control:getsockname()
-        server, answer = socket.bind(ctl_ip, 0)
-        server:settimeout(TIMEOUT)
-        local ip, p, ph, pl
-        ip, p = server:getsockname()
-        pl = math.mod(p, 256)
-        ph = (p - pl)/256
-    local arg = string.gsub(string.format("%s,%d,%d", ip, ph, pl), "%.", ",")
-        code, answer = command(control, "port", arg, {200})
-        if not code then 
-                server:close()
-                return nil, answer
-        else return server end
-end
+local metat = { __index = {} }
 
------------------------------------------------------------------------------
--- Closes control connection with server
--- Input
---   control: control connection with server
--- Returns
---   code: nil if error
---   answer: server answer or error message
------------------------------------------------------------------------------
-local function logout(control)
-        local code, answer = command(control, "quit", nil, {221})
-        if code then control:close() end
-        return code, answer
+function open(server, port)
+    local tp = socket.try(socket.tp.connect(server, port or PORT))
+    return setmetatable({tp = tp}, metat)
 end
 
------------------------------------------------------------------------------
--- Receives data and send it to a callback
--- Input
---   data: data connection
---   callback: callback to return file contents
--- Returns
---   nil if successfull, or an error message in case of error
------------------------------------------------------------------------------
-local function receive_indirect(data, callback)
-        local chunk, err, res
-        while not err do
-                chunk, err = try_receive(data, BLOCKSIZE)
-                if err == "closed" then err = "done" end
-                res = callback(chunk, err)
-                if not res then break end
-        end
+local function port(portt)
+    return portt.server:accept()
 end
 
------------------------------------------------------------------------------
--- Retrieves file or directory listing
--- Input
---   control: control connection with server
---   server: server socket bound to local address
---   name: file name 
---   is_directory: is file a directory name?
---   content_cb: callback to receive file contents
--- Returns
---   err: error message in case of error, nil otherwise
------------------------------------------------------------------------------
-local function retrieve(control, server, name, is_directory, content_cb)
-        local code, answer
-        local data
-        -- ask server for file or directory listing accordingly
-        if is_directory then 
-                code, answer = cwd(control, name) 
-                if not code then return answer end
-                code, answer = command(control, "nlst", nil, {150, 125})
-        else 
-                code, answer = command(control, "retr", name, {150, 125}) 
-        end
-        if not code then return nil, answer end
-        data, answer = server:accept()
-        server:close()
-        if not data then 
-                control:close()
-                return answer 
-        end
-        answer = receive_indirect(data, content_cb)
-        if answer then 
-                control:close()
-                return answer
-        end
-        data:close()
-        -- make sure file transfered ok
-        return check_answer(control, {226, 250})
+local function pasv(pasvt)
+    return socket.connect(pasvt.ip, pasvt.port)
 end
 
------------------------------------------------------------------------------
--- Stores a file
--- Input
---   control: control connection with server
---   server: server socket bound to local address
---   file: file name under current directory
---   send_cb: callback to produce the file contents
--- Returns
---   code: return code, nil if error
---   answer: server answer or error message
------------------------------------------------------------------------------
-local function store(control, server, file, send_cb)
-        local data, err
-        local code, answer = command(control, "stor", file, {150, 125})
-        if not code then 
-                control:close()
-                return nil, answer 
-        end
-        -- start data connection
-        data, answer = server:accept()
-        server:close()
-        if not data then 
-                control:close()
-                return nil, answer 
-        end
-        -- send whole file 
-        err = send_indirect(data, send_cb, send_cb())
-        if err then 
-                control:close()
-                return nil, err
-        end
-        -- close connection to inform that file transmission is complete
-        data:close()
-        -- check if file was received correctly
-        return check_answer(control, {226, 250})
-end
-
------------------------------------------------------------------------------
--- Change transfer type
--- Input
---   control: control connection with server
---   params: "type=i" for binary or "type=a" for ascii
--- Returns
---   err: error message if any
------------------------------------------------------------------------------
-local function change_type(control, params)
-        local type, _
-        _, _, type = string.find(params or "", "type=(.)")
-        if type == "a" or type == "i" then 
-                local code, err = command(control, "type", type, {200})
-                if not code then return err end
-        end
+function metat.__index:login(user, password)
+    socket.try(self.tp:command("USER", user))
+    local code, reply = socket.try(self.tp:check{"2..", 331})
+    if code == 331 then
+        socket.try(password, reply)
+        socket.try(self.tp:command("PASS", password))
+        socket.try(self.tp:check("2.."))
+    end
+    return 1
+end
+
+function metat.__index:pasv()
+    socket.try(self.tp:command("PASV"))
+    local code, reply = socket.try(self.tp:check("2.."))
+    local _, _, a, b, c, d, p1, p2 = 
+        string.find(reply, "(%d+)%D(%d+)%D(%d+)%D(%d+)%D(%d+)%D(%d+)")
+    socket.try(a and b and c and d and p1 and p2, reply)
+    self.pasvt = { 
+        ip = string.format("%d.%d.%d.%d", a, b, c, d), 
+        port = p1*256 + p2
+    }
+    if self.portt then 
+        self.portt.server:close()
+        self.portt = nil
+    end
+    return self.pasvt.ip, self.pasvt.port 
 end
 
------------------------------------------------------------------------------
--- Starts a control connection, checks the greeting and log on
--- Input
---   parsed: parsed URL components
--- Returns
---   control: control connection with server, or nil if error
---   err: error message if any
------------------------------------------------------------------------------
-local function open(parsed)
-        local control, err = socket.tp.connect(parsed.host, parsed.port)
-        if not control then return nil, err end
+function metat.__index:port(ip, port)
+    self.pasvt = nil
+    local server
+    if not ip then 
+        ip, port = socket.try(self.tp:getcontrol():getsockname())
+        server = socket.try(socket.bind(ip, 0))
+        ip, port = socket.try(server:getsockname())
+        socket.try(server:settimeout(TIMEOUT))
+    end
+    local pl = math.mod(port, 256)
+    local ph = (port - pl)/256
+    local arg = string.gsub(string.format("%s,%d,%d", ip, ph, pl), "%.", ",")
+    socket.try(self.tp:command("port", arg))
+    socket.try(self.tp:check("2.."))
+    self.portt = server and {ip = ip, port = port, server = server}
+    return 1
+end
+
+function metat.__index:send(sendt)
+    local data
+    socket.try(self.pasvt or self.portt, "need port or pasv first")
+    if self.pasvt then data = socket.try(pasv(self.pasvt)) end
+    socket.try(self.tp:command(sendt.command, sendt.argument))
+    if self.portt then data = socket.try(port(self.portt)) end
+    local step = sendt.step or ltn12.pump.step
     local code, reply
-    -- greet
-    code, reply = control:check({120, 220})
-    if code == 120 then -- busy, try again
-        code, reply = control:check(220) 
+    local checkstep = function(src, snk)
+        local readyt = socket.select(readt, nil, 0)
+        if readyt[tp] then
+            code, reply = self.tp:check{"2..", "1.."}
+            if not code then 
+                data:close()
+                return nil, reply 
+            end
+        end
+        local ret, err = step(src, snk)
+        if err then data:close() end
+        return ret, err
     end
-    -- authenticate
-        code, reply = control:command("user", user)
-        code, reply = control:check({230, 331})
-        if code == 331 and password then -- need pass and we have pass
-                control:command("pass", password)
-        code, reply = control:check({230, 202})
+    local sink = socket.sink("close-when-empty", data)
+    socket.try(ltn12.pump.all(sendt.source, sink, checkstep))
+    if not code then code = socket.try(self.tp:check{"1..", "2.."}) end
+    if string.find(code, "1..") then socket.try(self.tp:check("2..")) end
+    return 1
+end
+
+function metat.__index:receive(recvt)
+    local data
+    socket.try(self.pasvt or self.portt, "need port or pasv first")
+    if self.pasvt then data = socket.try(pasv(self.pasvt)) end
+    socket.try(self.tp:command(recvt.command, recvt.argument))
+    if self.portt then data = socket.try(port(self.portt)) end
+    local source = socket.source("until-closed", data)
+    local step = recvt.step or ltn12.pump.step
+    local checkstep = function(src, snk)
+        local ret, err = step(src, snk)
+        if err then data:close() end
+        return ret, err
     end
-    -- change directory
-        local segment = parse_path(parsed)
-        for i, v in ipairs(segment) do
-            code, reply = control:command("cwd")
-        code, reply = control:check(250) 
-        end
-    -- change type
-        local type = string.sub(params or "", 7, 7) 
-        if type == "a" or type == "i" then 
-                code, reply = control:command("type", type)
-        code, reply = control:check(200)
-        end
+    socket.try(ltn12.pump.all(source, recvt.sink, checkstep))
+    local code = socket.try(self.tp:check{"1..", "2.."})
+    if string.find(code, "1..") then socket.try(self.tp:check("2..")) end
+    return 1
 end
 
-        return change_dir(control, segment) or
-                change_type(control, parsed.params) or
-                download(control, request, segment) or 
-                close(control)
+function metat.__index:cwd(dir)
+    socket.try(self.tp:command("CWD", dir))
+    socket.try(self.tp:check(250))
+    return 1
 end
 
+function metat.__index:type(type)
+    socket.try(self.tp:command("TYPE", type))
+    socket.try(self.tp:check(200))
+    return 1
 end
 
------------------------------------------------------------------------------
--- Stores a file in current directory
--- Input
---   control: control connection with server
---   request: a table with the fields:
---     content_cb: send callback to send file contents
---   segment: parsed URL path segments
--- Returns
---   err: error message if any
------------------------------------------------------------------------------
-local function upload(control, request, segment)
-        local code, name, content_cb
-        -- get remote file name
-        name = segment[table.getn(segment)]
-        if not name then 
-                control:close()
-                return "Invalid file path" 
-        end
-        content_cb = request.content_cb
-        -- setup passive connection
-        local server, answer = port(control)
-        if not server then return answer end
-        -- ask server to receive file
-        code, answer = store(control, server, name, content_cb)
-        if not code then return answer end
+function metat.__index:greet()
+    local code = socket.try(self.tp:check{"1..", "2.."})
+    if string.find(code, "1..") then socket.try(self.tp:check("2..")) end
+    return 1
 end
 
------------------------------------------------------------------------------
--- Download a file from current directory
--- Input
---   control: control connection with server
---   request: a table with the fields:
---     content_cb: receive callback to receive file contents
---   segment: parsed URL path segments
--- Returns
---   err: error message if any
------------------------------------------------------------------------------
-local function download(control, request, segment)
-        local code, name, is_directory, content_cb
-        is_directory = segment.is_directory
-        content_cb = request.content_cb
-        -- get remote file name
-        name = segment[table.getn(segment)]
-        if not name and not is_directory then 
-                control:close()
-                return "Invalid file path" 
-        end
-        -- setup passive connection
-        local server, answer = port(control)
-        if not server then return answer end
-        -- ask server to send file or directory listing
-        code, answer = retrieve(control, server, name, 
-                is_directory, content_cb)
-        if not code then return answer end
+function metat.__index:quit()
+    socket.try(self.tp:command("QUIT"))
+    socket.try(self.tp:check("2.."))
+    return 1
 end
 
------------------------------------------------------------------------------
--- Parses the FTP URL setting default values
--- Input
---   request: a table with the fields:
---     url: the target URL
---     type: "i" for "image" mode, "a" for "ascii" mode or "d" for directory
---     user: account user name
---     password: account password
--- Returns
---   parsed: a table with parsed components
------------------------------------------------------------------------------
-local function parse_url(request)
-        local parsed = socket.url.parse(request.url, {
-                user = "anonymous", 
-                port = 21, 
-                path = "/",
-                password = EMAIL,
-                scheme = "ftp"
-        })
-        -- explicit login information overrides that given by URL
-        parsed.user = request.user or parsed.user
-        parsed.password = request.password or parsed.password
-        -- explicit representation type overrides that given by URL
-        if request.type then parsed.params = "type=" .. request.type end
-        return parsed
+function metat.__index:close()
+    socket.try(self.tp:close())
+    return 1
 end
 
 -----------------------------------------------------------------------------
--- Parses the FTP URL path setting default values
--- Input
---   parsed: a table with the parsed URL components
--- Returns
---   dirs: a table with parsed directory components
+-- High level FTP API
 -----------------------------------------------------------------------------
-local function parse_path(parsed_url)
-        local segment = socket.url.parse_path(parsed_url.path)
-        segment.is_directory = segment.is_directory or 
-        (parsed_url.params == "type=d")
-        return segment
-end
 
------------------------------------------------------------------------------
--- Builds a request table from a URL or request table
--- Input
---   url_or_request: target url or request table (a table with the fields:
---     url: the target URL
---     type: "i" for "image" mode, "a" for "ascii" mode or "d" for directory
---     user: account user name
---     password: account password)
--- Returns
---   request: request table
------------------------------------------------------------------------------
-local function build_request(data)
-    local request = {}
-    if type(data) == "table" then for i, v in data do request[i] = v end
-    else request.url = data end
-    return request
+function put(putt)
 end
 
------------------------------------------------------------------------------
--- Downloads a file from a FTP server
--- Input
---   request: a table with the fields:
---     url: the target URL
---     type: "i" for "image" mode, "a" for "ascii" mode or "d" for directory
---     user: account user name
---     password: account password
---     content_cb: receive callback to receive file contents
--- Returns
---   err: error message if any
------------------------------------------------------------------------------
-function get_cb(request)
-        local parsed = parse_url(request)
-        if parsed.scheme ~= "ftp" then 
-                return string.format("unknown scheme '%s'", parsed.scheme)
-        end
-        local control, err = open(parsed)
-        if not control then return err end
-        local segment = parse_path(parsed)
-        return change_dir(control, segment) or
-                change_type(control, parsed.params) or
-                download(control, request, segment) or 
-                close(control)
-end
-
------------------------------------------------------------------------------
--- Uploads a file to a FTP server
--- Input
---   request: a table with the fields:
---     url: the target URL
---     type: "i" for "image" mode, "a" for "ascii" mode or "d" for directory
---     user: account user name
---     password: account password
---     content_cb: send callback to send file contents
--- Returns
---   err: error message if any
------------------------------------------------------------------------------
-function put_cb(request)
-        local parsed = parse_url(request)
-        if parsed.scheme ~= "ftp" then 
-                return string.format("unknown scheme '%s'", parsed.scheme)
-        end
-        local control, err = open(parsed)
-        if not control then return err end
-        local segment = parse_path(parsed)
-        err = change_dir(control, segment) or
-                change_type(control, parsed.params) or
-                upload(control, request, segment) or 
-                close(control)
-        if err then return nil, err
-        else return 1 end
-end
-
------------------------------------------------------------------------------
--- Uploads a file to a FTP server
--- Input
---   url_or_request: target url or request table (a table with the fields:
---     url: the target URL
---     type: "i" for "image" mode, "a" for "ascii" mode or "d" for directory
---     user: account user name
---     password: account password)
---     content: file contents
---   content: file contents
--- Returns
---   err: error message if any
------------------------------------------------------------------------------
-function put(url_or_request, content)
-        local request = build_request(url_or_request)
-        request.content = request.content or content
-        request.content_cb = socket.callback.send_string(request.content)
-        return put_cb(request)
-end
-
------------------------------------------------------------------------------
--- Retrieve a file from a ftp server
--- Input
---   url_or_request: target url or request table (a table with the fields:
---     url: the target URL
---     type: "i" for "image" mode, "a" for "ascii" mode or "d" for directory
---     user: account user name
---     password: account password)
--- Returns
---   data: file contents as a string
---   err: error message in case of error, nil otherwise
------------------------------------------------------------------------------
-function get(url_or_request)
-        local concat = socket.concat.create()
-        local request = build_request(url_or_request)
-        request.content_cb = socket.callback.receive_concat(concat)
-        local err = get_cb(request)
-        return concat:getresult(), err
+function get(gett)
 end
 
-return socket.ftp
+return ftp
diff --git a/src/http.lua b/src/http.lua
index da18aaf..f787b9d 100644
--- a/src/http.lua
+++ b/src/http.lua
@@ -68,7 +68,7 @@ end
 
 local function receive_body(reqt, respt, tmp)
     local sink = reqt.sink or ltn12.sink.null()
-    local pump = reqt.pump or ltn12.pump
+    local step = reqt.step or ltn12.pump.step
     local source
     local te = respt.headers["transfer-encoding"]
     if te and te ~= "identity" then 
@@ -80,9 +80,9 @@ local function receive_body(reqt, respt, tmp)
         source = socket.source("by-length", tmp.sock, length)
     else 
         -- get it all until connection closes
-        source = socket.source("until-closed", tmp.sock)
+        source = socket.source(tmp.sock)
     end
-    socket.try(pump(source, sink))
+    socket.try(ltn12.pump.all(source, sink, step))
 end
 
 local function send_headers(sock, headers)
@@ -125,7 +125,7 @@ end
 local function send_request(reqt, respt, tmp)
     local uri = request_uri(reqt, respt, tmp)
     local headers = tmp.headers
-    local pump = reqt.pump or ltn12.pump
+    local step = reqt.step or ltn12.pump.step
     -- send request line
     socket.try(tmp.sock:send((reqt.method or "GET") 
         .. " " .. uri .. " HTTP/1.1\r\n"))
@@ -136,9 +136,11 @@ local function send_request(reqt, respt, tmp)
     -- send request message body, if any
     if not reqt.source then return end
     if headers["content-length"] then 
-        socket.try(pump(reqt.source, socket.sink(tmp.sock)))
+        socket.try(ltn12.pump.all(reqt.source, 
+            socket.sink(tmp.sock), step))
     else 
-        socket.try(pump(reqt.source, socket.sink("http-chunked", tmp.sock)))
+        socket.try(ltn12.pump.all(reqt.source, 
+            socket.sink("http-chunked", tmp.sock), step))
     end
 end
 
diff --git a/src/ltn12.lua b/src/ltn12.lua
index dac932b..56e6043 100644
--- a/src/ltn12.lua
+++ b/src/ltn12.lua
@@ -8,6 +8,7 @@ setfenv(1, ltn12)
 filter = {}
 source = {}
 sink = {}
+pump = {}
 
 -- 2048 seems to be better in windows...
 BLOCKSIZE = 2048
@@ -22,7 +23,6 @@ end
 
 -- returns a high level filter that cycles a cycles a low-level filter
 function filter.cycle(low, ctx, extra)
-    if type(low) ~= 'function' then error('invalid low-level filter', 2) end 
     return function(chunk)
         local ret
         ret, ctx = low(ctx, chunk, extra)
@@ -32,8 +32,6 @@ end
 
 -- chains two filters together
 local function chain2(f1, f2)
-    if type(f1) ~= 'function' then error('invalid filter', 2) end 
-    if type(f2) ~= 'function' then error('invalid filter', 2) end 
     local co = coroutine.create(function(chunk)
         while true do
             local filtered1 = f1(chunk)
@@ -58,7 +56,6 @@ end
 function filter.chain(...)
     local f = arg[1]
     for i = 2, table.getn(arg) do
-        if type(arg[i]) ~= 'function' then error('invalid filter', 2) end 
         f = chain2(f, arg[i])
     end
     return f
@@ -93,7 +90,6 @@ end
 
 -- turns a fancy source into a simple source
 function source.simplify(src)
-    if type(src) ~= 'function' then error('invalid source', 2) end 
     return function()
         local chunk, err_or_new = src()
         src = err_or_new or src
@@ -117,7 +113,6 @@ end
 
 -- creates rewindable source
 function source.rewind(src)
-    if type(src) ~= 'function' then error('invalid source', 2) end 
     local t = {}
     return function(chunk)
         if not chunk then
@@ -132,8 +127,6 @@ end
 
 -- chains a source with a filter
 function source.chain(src, f)
-    if type(src) ~= 'function' then error('invalid source', 2) end
-    if type(f) ~= 'function' then error('invalid filter', 2) end
     local co = coroutine.create(function()
         while true do 
             local chunk, err = src()
@@ -152,20 +145,21 @@ function source.chain(src, f)
     end
 end
 
--- creates a source that produces contents of several files one after the
+-- creates a source that produces contents of several sources, one after the
 -- other, as if they were concatenated
 function source.cat(...)
     local co = coroutine.create(function()
         local i = 1
-        while i <= table.getn(arg) do 
-            local chunk = arg[i]:read(2048)
+        while i <= table.getn(arg) do
+            local chunk, err = arg[i]()
             if chunk then coroutine.yield(chunk)
-            else i = i + 1 end
+            elseif err then return nil, err 
+            else i = i + 1 end 
         end
     end)
-    return source.simplify(function()
+    return function()
         return shift(coroutine.resume(co))
-    end)
+    end
 end
 
 -- creates a sink that stores into a table
@@ -180,7 +174,6 @@ end
 
 -- turns a fancy sink into a simple sink
 function sink.simplify(snk)
-    if type(snk) ~= 'function' then error('invalid sink', 2) end
     return function(chunk, err)
         local ret, err_or_new = snk(chunk, err)
         if not ret then return nil, err_or_new end
@@ -219,8 +212,6 @@ end
 
 -- chains a sink with a filter 
 function sink.chain(f, snk)
-    if type(snk) ~= 'function' then error('invalid sink', 2) end
-    if type(f) ~= 'function' then error('invalid filter', 2) end
     return function(chunk, err)
         local filtered = f(chunk)
         local done = chunk and ""
@@ -233,15 +224,18 @@ function sink.chain(f, snk)
     end
 end
 
--- pumps all data from a source to a sink
-function pump(src, snk)
-    if type(src) ~= 'function' then error('invalid source', 2) end
-    if type(snk) ~= 'function' then error('invalid sink', 2) end
+-- pumps one chunk from the source to the sink
+function pump.step(src, snk)
+    local chunk, src_err = src()
+    local ret, snk_err = snk(chunk, src_err)
+    return chunk and ret and not src_err and not snk_err, src_err or snk_err
+end
+
+-- pumps all data from a source to a sink, using a step function
+function pump.all(src, snk, step)
+    step = step or pump.step
     while true do
-        local chunk, src_err = src()
-        local ret, snk_err = snk(chunk, src_err)
-        if not chunk or not ret then
-            return not src_err and not snk_err, src_err or snk_err
-        end
+        local ret, err = step(src, snk)
+        if not ret then return not err, err end
     end
 end
diff --git a/src/luasocket.c b/src/luasocket.c
index eadb758..fe4c96c 100644
--- a/src/luasocket.c
+++ b/src/luasocket.c
@@ -25,6 +25,7 @@
 \*=========================================================================*/
 #include "luasocket.h"
 
+#include "base.h"
 #include "auxiliar.h"
 #include "timeout.h"
 #include "buffer.h"
@@ -39,34 +40,8 @@
 /*=========================================================================*\
 * Declarations
 \*=========================================================================*/
-static int base_open(lua_State *L);
 static int mod_open(lua_State *L, const luaL_reg *mod);
 
-/*-------------------------------------------------------------------------*\
-* Setup basic stuff.
-\*-------------------------------------------------------------------------*/
-static int base_open(lua_State *L)
-{
-    /* create namespace table */
-    lua_pushstring(L, LUASOCKET_LIBNAME);
-    lua_newtable(L);
-#ifdef LUASOCKET_DEBUG
-    lua_pushstring(L, "debug");
-    lua_pushnumber(L, 1);
-    lua_rawset(L, -3);
-#endif
-    /* make version string available so scripts */
-    lua_pushstring(L, "version");
-    lua_pushstring(L, LUASOCKET_VERSION);
-    lua_rawset(L, -3);
-    /* store namespace as global */
-    lua_settable(L, LUA_GLOBALSINDEX);
-    /* make sure modules know what is our namespace */
-    lua_pushstring(L, "LUASOCKET_LIBNAME");
-    lua_pushstring(L, LUASOCKET_LIBNAME);
-    lua_settable(L, LUA_GLOBALSINDEX);
-    return 0;
-}
 
 static int mod_open(lua_State *L, const luaL_reg *mod)
 {
@@ -79,6 +54,7 @@ static int mod_open(lua_State *L, const luaL_reg *mod)
 #include "tp.lch"
 #include "smtp.lch"
 #include "http.lch"
+#include "ftp.lch"
 #else
     lua_dofile(L, "ltn12.lua");
     lua_dofile(L, "auxiliar.lua");
@@ -87,6 +63,7 @@ static int mod_open(lua_State *L, const luaL_reg *mod)
     lua_dofile(L, "tp.lua");
     lua_dofile(L, "smtp.lua");
     lua_dofile(L, "http.lua");
+    lua_dofile(L, "ftp.lua");
 #endif
     return 0;
 }
diff --git a/src/mime.c b/src/mime.c
index 7bfa6aa..fe24f9b 100644
--- a/src/mime.c
+++ b/src/mime.c
@@ -14,14 +14,9 @@
 /*=========================================================================*\
 * Don't want to trust escape character constants
 \*=========================================================================*/
-#define CR 0x0D
-#define LF 0x0A
-#define HT 0x09
-#define SP 0x20
-
 typedef unsigned char UC;
-static const char CRLF[] = {CR, LF, 0};
-static const char EQCRLF[] = {'=', CR, LF, 0};
+static const char CRLF[] = "\r\n";
+static const char EQCRLF[] = "=\r\n";
 
 /*=========================================================================*\
 * Internal function prototypes.
@@ -121,9 +116,9 @@ static int mime_global_wrp(lua_State *L)
     luaL_buffinit(L, &buffer);
     while (input < last) {
         switch (*input) {
-            case CR:
+            case '\r':
                 break;
-            case LF:
+            case '\n':
                 luaL_addstring(&buffer, CRLF);
                 left = length;
                 break;
@@ -327,11 +322,10 @@ static int mime_global_unb64(lua_State *L)
 * all (except CRLF in text) can be =XX
 * CLRL in not text must be =XX=XX
 * 33 through 60 inclusive can be plain
-* 62 through 120 inclusive can be plain
+* 62 through 126 inclusive can be plain
 * 9 and 32 can be plain, unless in the end of a line, where must be =XX
 * encoded lines must be no longer than 76 not counting CRLF
 * soft line-break are =CRLF
-* !"#$@[\]^`{|}~ should be =XX for EBCDIC compatibility
 * To encode one byte, we need to see the next two. 
 * Worst case is when we see a space, and wonder if a CRLF is comming
 \*-------------------------------------------------------------------------*/
@@ -344,16 +338,10 @@ static void qpsetup(UC *qpclass, UC *qpunbase)
     int i;
     for (i = 0; i < 256; i++) qpclass[i] = QP_QUOTED;
     for (i = 33; i <= 60; i++) qpclass[i] = QP_PLAIN;
-    for (i = 62; i <= 120; i++) qpclass[i] = QP_PLAIN;
-    qpclass[HT] = QP_IF_LAST; qpclass[SP] = QP_IF_LAST;
-    qpclass['!'] = QP_QUOTED; qpclass['"'] = QP_QUOTED; 
-    qpclass['#'] = QP_QUOTED; qpclass['$'] = QP_QUOTED; 
-    qpclass['@'] = QP_QUOTED; qpclass['['] = QP_QUOTED;
-    qpclass['\\'] = QP_QUOTED; qpclass[']'] = QP_QUOTED; 
-    qpclass['^'] = QP_QUOTED; qpclass['`'] = QP_QUOTED; 
-    qpclass['{'] = QP_QUOTED; qpclass['|'] = QP_QUOTED;
-    qpclass['}'] = QP_QUOTED; qpclass['~'] = QP_QUOTED; 
-    qpclass['}'] = QP_QUOTED; qpclass[CR] = QP_CR;
+    for (i = 62; i <= 126; i++) qpclass[i] = QP_PLAIN;
+    qpclass['\t'] = QP_IF_LAST; 
+    qpclass[' '] = QP_IF_LAST;
+    qpclass['\r'] = QP_CR;
     for (i = 0; i < 256; i++) qpunbase[i] = 255;
     qpunbase['0'] = 0; qpunbase['1'] = 1; qpunbase['2'] = 2;
     qpunbase['3'] = 3; qpunbase['4'] = 4; qpunbase['5'] = 5;
@@ -377,7 +365,7 @@ static void qpquote(UC c, luaL_Buffer *buffer)
 
 /*-------------------------------------------------------------------------*\
 * Accumulate characters until we are sure about how to deal with them.
-* Once we are sure, output the to the buffer, in the correct form. 
+* Once we are sure, output to the buffer, in the correct form. 
 \*-------------------------------------------------------------------------*/
 static size_t qpencode(UC c, UC *input, size_t size, 
         const char *marker, luaL_Buffer *buffer)
@@ -389,7 +377,7 @@ static size_t qpencode(UC c, UC *input, size_t size,
             /* might be the CR of a CRLF sequence */
             case QP_CR:
                 if (size < 2) return size;
-                if (input[1] == LF) {
+                if (input[1] == '\n') {
                     luaL_addstring(buffer, marker);
                     return 0;
                 } else qpquote(input[0], buffer);
@@ -398,7 +386,7 @@ static size_t qpencode(UC c, UC *input, size_t size,
             case QP_IF_LAST:
                 if (size < 3) return size;
                 /* if it is the last, quote it and we are done */
-                if (input[1] == CR && input[2] == LF) {
+                if (input[1] == '\r' && input[2] == '\n') {
                     qpquote(input[0], buffer);
                     luaL_addstring(buffer, marker);
                     return 0;
@@ -492,19 +480,19 @@ static size_t qpdecode(UC c, UC *input, size_t size,
             case '=': 
                 if (size < 3) return size; 
                 /* eliminate soft line break */
-                if (input[1] == CR && input[2] == LF) return 0;
+                if (input[1] == '\r' && input[2] == '\n') return 0;
                 /* decode quoted representation */
                 c = qpunbase[input[1]]; d = qpunbase[input[2]];
                 /* if it is an invalid, do not decode */
                 if (c > 15 || d > 15) luaL_addlstring(buffer, (char *)input, 3);
                 else luaL_putchar(buffer, (c << 4) + d);
                 return 0;
-            case CR:
+            case '\r':
                 if (size < 2) return size; 
-                if (input[1] == LF) luaL_addlstring(buffer, (char *)input, 2);
+                if (input[1] == '\n') luaL_addlstring(buffer, (char *)input, 2);
                 return 0;
             default:
-                if (input[0] == HT || (input[0] > 31 && input[0] < 127))
+                if (input[0] == '\t' || (input[0] > 31 && input[0] < 127))
                     luaL_putchar(buffer, input[0]);
                 return 0;
         }
@@ -582,9 +570,9 @@ static int mime_global_qpwrp(lua_State *L)
     luaL_buffinit(L, &buffer);
     while (input < last) {
         switch (*input) {
-            case CR:
+            case '\r':
                 break;
-            case LF:
+            case '\n':
                 left = length;
                 luaL_addstring(&buffer, CRLF);
                 break;
@@ -623,7 +611,7 @@ static int mime_global_qpwrp(lua_State *L)
 * c is the current character being processed
 * last is the previous character
 \*-------------------------------------------------------------------------*/
-#define eolcandidate(c) (c == CR || c == LF)
+#define eolcandidate(c) (c == '\r' || c == '\n')
 static int eolprocess(int c, int last, const char *marker, 
         luaL_Buffer *buffer)
 {
diff --git a/src/select.c b/src/select.c
index 8590b96..41bdaa4 100644
--- a/src/select.c
+++ b/src/select.c
@@ -21,7 +21,6 @@ static int meth_set(lua_State *L);
 static int meth_isset(lua_State *L);
 static int c_select(lua_State *L);
 static int global_select(lua_State *L);
-static void check_obj_tab(lua_State *L, int tabidx);
 
 /* fd_set object methods */
 static luaL_reg set[] = {
@@ -68,9 +67,6 @@ static int global_select(lua_State *L)
     fd_set *read_fd_set, *write_fd_set;
     /* make sure we have enough arguments (nil is the default) */
     lua_settop(L, 3);
-    /* check object tables */
-    check_obj_tab(L, 1);
-    check_obj_tab(L, 2); 
     /* check timeout */
     if (!lua_isnil(L, 3) && !lua_isnumber(L, 3))
         luaL_argerror(L, 3, "number or nil expected");
@@ -127,24 +123,3 @@ static int c_select(lua_State *L)
                 timeout < 0 ? NULL : &tv));
     return 1;
 }
-
-static void check_obj_tab(lua_State *L, int tabidx)
-{
-    if (tabidx < 0) tabidx = lua_gettop(L) + tabidx + 1;
-    if (lua_istable(L, tabidx)) {
-        lua_pushnil(L);
-        while (lua_next(L, tabidx) != 0) {
-            if (aux_getgroupudata(L, "select{able}", -1) == NULL) {
-                char msg[45];
-                if (lua_isnumber(L, -2))
-                    sprintf(msg, "table entry #%g is invalid", 
-                            lua_tonumber(L, -2));
-                else
-                    sprintf(msg, "invalid entry found in table");
-                luaL_argerror(L, tabidx, msg);
-            }
-            lua_pop(L, 1);
-        }
-    } else if (!lua_isnil(L, tabidx))
-        luaL_argerror(L, tabidx, "table or nil expected");
-}
diff --git a/src/smtp.lua b/src/smtp.lua
index ed8bd15..6ddeaae 100644
--- a/src/smtp.lua
+++ b/src/smtp.lua
@@ -20,6 +20,7 @@ DOMAIN = os.getenv("SERVER_NAME") or "localhost"
 -- default time zone (means we don't know)
 ZONE = "-0000"
 
+
 local function shift(a, b, c)
     return b, c
 end
@@ -29,31 +30,66 @@ function stuff()
     return ltn12.filter.cycle(dot, 2)
 end
 
+---------------------------------------------------------------------------
+-- Low level SMTP API
+-----------------------------------------------------------------------------
+local metat = { __index = {} }
+
+function metat.__index:greet(domain)
+    socket.try(self.tp:check("2.."))
+    socket.try(self.tp:command("EHLO", domain or DOMAIN))
+    return socket.try(self.tp:check("2.."))
+end 
+
+function metat.__index:mail(from)
+    socket.try(self.tp:command("MAIL", "FROM:" .. from))
+    return socket.try(self.tp:check("2.."))
+end 
+
+function metat.__index:rcpt(to)
+    socket.try(self.tp:command("RCPT", "TO:" .. to))
+    return socket.try(self.tp:check("2.."))
+end
+
+function metat.__index:data(src)
+    socket.try(self.tp:command("DATA"))
+    socket.try(self.tp:check("3.."))
+    socket.try(self.tp:source(src))
+    socket.try(self.tp:send("\r\n.\r\n"))
+    return socket.try(self.tp:check("2.."))
+end
+
+function metat.__index:quit()
+    socket.try(self.tp:command("QUIT"))
+    return socket.try(self.tp:check("2.."))
+end
+
+function metat.__index:close()
+    return socket.try(self.tp:close())
+end
+
 -- send message or throw an exception
-local function send_p(control, mailt) 
-    socket.try(control:check("2.."))
-    socket.try(control:command("EHLO", mailt.domain or DOMAIN))
-    socket.try(control:check("2.."))
-    socket.try(control:command("MAIL", "FROM:" .. mailt.from))
-    socket.try(control:check("2.."))
+function metat.__index:send(mailt) 
+    self:mail(mailt.from)
     if type(mailt.rcpt) == "table" then
         for i,v in ipairs(mailt.rcpt) do
-            socket.try(control:command("RCPT", "TO:" .. v))
-            socket.try(control:check("2.."))
+            self:rcpt(v)
         end
     else
-        socket.try(control:command("RCPT", "TO:" .. mailt.rcpt))
-        socket.try(control:check("2.."))
+        self:rcpt(mailt.rcpt)
     end
-    socket.try(control:command("DATA"))
-    socket.try(control:check("3.."))
-    socket.try(control:source(ltn12.source.chain(mailt.source, stuff())))
-    socket.try(control:send("\r\n.\r\n"))
-    socket.try(control:check("2.."))
-    socket.try(control:command("QUIT"))
-    socket.try(control:check("2.."))
+    self:data(ltn12.source.chain(mailt.source, stuff()))
 end
 
+function open(server, port)
+    local tp, error = socket.tp.connect(server or SERVER, port or PORT)
+    if not tp then return nil, error end
+    return setmetatable({tp = tp}, metat)
+end
+
+---------------------------------------------------------------------------
+-- Multipart message source
+-----------------------------------------------------------------------------
 -- returns a hopefully unique mime boundary
 local seqno = 0
 local function newboundary()
@@ -147,13 +183,17 @@ function message(mesgt)
     return function() return shift(coroutine.resume(co)) end
 end
 
-function send(mailt)
-    local c, e = socket.tp.connect(mailt.server or SERVER, mailt.port or PORT)
-    if not c then return nil, e end
-    local s, e = pcall(send_p, c, mailt)
-    c:close()
-    if s then return true
-    else return nil, e end
-end
+---------------------------------------------------------------------------
+-- High level SMTP API
+-----------------------------------------------------------------------------
+send = socket.protect(function(mailt)
+    local server = mailt.server or SERVER
+    local port = mailt.port or PORT
+    local smtp = socket.try(open(server, port))
+    smtp:greet(mailt.domain or DOMAIN)
+    smtp:send(mailt)
+    smtp:quit()
+    return smtp:close()
+end)
 
 return smtp
diff --git a/src/tcp.c b/src/tcp.c
index 46efac2..d0bc957 100644
--- a/src/tcp.c
+++ b/src/tcp.c
@@ -15,6 +15,7 @@
 #include "socket.h"
 #include "inet.h"
 #include "options.h"
+#include "base.h"
 #include "tcp.h"
 
 /*=========================================================================*\
@@ -40,6 +41,7 @@ static int meth_dirty(lua_State *L);
 /* tcp object methods */
 static luaL_reg tcp[] = {
     {"__gc",        meth_close},
+    {"__tostring",  base_meth_tostring},
     {"accept",      meth_accept},
     {"bind",        meth_bind},
     {"close",       meth_close},
@@ -58,7 +60,6 @@ static luaL_reg tcp[] = {
     {"settimeout",  meth_settimeout},
     {"shutdown",    meth_shutdown},
     {NULL,          NULL}
-
 };
 
 /* socket option handlers */
diff --git a/src/tp.lua b/src/tp.lua
index 9365255..e9e38a1 100644
--- a/src/tp.lua
+++ b/src/tp.lua
@@ -18,22 +18,19 @@ setfenv(1, socket.tp)
 
 TIMEOUT = 60
 
--- gets server reply
-local function get_reply(sock)
+-- gets server reply (works for SMTP and FTP)
+local function get_reply(control)
     local code, current, separator, _
-    local line, err = sock:receive()
+    local line, err = control:receive()
     local reply = line
     if err then return nil, err end
     _, _, code, separator = string.find(line, "^(%d%d%d)(.?)")
     if not code then return nil, "invalid server reply" end
     if separator == "-" then -- reply is multiline
         repeat
-            line, err = sock:receive()
+            line, err = control:receive()
             if err then return nil, err end
-            _,_, current, separator = string.find(line, "^(%d%d%d)(.)")
-            if not current or not separator then 
-                return nil, "invalid server reply" 
-            end
+            _,_, current, separator = string.find(line, "^(%d%d%d)(.?)")
             reply = reply .. "\n" .. line
         -- reply ends with same code
         until code == current and separator == " " 
@@ -42,60 +39,73 @@ local function get_reply(sock)
 end
 
 -- metatable for sock object
-local metatable = { __index = {} }
+local metat = { __index = {} }
 
-function metatable.__index:check(ok)
-    local code, reply = get_reply(self.sock)
+function metat.__index:check(ok)
+    local code, reply = get_reply(self.control)
     if not code then return nil, reply end
     if type(ok) ~= "function" then
         if type(ok) == "table" then 
             for i, v in ipairs(ok) do
-                if string.find(code, v) then return code, reply end
+                if string.find(code, v) then return tonumber(code), reply end
             end
             return nil, reply
         else
-            if string.find(code, ok) then return code, reply 
+            if string.find(code, ok) then return tonumber(code), reply 
             else return nil, reply end
         end
-    else return ok(code, reply) end
+    else return ok(tonumber(code), reply) end
 end
 
-function metatable.__index:command(cmd, arg)
-    if arg then return self.sock:send(cmd .. " " .. arg.. "\r\n")
-    else return self.sock:send(cmd .. "\r\n") end
+function metat.__index:command(cmd, arg)
+    if arg then return self.control:send(cmd .. " " .. arg.. "\r\n")
+    else return self.control:send(cmd .. "\r\n") end
 end
 
-function metatable.__index:sink(snk, pat)
-    local chunk, err = sock:receive(pat)
+function metat.__index:sink(snk, pat)
+    local chunk, err = control:receive(pat)
     return snk(chunk, err)
 end
 
-function metatable.__index:send(data)
-    return self.sock:send(data)
+function metat.__index:send(data)
+    return self.control:send(data)
+end
+
+function metat.__index:receive(pat)
+    return self.control:receive(pat)
+end
+
+function metat.__index:getfd()
+    return self.control:getfd()
+end
+
+function metat.__index:dirty()
+    return self.control:dirty()
 end
 
-function metatable.__index:receive(pat)
-    return self.sock:receive(pat)
+function metat.__index:getcontrol()
+    return self.control
 end
 
-function metatable.__index:source(src, instr)
+function metat.__index:source(src, instr)
     while true do
         local chunk, err = src()
         if not chunk then return not err, err end
-        local ret, err = self.sock:send(chunk)
+        local ret, err = self.control:send(chunk)
         if not ret then return nil, err end
     end
 end
 
--- closes the underlying sock
-function metatable.__index:close()
-    self.sock:close()
+-- closes the underlying control
+function metat.__index:close()
+    self.control:close()
+        return 1
 end
 
--- connect with server and return sock object
+-- connect with server and return control object
 function connect(host, port)
-    local sock, err = socket.connect(host, port)
-    if not sock then return nil, err end
-    sock:settimeout(TIMEOUT)
-    return setmetatable({sock = sock}, metatable)
+    local control, err = socket.connect(host, port)
+    if not control then return nil, err end
+    control:settimeout(TIMEOUT)
+    return setmetatable({control = control}, metat)
 end
diff --git a/src/udp.c b/src/udp.c
index a0fdcc0..a2dff34 100644
--- a/src/udp.c
+++ b/src/udp.c
@@ -15,6 +15,7 @@
 #include "socket.h"
 #include "inet.h"
 #include "options.h"
+#include "base.h"
 #include "udp.h"
 
 /*=========================================================================*\
@@ -50,6 +51,7 @@ static luaL_reg udp[] = {
     {"close",       meth_close},
     {"setoption",   meth_setoption},
     {"__gc",        meth_close},
+    {"__tostring",  base_meth_tostring},
     {"getfd",       meth_getfd},
     {"setfd",       meth_setfd},
     {"dirty",       meth_dirty},
diff --git a/test/ltn12test.lua b/test/ltn12test.lua
index 1c1f3f2..04749a5 100644
--- a/test/ltn12test.lua
+++ b/test/ltn12test.lua
@@ -1,3 +1,5 @@
-sink = ltn12.sink.file(io.open("lixo", "w"))
-source = ltn12.source.file(io.open("luasocket", "r"))
-ltn12.pump(source, sink)
+a = ltn12.source.file(io.open("luasocket.lua"))
+b = ltn12.source.file(io.open("auxiliar.lua"))
+c = ltn12.source.cat(a, b) 
+d = ltn12.sink.file(io.stdout)
+socket.try(ltn12.pump.all(c, d))
diff --git a/test/testmesg.lua b/test/testmesg.lua
index 8b33133..796977e 100644
--- a/test/testmesg.lua
+++ b/test/testmesg.lua
@@ -49,5 +49,6 @@ print(socket.smtp.send {
     rcpt = "<diego@cs.princeton.edu>",
     from = "<diego@cs.princeton.edu>",
     source = socket.smtp.message(mesgt),
-    server = "mail.cs.princeton.edu"
+    server = "mail.iis.com.br",
+    port = 7
 })